User-generated data

1. Big picture and the three products

User-generated data are content created by the users of a product; here, sport and health data from wearables and phones. They create value for research, products and insights through a larger sample size, realistic settings, and unforeseen outcomes (findings nobody planned to look for).

The lecture's three products:

• HRV4Training (2012–). A phone app: each morning a fingertip on the camera measures resting heart rate and heart-rate variability (HRV, how much the time between heartbeats varies). Users tag days as "Sick", "Travel", "Menstruation". New use: identifying and managing stressors.
• Bloomlife (2015–). A pregnancy wearable recording uterine and cardiac activity. New question: can we detect or predict the onset of labour?
• Oura ring (2013–). Built for sleep tracking, but it also records heart rate, HRV and temperature. New question: can we detect or predict an infection?

What they share: none of these uses was the original goal of the product. User-generated data made them possible through two ingredients:

1. Contextual data: context, confounders (third factors that affect both the thing you study and the outcome) and other variables, monitored longitudinally (repeatedly, in the same people, over time).
2. Reference points: key events, obtained through APIs (section 8) or reported by the user, such as a clinical diagnosis.

Context explains a change: "drank alcohol last night". A reference point is the outcome or key event you predict or line the data up against: a race time, the day someone got sick, a delivery date.

Figure: The path from raw wearable data to a useful product. Skip any gate and the product fails.

2. Lab study vs user-generated data

A typical study has five steps: (1) design the study, choosing the dependent variable (what you want to explain) and the independent variables (what might explain it); (2) recruit participants, usually a small N; (3) collect high-quality data; (4) analyse them; (5) use the outcome: write a paper, or, in a company, deploy a feature.

The lecture's example: HRV in response to exercise intensity, with training intensity, age and sex as independent variables, in N = 10 male students. Does the result hold for women, other menstrual-cycle phases, other ages, people with health conditions, other sports? Not much.

Typical limitations of lab studies:

• Small samples. Many sport science studies have N = 2 to 10.
• Low generalizability. Results are valid only for the specific sample analysed.
• Costly to extend. Studying another group means running another study, with new costs and time.
• Lab is not daily life. The data are high quality, but people measured under a strict protocol (fasted, no coffee, told when to relax) may not represent what happens in real life.

The new paradigm. Phones and their sensors now collect data anywhere, at large scale, in realistic settings where unforeseen outcomes can appear, and data-science infrastructure makes analysing it cheap. Instead of recruiting ten students, you put a tool on the market and thousands of people collect data for years.

The price. A lab study chooses its variables and controls quality before anyone is measured. A product collects data before any research question exists, so you inherit whatever it recorded: noise, gaps, an odd sample, missing outcomes. That is why judging data quality becomes the hard step, the one that differs most from a lab study. Setting up the software for processing and modelling stays easy.

Figure: User data win on scale and realism and lose on control and quality.

3. Three key steps: the HRV case

Heart rate (HR) is beats per minute (bpm). HRV is measured here as rMSSD, a number in milliseconds that grows when neighbouring beat intervals differ more. Stress (hard training, illness, alcohol) usually raises resting HR and lowers HRV. People differ a lot, so the app learns each user's normal range from daily morning measurements and reads today's value against it.

The lecture's three key steps, in order:

1. Validate the technology, or know its limits. Garbage in, garbage out. The phone-camera measurement (PPG: reading the pulse from changes in light through the fingertip) gave HRV values equivalent to a chest ECG (electrocardiogram, the clinical reference). Alternative: use a tool someone else has validated.
2. Deploy, and confirm lab-based insights if possible. About 30,000 users, up to 5 years each, 9 million measurements. The known lab finding, that HRV drops after higher-intensity exercise, appeared here too: the morning after hard training, HRV was about 3% below normal and resting HR about 0.6% above (after easy training, HRV was slightly above normal). Data preparation becomes the most important step.
3. Discover new relations, build new products. The pattern held in every age group from 20 to 60. Four intensity categories (rest, low, average, high) gave a staircase: the harder the day, the lower the next morning's HRV. The data also showed the effects of sickness, alcohol and the menstrual cycle, outcomes you cannot ethically test in a lab.

Why this order. Nobody supervises measurements in the wild. Step 1 shows the sensor works. If step 2 reproduces a known result, the whole pipeline (sensor, users measuring at home, cleaning, analysis) works, so new findings in step 3 are more likely to be real.

Figure: Each step earns the trust needed for the next.

4. HRV needs context

Compared with hard training (HRV about 3% down, resting HR 0.6% up), sickness lowered HRV about 10% and high alcohol intake about 12%, both raising resting HR about 6%.

From the lecture: Sickness and alcohol shifted HR and HRV two to three times as much as training. A training app that ignores lifestyle stressors mostly picks those up, and gives wrong training advice.

HRV is a sensitive but not specific marker of stress: it reacts strongly to every stressor, but a drop does not say which one. Training, a cold, a party and a long flight can look the same. Only context separates them, either manually from the user (tags such as "Sick") or automatically through APIs (a workout app, a weather service).

Group vs individual. These percentages are averages over thousands of users. A given person may react more, less, or in the opposite direction.

Figure: Lifestyle stressors move morning physiology far more than training does.

5. Quality control: the max-HR problem

The challenges of user data fall into three groups: data preparation (quality control, noisy data, missing data), reference data (which outcomes are available?) and data engineering (building the systems that collect and store data; not covered).

Wearable data are extremely noisy and often inaccurate, and typically no signal-quality metric is reported: the watch shows a heart rate even when the signal is bad, so you must decide when to trust it.

The lecture's example: training intensity. How hard a session was for a person is its heart rate relative to their maximum:

Without lab tests, max HR must be estimated from the workouts. That assumes the monitored period contains some hard sessions, so it must be long enough. Across 500 users, some per-workout maxima were above 300 bpm (impossible) or below 100 bpm. A simple method flags values more than 3 standard deviations (SD, the typical distance of values from their mean) from the mean as outliers.

For one user, the highest recorded value was 234 bpm: an artifact, a reading caused by a measurement problem rather than by the body. After removing outliers the estimate was 208 bpm. An age-based formula gave 180 bpm, which cannot be trusted because max HR differs hugely between people.

Worked example: A session at 170 bpm for this user.

• Raw max 234: 170 / 234 = 73% of max, a moderate session.
• Cleaned max 208: 170 / 208 = 82%, the best estimate.
• Age-based max 180: 170 / 180 = 94%, close to all-out.

But did they ever go hard? This is a separate problem. If someone always trains easy, their true maximum is simply not in the data. Removing the 234 bpm glitch fixes the artifact; it does nothing for the never-hard user, whose clean maximum is still too low, so every session looks harder than it was. A high percentile (the value below which, say, 95% of readings fall) resists one wild reading better than the maximum does (see Lecture 7), but it cannot create a maximal effort either.

From the lecture: Never-hard users show up as a very narrow spread of session heart rates. How much they matter depends on how common they are: 3 out of 7 users is a big problem, 3 out of a million barely matters.

Only a fraction of the data will be usable, so you need automatic methods that decide what to trust. Keep unreliable data and estimates suffer; throw too much away and the sample shrinks.

Figure: Panel A uses the lecture's values (234, 208, 180). Panel B illustrates a runner who never trains hard.

6. Missing data and selection bias

Missing data are values that should exist but were not recorded: workouts not synced, or no hard effort at all. With so much data, it is tempting to drop every user with gaps. You sometimes can, but it could introduce a bias, because you no longer have the full picture.

People with gaps are not random. Keep only complete logs and you keep the disciplined, consistent, probably fitter users. That is selection bias: the people left differ systematically from the people you want to describe. App data add self-selection: users chose the app, so they are health-interested and, in the running study, mostly men. A bigger N does not fix this. More data shrink random error, not a systematic difference in who is included.

And you can always do one more stratification (split into subgroups): one age group, then one sport, then one type of person, and thousands of users shrink to a handful. There is no universal answer: think critically about what is missing and why.

7. Reference data

Reference data are the outcomes or key events you need to learn anything: a race time, the day someone got sick, a delivery date. For a supervised model (one that learns from examples with known answers), the reference outcome is the label, or ground truth, it learns to predict.

Lack of reference data is one of the biggest challenges with user data: users don't come to the lab for tests or report the outcomes models need. What helps: tags, annotations and APIs. The food-diary app MyFitnessPal shows the risk: millions of users logged food, but with no health or performance outcomes there was nothing to learn.

Before collecting data, ask:

• What are the outcomes?
• Can we track them?
• Are we asking too much of the user? This is not a clinical study, so the burden must stay low.
• What can we do about it? For example, fetch data automatically through APIs.
• Is it ethical to collect them?

Sickness: when did it start? To detect an infection (the Oura question) you need the day the sickness began. The infection day is usually unknown, and symptoms and tests each give a different day, so day 0 is a choice: the day physiology changes, the first symptom, a cluster of symptoms, or a positive test. The choice changes the results.

From the lecture: Wearables detect sickness at best; they do not predict it. If the data show a change, the body is already sick. A ring that "predicts" infection because heart rate rose before reported symptoms is only ahead of a late reference point.

Figure: Each event is a different possible day 0. A positive test date is not the infection date.

8. APIs

Definition. An API (application programming interface) is the access point through which one program automatically requests data from another, usually by sending requests to a server. No person types anything and no user interface is involved.

Sport example. HRV4Training users can link their Strava or TrainingPeaks account (apps where athletes log workouts). Through the Strava API, HRV4Training then fetches each run's date, distance, time, GPS and heart rate; the running study in section 9 got its workouts this way. Likewise, Sport Data Valley pulls new data from Garmin or Polar watches, and a weather API can add temperature or altitude as context without asking the user anything.

What an API cannot fix. It only moves data that already exist. It does not make a sensor accurate, fill in missing workouts, make users representative, or check that an outcome label is correct.

9. Case: 10 km running performance (Altini & Amft, 2018)

• Question and data. Can a runner's 10 km time be estimated from data recorded in daily life, without lab tests? About 2,100 HRV4Training users (2016–2017) who linked Strava or TrainingPeaks; each runner's fastest 10 km over two years was the reference outcome.
• Method. Features (predictors computed from the data) from the 3 months before that best 10 km, added in six sets; multiple linear regression, tested with 10-fold cross-validation.
• Finding. Explained variance rose from R² = 0.33 to 0.87, with a typical error of about 2.6 minutes; previous performance was the strongest predictor, and about 15 workouts were enough.
• Limitation. Associations in observational data, not causes; a self-selected, mostly male sample; and the reference is the best logged workout, not a standardized race.

The answer is a number (minutes), so this is regression (Lecture 5). Multiple linear regression predicts the time as a weighted sum of several features, and was chosen because its coefficients are easy to interpret. In 10-fold cross-validation the data are split into 10 parts and each part is predicted by a model trained on the other nine, so no runner is predicted by a model that saw them.

Feature sets, added one at a time:

1. Anthropometrics (body measures): BMI, age, sex.
2. Resting physiology: morning HR and HRV.
3. Training volume and average speed.
4. Heart rate during training, relative to running speed.
5. Training-intensity distribution: how polarized the training is (mostly easy plus some clearly hard sessions, little in between).
6. Previous performance: the best 10 km in the 3 months before the target one.

Fast vs slow runners. Slower quarters of the sample had a higher BMI, fewer kilometres per workout, a higher heart rate for the same speed (lower fitness), and less polarized training. This is association only: fitter runners may simply choose to train more, and more polarized.

Figure: Error falls as features are added. Volume and speed give the biggest jump; previous performance gives the best model.

Reading the results.

• RMSE (root mean squared error; computed in Lecture 5) is the typical size of a prediction error, in minutes here. It fell from about 6.3 minutes (anthropometrics only) to about 2.6.
• R² is the share of the differences in 10 km time between runners that the model explains: 0.33 with resting physiology, 0.71 with volume and speed, 0.76 with intensity distribution, 0.87 with previous performance. R² is not accuracy: 0.87 does not mean 87% of predictions are correct.
• 4% is the mean percentage error: on average a prediction missed by about 4% of the real time.
• How much data? The error dropped until about 15 workouts and no further after that, which tells a product when to show a first estimate.

10. Case: resting heart rate in the pandemic

During the first COVID-19 lockdown (March to May 2020), about two thirds of people in a poll expected resting heart rate to rise, because of stress.

Data from about 5,500 users, 3 months each, half a million measurements, showed the opposite: resting HR fell, to roughly 1.4 bpm below its January level by April and May, against a dip of about 0.4 bpm in 2019. The same app data suggest why: travel dropped sharply and sleep time rose. People stopped travelling and commuting, so they slept more. These are plausible explanations, not proven causes.

No lab could plan this; the app was already measuring thousands of people daily, 2019 included. But limitations still apply:

• Who are we talking about? The users are self-selected, health-interested people, so the result may not generalize.
• Group averages. Some individuals' resting HR went up; the average hides them.

Figure: Resting heart rate relative to January, 2019 in grey and 2020 in blue, first wave boxed. Negative means lower than in January.

11. Scale and data products

A false-positive rate (the share of healthy people who get a wrong alarm) is a property of the method and does not change with the number of users; the number of false alarms does.

Worked example: An illness alert with a 5% false-positive rate. Study of 20 healthy people: 0.05 × 20 = 1 false alarm. App with 1,000,000 healthy users: 0.05 × 1,000,000 = 50,000 false alarms, enough to swamp family doctors.

The lecture's conclusions:

• Not everything is, or can be, a data product. Data are often collected but never used meaningfully, creating no value for the company or the user.
• Reference points are key: you can have unlimited data and still have no use for it.
• More research uses consumer products: a different opportunity, not better or worse than academic research.
• Think critically about reference points, data preparation and estimated vs measured values: the age-based max HR (180 bpm) is an estimate; the cleaned max from the user's workouts (208 bpm) is measured.

Exam traps

• Hardest workflow step with user data: judging data quality. Easiest: setting up the software.
• Order: validate → confirm a known lab result → discover. Wrong options swap steps 2 and 3.
• HRV is sensitive, not specific: a drop says "stress", not which stressor.
• Cleaning fixes the artifact (234 bpm), not the missing hard effort.
• Context explains a change; reference data are the outcome or key event.
• A bigger N does not fix selection or self-selection bias.
• Wearables detect sickness; they don't predict it. Infection, symptom and test days differ.
• Running study: the lecture's summary says "N = 2100, RMSE = 2 minutes (4%)"; the paper reports 2,113 runners and 2.6 minutes. 4% is the mean percentage error, not accuracy.
• R² = 0.87 is variance explained, not 87% correct; a time outcome means regression, not classification.
• About 15 workouts (training sessions), not 15 days.
• The false-positive rate stays fixed; the count grows with users.