Cleaning UK Biobank phenotype and NMR metabolomics data in R

# Cleaning UK Biobank Phenotype and NMR Metabolomics Data

## When to use this
You are working with UK Biobank (UKBB) data in R: cleaning phenotypes for
analysis, or handling the NMR metabolomics biomarker data (Nightingale/UKBB
data, Showcase label 220), especially when exporting from the UK Biobank
Research Analysis Platform (RAP) into RStudio.

## NMR metabolomics: use ukbnmr

For the NMR metabolomics data, use **ukbnmr** (Ritchie S.C. et al., "Quality
control and removal of technical variation of NMR metabolic biomarker data in
~120,000 UK Biobank participants", Scientific Data 10:64, 2023). It is the
community-standard R package for this data. Available on CRAN as `ukbnmr`;
development version at github.com/sritchie73/ukbnmr. Updated for the full
~500k-participant NMR release.

What it does, in order:

1. Extracts the biomarkers from the decoded field data.
2. Applies the published QC and technical-variation removal (plate and
   spectrometer drift and related effects).
3. Recomputes derived biomarkers and ratios consistently after QC.

Critical step many people skip: read the paper's diagnostic plots before
touching the data. They show how large the technical effects are. Running
analysis on uncorrected NMR data misattributes technical variation to biology.

## General phenotype cleaning

### ukbtools
**ukbtools** (Hanscombe, Coleman, Traylor, Lewis; GitHub kenhanscombe, on
CRAN) collapses the raw UKB fileset (.tab/.r/.html) into a usable dataframe
with human-readable column names, and includes helpers for ICD-code
retrieval. Less relevant if you stay entirely on RAP exports, but useful the
moment you work from a bulk download.

### UKBB encoding gotchas (the bulk of the work)
Most phenotype-cleaning pain is UKBB's own encoding, not tooling:

- **Missing-value codes.** UKBB uses codes like -1 (do not know), -3
  (prefer not to answer), etc. Treat them as missing before any statistics;
  never let them enter a numeric calculation as real values.
- **Instance indices.** Instance 0 is the baseline assessment; instance 1
  is the first repeat imaging visit, and so on. Decide your harmonization
  rule per field *before* merging instances — for example, "baseline unless
  missing, then earliest repeat."
- **Array indices.** Multi-answer fields carry multiple values per instance
  (e.g. several medication entries). Flatten deliberately, documenting which
  rule you used (e.g. collapse to presence flags or keep as lists).

Write the per-field harmonization rules down. That decision log is what makes
the cleaning reproducible and reviewable.

### RAP export strategy
On the Research Analysis Platform, push the heavy joins and filtering into
the Table Exporter / Cohort Browser (or `dxpy`) so you export an
analysis-ready slice, not the bulk tables. Then apply the phenotype rules
above inside RStudio.

## Checklist

1. Identify which fields need NMR treatment (Showcase label 220) versus
   phenotype treatment.
2. NMR: run the ukbnmr QC pipeline first; read its diagnostic plots.
3. Bulk-download path: collapse the fileset with ukbtools.
4. Recode missing-value codes (-1, -3, etc.) to NA before analysis.
5. Fix a per-field harmonization rule for instances and arrays, and log it.
6. Read the Showcase documentation for your specific fields — no package
   replaces it.

## References
- Ritchie S.C. et al., Sci Data 10:64 (2023) — ukbnmr QC pipeline.
- ukbnmr: CRAN package `ukbnmr`; source at github.com/sritchie73/ukbnmr.
- ukbtools: CRAN package `ukbtools`; source at github.com/kenhanscombe/ukbtools.