# Cleaning UK Biobank phenotype and NMR metabolomics data in R

Practical workflow for cleaning UK Biobank data: the ukbnmr R package for QC and technical-variation removal on the Nightingale NMR metabolomics data, ukbtools for collapsing bulk-download phenotypes, and the UKBB encoding gotchas (missing-value codes, instance and array indices) plus a per-field harmonization decision log and RAP export strategy.

Exact reference: {"kind":"skill_version","skill_id":"skl_4ivrn0sa5Zq5wQF_HHj07Q","version_id":"skv_yROzjpGxgYgsFGODYtnrcQ"}

Applicability: [{"constraint":"ukbnmr on CRAN; ukbtools on CRAN","technology":"R","version_scheme":"unknown"}]

# Cleaning UK Biobank Phenotype and NMR Metabolomics Data

## When to use this
You are working with UK Biobank (UKBB) data in R: cleaning phenotypes for
analysis, or handling the NMR metabolomics biomarker data (Nightingale/UKBB
data, Showcase label 220), especially when exporting from the UK Biobank
Research Analysis Platform (RAP) into RStudio.

## NMR metabolomics: use ukbnmr

For the NMR metabolomics data, use **ukbnmr** (Ritchie S.C. et al., "Quality
control and removal of technical variation of NMR metabolic biomarker data in
~120,000 UK Biobank participants", Scientific Data 10:64, 2023). It is the
community-standard R package for this data. Available on CRAN as `ukbnmr`;
development version at github.com/sritchie73/ukbnmr. Updated for the full
~500k-participant NMR release.

What it does, in order:

1. Extracts the biomarkers from the decoded field data.
2. Applies the published QC and technical-variation removal (plate and
   spectrometer drift and related effects).
3. Recomputes derived biomarkers and ratios consistently after QC.

Critical step many people skip: read the paper's diagnostic plots before
touching the data. They show how large the technical effects are. Running
analysis on uncorrected NMR data misattributes technical variation to biology.

## General phenotype cleaning

### ukbtools
**ukbtools** (Hanscombe, Coleman, Traylor, Lewis; GitHub kenhanscombe, on
CRAN) collapses the raw UKB fileset (.tab/.r/.html) into a usable dataframe
with human-readable column names, and includes helpers for ICD-code
retrieval. Less relevant if you stay entirely on RAP exports, but useful the
moment you work from a bulk download.

### UKBB encoding gotchas (the bulk of the work)
Most phenotype-cleaning pain is UKBB's own encoding, not tooling:

- **Missing-value codes.** UKBB uses codes like -1 (do not know), -3
  (prefer not to answer), etc. Treat them as missing before any statistics;
  never let them enter a numeric calculation as real values.
- **Instance indices.** Instance 0 is the baseline assessment; instance 1
  is the first repeat imaging visit, and so on. Decide your harmonization
  rule per field *before* merging instances — for example, "baseline unless
  missing, then earliest repeat."
- **Array indices.** Multi-answer fields carry multiple values per instance
  (e.g. several medication entries). Flatten deliberately, documenting which
  rule you used (e.g. collapse to presence flags or keep as lists).

Write the per-field harmonization rules down. That decision log is what makes
the cleaning reproducible and reviewable.

### RAP export strategy
On the Research Analysis Platform, push the heavy joins and filtering into
the Table Exporter / Cohort Browser (or `dxpy`) so you export an
analysis-ready slice, not the bulk tables. Then apply the phenotype rules
above inside RStudio.

## Checklist

1. Identify which fields need NMR treatment (Showcase label 220) versus
   phenotype treatment.
2. NMR: run the ukbnmr QC pipeline first; read its diagnostic plots.
3. Bulk-download path: collapse the fileset with ukbtools.
4. Recode missing-value codes (-1, -3, etc.) to NA before analysis.
5. Fix a per-field harmonization rule for instances and arrays, and log it.
6. Read the Showcase documentation for your specific fields — no package
   replaces it.

## References
- Ritchie S.C. et al., Sci Data 10:64 (2023) — ukbnmr QC pipeline.
- ukbnmr: CRAN package `ukbnmr`; source at github.com/sritchie73/ukbnmr.
- ukbtools: CRAN package `ukbtools`; source at github.com/kenhanscombe/ukbtools.


## Supporting basis and limitations

Source draft: ~/workspace/goals/vectle-growth-strategy/tests/drafts/biostars-9608995-answer.md (tested draft answering BioStars question 9608995, 'Pipelines/Tools for cleaning UK Biobank data?', drafted 2026-09-17, NOT POSTED). Its web-search verification on 2026-09-17 confirmed: (1) ukbnmr, R package by sritchie73 on CRAN and GitHub, from Ritchie S.C. et al., 'Quality control and removal of technical variation of NMR metabolic biomarker data in ~120,000 UK Biobank participants', Sci Data 10:64 (2023), updated for the full ~500k-participant NMR release; (2) ukbtools, R package by kenhanscombe (Hanscombe, Coleman, Traylor, Lewis) on CRAN, collapses UKB fileset (.tab/.r/.html) into a dataframe with meaningful variable names, with ICD retrieval helpers. All recommendations (diagnostic plots first, missing-value codes -1/-3, instance indices 0=baseline, array indices, per-field harmonization log, RAP Table Exporter / dxpy push-down) reproduce the draft verbatim with no invented facts.

## Change and rationale

New skill drafted from the tested BioStars 9608995 answer draft. Covers the ukbnmr NMR QC pipeline (Ritchie et al., Sci Data 2023), the ukbtools phenotype-collapsing package, UKBB encoding rules (missing codes -1/-3, instance indices, array indices), a written per-field harmonization plan, and pushing heavy filtering into the RAP Table Exporter / dxpy before RStudio analysis. No changes to existing skills; this is an initial draft and has not been published.

Cleaning UK Biobank data is a repeated, error-prone workflow that many researchers hit independently: the NMR metabolomics data requires a specific published QC pipeline to avoid misattributing technical variation (plate/spectrometer drift) to biology, and phenotype cleaning is dominated by UKBB's own encoding conventions rather than tooling. A single skill capturing the package choice, the diagnostic-plot discipline, the encoding gotchas, and the RAP export strategy addresses the exact question the source draft was verified against. Faithful to the source draft: no package names, citations, or version claims added beyond what the draft verified on 2026-09-17.