# Fix bcftools view/filter producing empty or wrong output

Diagnose bcftools filtering failures: classify usage-dump parse errors (shell quoting) vs silent zero-record output (flag-tag =1/=0 semantics, -q minor-allele-frequency, FILTER exact-vs-subset matching) vs regions-file parse errors (tab delimiter) vs unindexed VCF region queries, then apply the matching fix.

Exact reference: {"kind":"skill_version","skill_id":"skl_LGaFuUOhzO-gLSvJcG3jkg","version_id":"skv_OEfF0GVDR9KZdAhVggfROg"}

Applicability: [{"constraint":">=1.10 (filtering expression syntax, -i/-e/-q/-f options, -T/-R region file behavior)","technology":"bcftools","version_scheme":"semver"}]

# Fix bcftools view/filter producing empty or wrong output

Use this when `bcftools view` / `bcftools filter` / `bcftools query` with
`-i`/`-e` expressions, `-q`, `-f`, or `-T`/`-R` region options returns zero
records, the wrong records, or an error. These failures look identical ("my
filter didn't work") but split into four classes with different fixes.
Classify from the exact symptom FIRST — re-running with a different guess at
the expression wastes the most time here.

## 1. Classify the symptom

### A. bcftools prints the usage text and exits

The expression failed to parse — usually **shell quoting**, not bcftools
logic. Classic real case: `-i "INFO/IMP'` (mismatched quote) makes the shell
hand bcftools a broken argument and bcftools dumps usage. Rule: wrap the
whole expression in single quotes and never mix quote types:

```bash
# right
bcftools view -i 'INFO/IMP=1' in.vcf.gz
# wrong: the double-quote never closes
bcftools view -i "INFO/IMP' in.vcf.gz
```

If quoting looks fine, simplify the expression until it parses, then rebuild
it — the parser chokes on the first bad token, and the usage dump doesn't
say which one.

### B. No error, but zero (or far too few) records out

The expression parsed; it just doesn't match what you think it matches.
**Do not eyeball `| head`** — a VCF header alone is often >10 lines, so
`bcftools view -i '...' file.vcf | head -10` showing "only header" proves
nothing. Count records properly:

```bash
bcftools view -H -i 'INFO/IMP=1' in.vcf.gz | wc -l   # -H: no header
```

Then check the three classic mismatches, in order:

1. **Flag tags need `=1` / `=0`.** An INFO flag (`Number=0,Type=Flag` in the
   header, e.g. `##INFO=<ID=IMP,Number=0,Type=Flag,...>`) has no value to
   compare — test presence, per the docs ("1 (or 0) to test the presence
   (or absence) of a flag"):

   ```bash
   bcftools view -i 'INFO/IMP=1' in.vcf.gz    # flag present
   bcftools view -i 'INFO/IMP=0' in.vcf.gz    # flag absent
   ```

   Bare `'INFO/IMP'` also tests presence, but `=1` is explicit and matches
   the documented idiom.

2. **`-q` / `--min-af` is a minor-allele-frequency filter, not "AF column
   greater than x".** `bcftools view -q 0.1` keeps sites with MAF ≥ 0.1 and
   drops everything else — if your AF annotation is missing, computed
   differently, or mostly rare variants, you get an empty VCF with no error.
   Inspect what the filter actually sees before assuming:

   ```bash
   bcftools view -h in.vcf.gz | grep '##INFO=<ID=AF'   # is AF even annotated?
   bcftools query -f '%CHROM\t%POS\t%INFO/AF\n' in.vcf.gz | head
   ```

   If you wanted "AF > 0.1", write it as an expression instead:
   `bcftools view -i 'AF>0.1' in.vcf.gz`.

3. **FILTER-column matching is exact vs subset.** In expressions,
   `FILTER="PASS"` is an exact match (`"A;B"` fails it), while `FILTER~"A"`
   is a subset match. And `-f/--apply-filters` *skips* sites whose FILTER
   lacks every listed string — to keep unfiltered sites, include `.`:
   `bcftools view -f .,PASS in.vcf.gz`.

General probe for any suspicious `-i`/`-e`: run the same expression through
`bcftools query -f` printing the tags involved, on a file you can inspect:

```bash
bcftools query -f '%CHROM\t%POS\t%FILTER\t%INFO/DP\n' -i 'DP>10' in.vcf.gz | head
```

If the printed rows don't satisfy your mental model of the expression,
the model — not the data — is wrong.

### C. `[E::bcf_sr_regions_init] Could not parse N-th line of file`

The regions/targets file format is wrong. bcftools expects **tab-delimited**,
2 columns (`CHROM`, `POS`, 1-based inclusive) or 3 columns (`CHROM`, `BEG`,
`END`) — never space-delimited. The recurring real-world cause: `awk`
prints with `OFS=" "` by default. Fix at creation:

```bash
awk -F';' 'NR>1 {OFS="	"; print $1,$2}' chr.csv > regions.txt
bcftools view -T regions.txt -Oz -o filtered.vcf.gz in.vcf.gz
```

(`-T`/`--targets-file` streams by POS; `-R`/`--regions-file` index-jumps and
checks proper overlaps including indel ends.)

### D. Region query fails or returns nothing on an indexed-looking file

1. **The VCF/BCF must actually be indexed** for `-r`/`-R` ("This option
   requires indexed VCF/BCF files"). Index it:

   ```bash
   bcftools index in.vcf.gz        # creates in.vcf.gz.csi
   ```

   Note `-T` is stricter still: its targets file "must be compressed and
   indexed" (`bgzip` + `tabix` the targets file, or use `-R` with a plain
   text list instead).

2. **Contig names must match exactly.** Per the docs: `"chr20"` is not the
   same as `"20"`. Compare `bcftools view -h in.vcf.gz | grep '^##contig'`
   against your region strings; normalize one side (`bcftools annotate
   --rename-chrs`) if they disagree.

3. `-r` and `-R` can't be combined; `-t` and `-T` can't be combined.
   Pick one.

## 2. Useful expression idioms (documented)

```bash
bcftools view -i 'QUAL>20 && DP>10' in.vcf.gz          # compound threshold
bcftools view -e 'FILTER!="PASS"' in.vcf.gz           # drop non-PASS (exact match)
bcftools view -i 'FILTER~"q10"' in.vcf.gz             # FILTER contains q10 (subset)
bcftools view -m2 -M2 -v snps in.vcf.gz               # biallelic SNPs only
bcftools view -i 'GT="alt"' in.vcf.gz                 # at least one alt genotype
bcftools view -i 'DP="."' in.vcf.gz                   # DP is missing
```

Tags can be written `INFO/DP` or bare `DP`, `FORMAT/DV` / `FMT/DV` / `DV`.

## Quick decision tree

1. Usage text dumped → quoting/parse error (class A); single-quote the expression.
2. Zero records, no error → count with `-H | wc -l`, then flag `=1`/`=0`, then `-q`-is-MAF, then FILTER exact-vs-subset (class B).
3. `Could not parse N-th line` → tab-delimit the regions file (class C).
4. Region query broken → index the VCF (`bcftools index`), check exact contig names, `-T` needs bgzipped+indexed targets (class D).


## Supporting basis and limitations

Built from the bcftools 1.21 documentation (filtering expressions: flag presence tests, FILTER exact/subset matching, missing-value tests; -R requires indexed VCF/BCF; -T targets file must be compressed and indexed; contig names must match exactly; -f .,PASS idiom) and recurring Biostars threads on empty filter output and regions-file parse errors.

## Change and rationale

New skill: fix bcftools view/filter producing empty or wrong output.

bcftools filtering questions recur on Biostars because the same symptom (empty output) has unrelated causes: broken shell quoting, flag tags tested without =1, -q misunderstood as an AF threshold, space-delimited regions files, and unindexed VCFs. Agents guess at expressions instead of diagnosing. This skill adds a classify-first procedure with empirical probes (bcftools query, -H record counts) before any expression is rewritten.