Fix Galaxy upload and datatype-detection problems

TL;DR

For "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte": a tool's input selector lists only datasets of compatible types. - You can fix a wrong assignment at any time without re-uploading: click the pencil (Edit Attributes) icon → Datatypes tab → pick the correct type → Save. Fix: assign the matching.gz datatype (e.g. fastqsanger.gz) via Edit Attributes → Datatypes, or re-upload with Auto-detect. Do not reassign datatypes on a dataset that has no data.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

When to use

You are seeing this: Galaxy tools only see datasets whose datatype they accept, so a wrong datatype looks exactly like missing data, classify the mismatch FIRST, then reassign. Use this skill when you run into "Fix Galaxy upload and datatype-detection problems".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Use this when uploaded data gets the wrong format, a tool's input dropdown refuses to list your dataset, the preview shows garbage, or metadata setting fails after upload. Galaxy tools only see datasets whose datatype they accept, so a wrong datatype looks exactly like missing data, classify the mismatch FIRST, then reassign.

1. How Galaxy assigns datatypes

• The Upload dialog has a Type option, default Auto-detect, which sniffs the format. Leave it on Auto-detect unless you have a specific reason not to, forced types are the most common source of these problems.

• After upload, the dataset's assigned datatype is shown in its expanded history view. A tool's input selector lists only datasets of compatible types.

• You can fix a wrong assignment at any time without re-uploading: click the pencil (Edit Attributes) icon → Datatypes tab → pick the correct type → Save. Re-detecting is cheaper than re-uploading, so try it first.

2. Classify from observable evidence

A. Preview shows binary garbage or metadata fails with a decode error

Signature in the dataset's error/metadata log:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

0x8b in position 1 is the tell: the bytes 1f 8b are the gzip magic number, so the file is gzip-compressed but was labeled as an uncompressed type (txt, fastqsanger, ...). Fix: assign the matching .gz datatype (e.g. fastqsanger.gz) via Edit Attributes → Datatypes, or re-upload with Auto-detect. This also happens when a .gz file is renamed to drop the extension before upload.

B. A tool's input dropdown won't list the dataset

The dataset's datatype does not match the tool's accepted formats. Verify the data genuinely is the format you want to claim (preview the first lines), then reassign via the Datatypes tab. Two rules:

• Never assign a datatype the bytes don't conform to, the tool will fail or, worse, silently misbehave on mislabeled data.

• fastq vs fastqsanger is about quality-score encoding, not filename; pick by the actual encoding. The FASTQ Groomer tool can convert between FASTQ variants if you need a real conversion rather than a relabel.

C. Auto-detect picks the wrong type

Don't fight the sniffer by forcing types blindly, inspect the file first: preview the head of the dataset, check for compression (class A), check for truncation or mixed content. For fastq and bam data, Auto-detect "should always work well"; when it doesn't, suspect the file itself (corrupt, truncated, or an unusual flavor) rather than the detector, and validate or re-obtain the source file.

D. The upload itself fails

The file never arrived, a network, quota, or source problem, not a datatype problem. Do not reassign datatypes on a dataset that has no data.

3. Checklist

1. Leave Upload on Auto-detect; only set a type manually when you know the sniffer gets that source wrong.

2. Wrong type after upload → pencil icon → Datatypes tab → correct type → Save. Re-upload only if the bytes themselves are bad.

3. 0x8b decode errors → compressed data with an uncompressed datatype; assign the .gz variant.

4. Tool can't see the dataset → datatype mismatch; confirm the bytes match before relabeling.

5. Still stuck → ask on help.galaxyproject.org with a shared history link; include the dataset's assigned datatype and the tool's expected format.

Built from Galaxy help-forum threads: the UnicodeDecodeError 'utf-8 codec can't decode byte 0x8b in position 1' metadata failure (gzipped data with an uncompressed datatype), and staff guidance to allow Galaxy to guess the datatype at upload and to use the pencil icon to re-detect the datatype, noting fastq and bam data work well with the guess.