# Debug a failed Galaxy job (red dataset): read the job logs before changing anything

Debug a red Galaxy history dataset with a decision procedure: retrieve the tool's stdout/stderr from the dataset details Job Information view, classify the failure (queued-gray vs OOM-Killed vs tool's own error vs silent failure) from the log evidence, rerun once to rule out transient cluster issues, and file a useful bug report.

Exact reference: {"kind":"skill_version","skill_id":"skl_R9pSrSQcVbZGIkngoxxQ-Q","version_id":"skv_FGXAgo6qUO_c-RX5-l41bQ"}

Applicability: [{"constraint":"Galaxy releases with the history dataset details view and Job Information section (Tool Standard Output / Tool Standard Error)","technology":"Galaxy","version_scheme":"unknown"}]

# Debug a failed Galaxy job (red dataset)

Use this when a history dataset turns red, a tool run or workflow step fails,
or a rerun produces the same red output. Do not change parameters or
reinstall anything until you have read the job's own logs — Galaxy records
the tool's stdout and stderr for every job, and the failure class is visible
there in under a minute.

## 1. Get the actual error (the 30-second procedure)

1. Click the red dataset to expand it — the summary sometimes already shows
   the error message.
2. Click the **bug icon** to see the tool's reported error message.
3. Click the ⓘ (view details) icon, scroll to the **Job Information** section,
   and read **Tool Standard Output** and **Tool Standard Error**. These are
   the tool's own stdout/stderr from the failed run — ground truth, not a
   Galaxy interpretation.

Everything below keys off what those two logs contain.

## 2. Classify FIRST from the logs

### Gray dataset that never runs — not a failure

A gray dataset is **queued**, waiting for cluster capacity. On public servers
this is normal, and worse during training events. Nothing is broken; wait,
or run a smaller test job. Do not "fix" this by rerunning repeatedly.

### stderr contains `Killed` — out of memory

The signature, from the job's stderr:

```
command.sh: line 120: 2997 Killed blastn -query '...' -subject '...' \
  -num_threads "${GALAXY_SLOTS:-8}" ...
```

The scheduler killed the process for exceeding the server's memory limit.
Rerunning the identical job will be killed again — this is deterministic,
not transient. The fix is to need less memory, not to retry:

- Shrink or split the inputs (e.g. run per-chromosome instead of
  whole-genome, use a smaller reference database).
- Reduce threads, batch size, or output volume via tool parameters.
- On a public server like usegalaxy.org, very large or time-sensitive
  projects are not appropriate — move to a personal/lab/institutional
  Galaxy with sufficient resources. The tool needs the same resources it
  would need on the command line; Galaxy adds no magic.

(`num_threads` warnings from the tool itself, like BLAST+'s "'num_threads'
is currently ignored when 'subject' is specified", are the *tool's* stderr
chatter, not the failure — the `Killed` line is the failure.)

### stderr has the tool's own error text — input or parameter problem

Usage errors, "file not found", format complaints, and assertion messages
from the tool mean Galaxy ran it correctly and the tool rejected the work.
Check: input datatypes, empty or truncated inputs, mutually exclusive
options, reference/input genome builds matching. The tool's documentation
explains the message — Galaxy just relays it.

### Red dataset with empty stderr and empty outputs — silent failure

The tool exited non-zero without saying why, or the wrapper mis-detected
success. Read **Tool Standard Output** and the full command line in Job
Information for clues. For custom tools, check the wrapper XML:
`<command detect_errors="exit_code">` (or `<stdio>` rules) is what turns a
non-zero exit into the red dataset — a missing error policy can also hide
real failures.

## 3. Rerun once, then stop rerunning

Cluster hiccups are real: start one rerun (the circular-arrow icon on the
dataset) to rule out a transient failure. If the rerun fails identically,
the problem is deterministic — debug it, don't loop reruns. Keep the
original and the rerun in the same history so the comparison is visible.

## 4. Report it so someone can actually help

- Click the bug icon on the red dataset, describe what you expected vs what
  happened, and press **Report**.
- On help.galaxyproject.org, share a link to the history (make it
  accessible) and paste the **Tool Standard Error** text.
- Always include: tool name and version (from the tool form header), the
  stderr text, input sizes, and what you already tried (rerun? smaller
  input?). "It turned red" with no logs gets you asked for the logs.


## Supporting basis and limitations

Built from the Galaxy Training Network FAQ troubleshooting steps (expand the red dataset, bug icon, Tool Standard Output / Tool Standard Error under Job Information) and recurring help-forum threads: OOM-killed jobs identified by 'Killed' in stderr with the 'modify the job or consider a custom Galaxy server' resolution, queued (gray) jobs being normal on public servers, and the standard advice to rerun once to rule out transient failures.

## Change and rationale

New skill: debug failed Galaxy jobs by classifying from the job's stdout/stderr before changing parameters or rerunning.

Red datasets are the most common Galaxy user support question, and most advice jumps to rerunning or tweaking parameters without reading the logs Galaxy already records. The Killed/OOM signature in particular is deterministic — rerunning never helps — so a classify-first procedure saves users and support staff from wasted cycles.