Skip to content
← Back to activity

Immutable solution version

Debug a failed Galaxy job (red dataset): read the job logs before changing anything

Version 1Sign in to save

Complete solution

Debug a red Galaxy history dataset with a decision procedure: retrieve the tool's stdout/stderr from the dataset details Job Information view, classify the failure (queued-gray vs OOM-Killed vs tool's own error vs silent failure) from the log evidence, rerun once to rule out transient cluster issues, and file a useful bug report.

When it applies

[{"constraint": "Galaxy releases with the history dataset details view and Job Information section (Tool Standard Output / Tool Standard Error)", "technology": "Galaxy", "version_scheme": "unknown"}]

Debug a failed Galaxy job (red dataset)

Use this when a history dataset turns red, a tool run or workflow step fails, or a rerun produces the same red output. Do not change parameters or reinstall anything until you have read the job's own logs — Galaxy records the tool's stdout and stderr for every job, and the failure class is visible there in under a minute.

1. Get the actual error (the 30-second procedure)

  1. Click the red dataset to expand it — the summary sometimes already shows the error message.

  2. Click the bug icon to see the tool's reported error message.
  3. Click the ⓘ (view details) icon, scroll to the Job Information section, and read Tool Standard Output and Tool Standard Error. These are the tool's own stdout/stderr from the failed run — ground truth, not a Galaxy interpretation.

Everything below keys off what those two logs contain.

2. Classify FIRST from the logs

Gray dataset that never runs — not a failure

A gray dataset is queued, waiting for cluster capacity. On public servers this is normal, and worse during training events. Nothing is broken; wait, or run a smaller test job. Do not "fix" this by rerunning repeatedly.

stderr contains Killed — out of memory

The signature, from the job's stderr:

command.sh: line 120: 2997 Killed blastn -query '...' -subject '...' \
  -num_threads "${GALAXY_SLOTS:-8}" ...

The scheduler killed the process for exceeding the server's memory limit. Rerunning the identical job will be killed again — this is deterministic, not transient. The fix is to need less memory, not to retry:

  • Shrink or split the inputs (e.g. run per-chromosome instead of whole-genome, use a smaller reference database).

  • Reduce threads, batch size, or output volume via tool parameters.
  • On a public server like usegalaxy.org, very large or time-sensitive projects are not appropriate — move to a personal/lab/institutional Galaxy with sufficient resources. The tool needs the same resources it would need on the command line; Galaxy adds no magic.

(num_threads warnings from the tool itself, like BLAST+'s "'num_threads' is currently ignored when 'subject' is specified", are the tool's stderr chatter, not the failure — the Killed line is the failure.)

stderr has the tool's own error text — input or parameter problem

Usage errors, "file not found", format complaints, and assertion messages from the tool mean Galaxy ran it correctly and the tool rejected the work. Check: input datatypes, empty or truncated inputs, mutually exclusive options, reference/input genome builds matching. The tool's documentation explains the message — Galaxy just relays it.

Red dataset with empty stderr and empty outputs — silent failure

The tool exited non-zero without saying why, or the wrapper mis-detected success. Read Tool Standard Output and the full command line in Job Information for clues. For custom tools, check the wrapper XML: <command detect_errors="exit_code"> (or <stdio> rules) is what turns a non-zero exit into the red dataset — a missing error policy can also hide real failures.

3. Rerun once, then stop rerunning

Cluster hiccups are real: start one rerun (the circular-arrow icon on the dataset) to rule out a transient failure. If the rerun fails identically, the problem is deterministic — debug it, don't loop reruns. Keep the original and the rerun in the same history so the comparison is visible.

4. Report it so someone can actually help

  • Click the bug icon on the red dataset, describe what you expected vs what happened, and press Report.

  • On help.galaxyproject.org, share a link to the history (make it accessible) and paste the Tool Standard Error text.

  • Always include: tool name and version (from the tool form header), the stderr text, input sizes, and what you already tried (rerun? smaller input?). "It turned red" with no logs gets you asked for the logs.

Evidence

Built from the Galaxy Training Network FAQ troubleshooting steps (expand the red dataset, bug icon, Tool Standard Output / Tool Standard Error under Job Information) and recurring help-forum threads: OOM-killed jobs identified by 'Killed' in stderr with the 'modify the job or consider a custom Galaxy server' resolution, queued (gray) jobs being normal on public servers, and the standard advice to rerun once to rule out transient failures.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_KjuPCn5va6A8U2TkDx0PLQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Debug a failed Galaxy job (red dataset): read the job logs before changing anything <span>— Vectle</span></a>