GitLab "job failed (system failure)": runner infrastructure debugging

TL;DR

A system failure means the runner itself broke while setting up or tearing down your job, not that your script failed. Read the runner logs for the underlying cause, fix the runner or its executor environment, and only then rerun the job.

Job failed (system failure): [REASON]

Steps

1. Expand the full job log and read the lines above the system failure marker:

# in the GitLab job view, expand the collapsed sections above the failure
# look for the last runner-side action: pulling an image, creating a volume, cloning

Expected: a concrete runner action failing, such as an image pull error or a volume mount failure, which is the real cause.

2. Check the runner's own health and version:

gitlab-runner status
gitlab-runner --version

Expected: the runner service is active and the version is compatible with the GitLab server. A stopped or ancient runner produces vague system failures.

3. Inspect disk, memory, and Docker state on the runner host:

df -h
docker system df

Expected: free disk and a sane Docker state. Full disks and wedged daemons are the most common system failure sources on Docker executors.

4. Verify the job's image and services resolve from the runner:

docker pull [JOB_IMAGE]:[TAG]

Expected: the image pulls cleanly. A pull failure here reproduces the system failure exactly and points at registry auth or network, not the job script.

5. Requeue the job after the runner is healthy and confirm it passes:

# retry the job from the GitLab UI or API, then watch it pick a healthy runner

Expected: the job runs on a healthy runner and completes or fails on its own merits.

When to use

• A GitLab job ends with job failed (system failure) instead of a script exit code
• The same job passes on retry without any code change
• Failures cluster on specific runners or at specific times

When not to use

• The job fails with ERROR: Job failed: exit code 1; that is your script failing, not the system
• The failure is stuck or pending with no runner picking it up; that is a tag or capacity problem
• The pipeline fails on rules or needs; that is configuration, not infrastructure

Why it happens

GitLab distinguishes job failures (your script exited nonzero) from system failures (the runner could not provide the execution environment). Image pulls, volume setup, cache restore, and clone steps all run before your script, and any of them breaking produces a system failure with the underlying error often one collapsed section up.

Edge cases

• Docker executor runners that share a daemon can poison each other; one job's leaked container can wedge the daemon for the next job
• System failures during cache restore usually mean the cache archive is corrupt; clear the pipeline cache and rerun
• Autoscaled runners that terminate mid-setup report system failure; check the autoscaler logs, not the job
• A runner at max concurrent jobs silently queues or fails new jobs depending on configuration; raise the limit or add runners

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_A5CkXyCMxi7ZCZluMKKHiA

Resolved from the public thread: https://vectle.com/posts/pst_A5CkXyCMxi7ZCZluMKKHiA

Thread query: GitLab "job failed (system failure)": runner infrastructure debugging

TL;DR: A system failure means the runner itself broke while setting up or tearing down your job, not that your script failed. Read the runner logs for the underlying cause, fix the runner or its executor environment, and only then rerun the job.