Solution skill
Semaphore CI: job stalled and won't end - attach and inspect
Complete solution
Shows how to fix semaphore CI: job stalled and won't end - attach and inspect. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
Check running processes with ps aux or top, strace the suspicious one to see the kernel instruction it is waiting on (select can mean it is waiting for input). Set a shorter executiontimelimit on the pipeline so stale builds cannot run for the full hour while you debug.
Steps
- [Semaphore docs FAQ]: attach a terminal to the stalled job with sem attach. Check running processes with ps aux or top, strace the suspicious one to see the kernel instruction it is waiting on (select can mean it is waiting for input). Check /tmp/system-metrics for disk/memory exhaustion, /tmp/agentlogs and /tmp/joblogs.json for agent-side waits, and /var/log/syslog for Out of memory conditions. Set a shorter executiontimelimit on the pipeline so stale builds cannot run for the full hour while you debug. A job that fails even though all commands passed is usually a coverage tool failing the suite on a minimum-coverage threshold.
When to use
You are seeing this: [Semaphore docs FAQ]: attach a terminal to the stalled job with sem attach. Use this skill when you run into "Semaphore CI: job stalled and won't end - attach and inspect".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Evidence
Source: A Semaphore job hangs forever without finishing. The most common cause is a process that refuses to shut down.
Verified resolution: [Semaphore docs FAQ]: attach a terminal to the stalled job with sem attach. Check running processes with ps aux or top, strace the suspicious one to see the kernel instruction it is waiting on (select can mean it is waiting for input). Check /tmp/system-metrics for disk/memory exhaustion, /tmp/agentlogs and /tmp/joblogs.json for agent-side waits, and /var/log/syslog for Out of memory conditions. Set a shorter executiontimelimit on the pipeline so stale builds cannot run for the full hour while you debug. A job that fails even though all commands passed is usually a coverage tool failing the suite on a minimum-coverage threshold.
Source URL: https://github.com/semaphoreci/semaphore/blob/HEAD/docs/versioned_docs/version-CE/getting-started/faq.md
By @issueatlas · Version 3 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_HrT9iuLUD-fuK9Tv4kdXPQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Semaphore CI: job stalled and won't end - attach and inspect <span>— Vectle</span></a>