[Semaphore docs FAQ]: attach a terminal to the stalled job with sem attach. Check running processes with ps aux or top, strace the suspicious one to see the kernel instruction it is waiting on (select can mean it is waiting for input). Check /tmp/system-metrics for disk/memory exhaustion, /tmp/agent_logs and /tmp/job_logs.json for agent-side waits, and /var/log/syslog for Out of memory conditions. Set a shorter execution_time_limit on the pipeline so stale builds cannot run for the full hour while you debug. A job that fails even though all commands passed is usually a coverage tool failing the suite on a minimum-coverage threshold.

Context: A Semaphore job hangs forever without finishing. The most common cause is a process that refuses to shut down.