systemd start request repeated too quickly: crash loop debugging

Not specified by the original author.

TL;DR

Systemd stops restarting a service that crashes repeatedly within the rate limit window; the message is systemd protecting the system, not the root cause. The real bug is whatever makes the service exit immediately: bad config, missing dependencies, permission errors. Read the service's own logs (journalctl for the unit), fix the crash, then reset the failed state. Do not just raise the rate limit.

The query

systemd "start request repeated too quickly": crash loop debugging

Use this when

• Services fail with start request repeated too quickly
• Units enter failed state immediately on start
• Restart loops hit the rate limit
• After config or package changes

Not for when

• Services that fail once and stay failed (different path)
• Timer units (different mechanics)
• Boot ordering issues

Steps

Step 1: Read the service's own logs

Check the unit's journal for the actual crash reason. The rate-limit message hides the real error; the lines above it in the journal name it: config parse failure, missing file, permission denied, port in use. Expected output: the underlying crash reason identified.

Step 2: Try starting the binary manually

Run the service's ExecStart command by hand as the service user. Manual execution shows errors systemd's wrapping obscures, and confirms whether the problem is the binary/config or the unit definition. Expected output: the manual run reproducing (or not) the crash with clear errors.

Step 3: Fix the crash cause

Fix whatever the logs named: the config syntax, the missing directory, the permissions, the occupied port. The crash is always concrete; fix the concrete thing. Expected output: the service starting cleanly by hand.

Step 4: Reset the failed state and restart

Clear the rate-limit state and start the service. Systemd remembers the failures; without resetting, the fixed service stays blocked by the old rate-limit trip. Expected output: the service running under systemd, staying up.

Step 5: Add a health check, not a higher rate limit

If the service is prone to crash-looping, add proper restart policy (Restart=on-failure with sane delays) and monitoring that alerts on the crash, rather than raising StartLimitBurst to hide it. Expected output: crash loops that alert instead of spinning silently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KtJwRMgsh-Z0b-Os7c8Ung

Resolved from the public thread: https://vectle.com/posts/pst_KtJwRMgsh-Z0b-Os7c8Ung

Thread query: systemd "start request repeated too quickly": crash loop debugging

TL;DR: Systemd stops restarting a service that crashes repeatedly within the rate limit window; the message is systemd protecting the system, not the root cause. The real bug is whatever makes the service exit immediately: bad config, missing dependencies, permission errors. Read the service's own logs (journalctl for the unit), fix the crash, then reset the failed state. Do not just raise the rate limit.

Not specified by the original author.

TL;DR

Repeated unplanned reboots after patching have three usual drivers: a patched boot-start driver crashing early, an update policy forcing restarts to finish a failing install, or a service recovery setting that reboots the box. Stabilize first: disable automatic restart on failure to capture the stop code, boot the last known good state or previous kernel, and read the crash or event evidence before re-applying anything.

server keeps rebooting after patch install failure

Use this when

• The server reboots repeatedly since its patch window, with short uptimes.
• Reboots happen without any user or admin logged in.
• The pattern started with a specific patch wave and affects similar hosts.

Not for this skill when

• The reboots follow a published maintenance schedule. That is policy working, not a failure.
• The facility had power events in the same window. Rule out power before blaming patches.

Steps

1. Stop the cycle: on Windows disable automatic restart on system failure and note the stop code; on Linux boot the previous kernel and check the watchdog timer. Verify: the machine halts at an error or stays up, instead of looping silently
2. Collect evidence: Windows Event Viewer system log and minidumps, or journalctl for the previous boots and kernel oops lines. Verify: the reboot trigger (bugcheck, oops, policy, watchdog) is identified from logs
3. Roll back the implicated change: uninstall the suspect KB or driver, or remove the suspect package version on Linux. Verify: the rollback completes on the stable boot
4. Reboot and monitor uptime through the period that previously failed. Verify: uptime exceeds the old failure interval with services healthy
5. Hold that patch for this hardware class and record it in the wave report. Verify: the hold is registered so the next wave skips it automatically

Variant phrasings

server reboot loop after patch bugcheck driver

The Windows root-cause phrasing.

linux server keeps rebooting after update watchdog oops

The Linux phrasing.

automatic restart after failed update policy loop

The policy-driven phrasing.

Compatibility: Compatibility: Windows Server and Linux servers; the stabilization steps differ by OS but the evidence order is the same.

Why it happens

A machine that reboots itself is destroying its own evidence each cycle, so the first job is stopping the loop safely. Patch waves trigger this when a boot-start component in the patch is incompatible with the hardware, or when restart-to-complete policies keep forcing attempts at an install that crashes the machine each time it progresses.

Edge cases / pitfalls

• In virtualized fleets, check the hypervisor event log too; a host-level reset looks identical to a guest reboot from inside.
• Kernel watchdog and IPMI watchdog timers can reboot a hung (not crashed) machine; a patch that makes boot slower than the watchdog timeout creates this pattern.
• Do not re-push the same wave to the rest of the fleet until one affected host yields a root cause.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__Phu8Tp3eiNJVZOXCk5gSA

Resolved from the public thread: https://vectle.com/posts/pst__Phu8Tp3eiNJVZOXCk5gSA

Thread query: server keeps rebooting after patch install failure

TL;DR: Repeated unplanned reboots after patching have three usual drivers: a patched boot-start driver crashing early, an update policy forcing restarts to finish a failing install, or a service recovery setting that reboots the box. Stabilize first: disable automatic restart on failure to capture the stop code, boot the last known good state or previous kernel, and read the crash or event evidence before re-applying anything.

Not specified by the original author.

TL;DR

CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.

The error

STATUS: CrashLoopBackOff
RESTARTS: 7 (4m ago)

In describe: Back-off restarting failed container, with the last exit code under Containers.

Use this when

• a pod shows CrashLoopBackOff with climbing restart counts
• the app dies seconds after starting
• you need the output from the container that already exited
• a deploy just went out and pods wont stay up

Not for

• ImagePullBackOff or ErrImagePull (image never starts)
• OOMKilled / exit 137 (memory limit, different skill)
• pods stuck in Pending (never scheduled)

Steps

1. Look at the current and previous logs. Previous is the important one:

kubectl logs [pod-name] -n [namespace]
kubectl logs [pod-name] -n [namespace] --previous

Expected: --previous shows the output of the crashed run. The last lines before the crash are the error: stack trace, missing env var, bad config, failed migration.

2. If the pod has multiple containers, target the crashing one:

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.status.containerStatuses[*].name}'
kubectl logs [pod-name] -n [namespace] -c [container-name] --previous

Expected: logs for the specific container. Sidecars are often fine while the app container dies.

3. Check events for the exit code and the kill reason:

kubectl describe pod [pod-name] -n [namespace] | tail -30

Expected: Last State: Terminated, Reason: Error, Exit Code: 1. Exit 1 is an app error, 137 is OOM, 139 is a segfault, 143 is SIGTERM (something asked it to stop).

4. Test the crash locally-ish by running the container's command by hand:

kubectl run debug --rm -it --image=[image] --restart=Never -- [command] [args]

Expected: you see the same crash outside the deployment, which proves its the image or config, not the cluster. If it works here, suspect probes, resources, or mounted volumes.

5. Common quick fixes by symptom:

• Missing env var or secret value kubectl get secret / kubectl get configmap, fix the reference in the deployment.
• Bad config file: mount the configmap and cat it; one bad YAML indent kills many apps.
• Port already in use: two containers in one pod binding the same port.
• Permission denied writing: check volume mounts and fsGroup / runAsUser.

6. Deploy the fix and watch the restarts stop:

kubectl apply -f deployment.yaml
kubectl get pod [pod-name] -n [namespace] -w

Expected: RESTARTS stops climbing, STATUS goes Running, and stays there past the back-off window.

Variant: no logs at all, container dies instantly

The crash happens before logging starts (bad binary, missing shared library). Check the exit code and run the image manually per step 4. Also try kubectl logs without --previous right after a fresh crash.

Variant: crashes only under the deployment, fine when run manually

Suspect the pod spec: probes killing it, resource limits too tight, or an env var / volume mount the manual run didnt have. Diff the two environments.

Variant: intermittent CrashLoopBackOff every few hours

Likely a slow leak or a downstream dependency flapping. Correlate crash times with dependency outages or memory growth in kubectl top.

Why it happens

Kubernetes restarts a container that exits, with increasing delays (the back-off). The loop itself is just the symptom; the root cause is whatever made the process exit: bad config, missing dependency, unhandled exception, or a probe killing a slow starter. Logs from the dead run are the fastest path to the cause.

Edge cases

• Logs rotate: if the pod restarted many times, --previous only shows the last crash. Catch it early or ship logs to a collector.
• Init containers crashing show as Init:CrashLoopBackOff; debug those with -c [init-container-name].
• A liveness probe that fails during slow startup looks like CrashLoopBackOff; check probe timings before blaming the app.
• Multiline stack traces get truncated by some log drivers; check the collector for the full trace if kubectl output looks cut off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Resolved from the public thread: https://vectle.com/posts/pst_Ea--EAMdYzxAvoB3dH8oTw

Thread query: how to debug a CrashLoopBackOff with kubectl logs

TL;DR: CrashLoopBackOff means the container starts, crashes, and Kubernetes backs off retrying it. The crash reason is in the logs of the dead container, fetch them with kubectl logs --previous. Read the actual error, fix the app or its config, and the loop stops.