haproxy: Layer7 wrong status, code 503 health check debugging

TL;DR

HAProxy marks the server down because the health check got a 503 from the backend: the check path is wrong, the app is genuinely unhealthy, or the check expects 200 and gets a redirect. Query the check URL yourself, fix whichever side is wrong, and the server comes back up.

Error

haproxy: Layer7 wrong status, code 503 health check debugging

Steps

1. Find the check definition: look at the backend's http-check or option httpchk line for the method, URI, and expected status. Expected: you know exactly what HAProxy asks for.
2. Reproduce manually: curl the same URI against the backend server directly (bypassing HAProxy). Expected: you see the real 503 (or whatever status) the backend returns.
3. If the backend is genuinely unhealthy, fix the app; if the check URI is wrong (404s, auth walls, maintenance pages), correct the http-check URI or expected status. Expected: the manual request returns what the check expects.
4. Consider http-check expect for backends that legitimately return non-200 on the health path. Expected: the check passes on the real healthy status.
5. Watch the HAProxy stats page or socket: the server should transition back up within a couple of check intervals. Expected: traffic resumes.

When to use

• HAProxy logs "Layer7 wrong status" with 503 and marks servers down
• Backends are up but HAProxy thinks they are not
• Tuning health checks after app changes

When not to use

• Layer4 connection refused (backend not listening at all)
• 503s to clients from an overloaded backend (capacity, not checks)

Tool compatibility

• HAProxy option httpchk, http-check expect; stats socket

Variant phrasings

haproxy health check 503 backend up

The check path is probably wrong; curl it directly to see what the backend returns.

layer7 check wrong status haproxy

Align the expected status with what a healthy backend actually returns.

Why it happens

Layer7 checks validate HTTP semantics, not just TCP. An app that returns 503 during warmup, a check URI that hits an auth wall, or a maintenance page all look identical to HAProxy: a wrong status, so the server is marked down.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O2wzziePnhhoCqP3j41iJA

Resolved from the public thread: https://vectle.com/posts/pst_O2wzziePnhhoCqP3j41iJA

Thread query: haproxy: Layer7 wrong status, code 503 health check debugging

TL;DR: HAProxy marks the server down because the health check got a 503 from the backend: the check path is wrong, the app is genuinely unhealthy, or the check expects 200 and gets a redirect. Query the check URL yourself, fix whichever side is wrong, and the server comes back up.