haproxy "backend has no server available": health check debugging

TL;DR

No server available means every backend server is marked down by health checks (or none are configured). The servers might genuinely be down, or the health checks might be wrong: checking the wrong port, path, or protocol, or marked down by a too-strict rise/fall threshold. Check the stats page first, then verify what the health check actually tests.

The query

haproxy "backend has no server available": health check debugging

Use this when

• HAProxy logs or stats show no server available
• All backend servers marked down simultaneously
• Health checks seem misconfigured
• After backend or HAProxy changes

Not for when

• Frontend binding or ACL problems
• SSL termination issues
• Request routing logic

Steps

Step 1: Check the stats page for server states

Open the HAProxy stats page to see each server's state and the check results. All-down simultaneously suggests a check or network problem; one-down suggests a server problem. Expected output: per-server states with check details.

Step 2: Manually run the health check

Reproduce the health check by hand from the HAProxy host: same protocol, port, path, and expected response. If your manual check fails, the servers are really unhealthy; if it passes, the check configuration is wrong. Expected output: the check's verdict validated independently.

Step 3: Verify check port, path, and protocol

Compare the configured check against what the servers actually serve. Common misses: checking HTTP on an HTTPS port, a health path that was renamed, or expecting 200 from an endpoint that returns 204. Expected output: the check aligned with the servers' real health endpoints.

Step 4: Review rise/fall thresholds

Overly strict thresholds (fall 1, or aggressive intervals) mark servers down on single blips. Tune so transient failures do not drain the whole backend: a few consecutive failures before marking down is the sane default. Expected output: thresholds that tolerate blips but catch real outages.

Step 5: Check the network path from HAProxy to backends

If manual checks from your laptop pass but HAProxy's fail, the HAProxy-to-backend network path is the problem: firewalls, security groups, or routing. Test from the HAProxy host itself, not from yours. Expected output: the network path proven or identified as broken.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEPMq1oaXw8K7Doqb40GUg