agent failed with max concurrent ssh sessions during fleet sweep

TL;DR

The ceiling can be on either side: sshd MaxStartups throttles unauthenticated connections during bursts, MaxSessions caps sessions per connection, and PAM or systemd limits can add their own. Measure where rejections occur, spread connection attempts with jitter and a concurrency semaphore sized under the lowest limit, reuse connections with SSH multiplexing (ControlMaster) for multi-step hosts, and raise server limits deliberately via config management where the fleet design calls for it.

agent failed with max concurrent ssh sessions during fleet sweep

Use this when

• Errors like channel open failed, connection reset during handshake, or kexexchangeidentification appear at sweep scale but never in single tests.
• Failures start after roughly the same number of parallel hosts each run.
• The same hosts work fine when the agent runs sequentially.

Not for this skill when

• A single host refuses SSH entirely. That is host or network, not concurrency.
• Authentication failures dominate the errors. Concurrency tuning will not fix bad credentials.

Steps

1. Reproduce at controlled concurrency: run the sweep at 5, 10, 20 parallel and note where rejections begin. Verify: a clear threshold emerges
2. Check sshd config on representative hosts for MaxStartups and MaxSessions, and the auth log for drop messages. Verify: the limiting setting and its value are identified
3. Set the agent concurrency semaphore below the measured threshold with headroom, and add jittered connection starts. Verify: the agent config enforces the cap across all its workers, not per worker
4. Enable SSH connection multiplexing so the several steps per host share one connection. Verify: the auth log shows one authentication per host per sweep instead of many
5. If the fleet standard allows, raise MaxStartups via config management in step with the agent cap, and re-run the full sweep. Verify: zero rejections at production concurrency

Variant phrasings

sshd maxstartups agent fleet sweep connection drops

The server-limit phrasing.

ssh controlmaster multiplexing reduce connections agent

The client-pattern phrasing.

kexexchangeidentification connection reset parallel ssh

The error-string phrasing seen in sweep logs.

Compatibility: Compatibility: OpenSSH server 7 and later; multiplexing behavior differs slightly on Windows SSH clients.

Why it happens

sshd protects itself during the expensive pre-auth phase with MaxStartups probabilistic dropping, which is invisible at human scale and brutal at agent scale: a hundred near-simultaneous connection attempts look like an attack. Because the dropping is probabilistic, failures look flaky rather than capped, which sends teams chasing network ghosts until someone graphs failures against concurrency.

Edge cases / pitfalls

• Load balancers and jump hosts have their own connection limits; the bottleneck may be the bastion, not the targets.
• Multiplexing shares fate: one dropped master connection kills its sessions, so keep per-host step batches short.
• Raising limits without the agent cap just moves the cliff; set both sides as one change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7SKss319VAbqNJZuFCsPlg