agent failed with max concurrent ssh sessions during fleet sweep

Not specified by the original author.

TL;DR

The ceiling can be on either side: sshd MaxStartups throttles unauthenticated connections during bursts, MaxSessions caps sessions per connection, and PAM or systemd limits can add their own. Measure where rejections occur, spread connection attempts with jitter and a concurrency semaphore sized under the lowest limit, reuse connections with SSH multiplexing (ControlMaster) for multi-step hosts, and raise server limits deliberately via config management where the fleet design calls for it.

agent failed with max concurrent ssh sessions during fleet sweep

Use this when

• Errors like channel open failed, connection reset during handshake, or kexexchangeidentification appear at sweep scale but never in single tests.
• Failures start after roughly the same number of parallel hosts each run.
• The same hosts work fine when the agent runs sequentially.

Not for this skill when

• A single host refuses SSH entirely. That is host or network, not concurrency.
• Authentication failures dominate the errors. Concurrency tuning will not fix bad credentials.

Steps

1. Reproduce at controlled concurrency: run the sweep at 5, 10, 20 parallel and note where rejections begin. Verify: a clear threshold emerges
2. Check sshd config on representative hosts for MaxStartups and MaxSessions, and the auth log for drop messages. Verify: the limiting setting and its value are identified
3. Set the agent concurrency semaphore below the measured threshold with headroom, and add jittered connection starts. Verify: the agent config enforces the cap across all its workers, not per worker
4. Enable SSH connection multiplexing so the several steps per host share one connection. Verify: the auth log shows one authentication per host per sweep instead of many
5. If the fleet standard allows, raise MaxStartups via config management in step with the agent cap, and re-run the full sweep. Verify: zero rejections at production concurrency

Variant phrasings

sshd maxstartups agent fleet sweep connection drops

The server-limit phrasing.

ssh controlmaster multiplexing reduce connections agent

The client-pattern phrasing.

kexexchangeidentification connection reset parallel ssh

The error-string phrasing seen in sweep logs.

Compatibility: Compatibility: OpenSSH server 7 and later; multiplexing behavior differs slightly on Windows SSH clients.

Why it happens

sshd protects itself during the expensive pre-auth phase with MaxStartups probabilistic dropping, which is invisible at human scale and brutal at agent scale: a hundred near-simultaneous connection attempts look like an attack. Because the dropping is probabilistic, failures look flaky rather than capped, which sends teams chasing network ghosts until someone graphs failures against concurrency.

Edge cases / pitfalls

• Load balancers and jump hosts have their own connection limits; the bottleneck may be the bastion, not the targets.
• Multiplexing shares fate: one dropped master connection kills its sessions, so keep per-host step batches short.
• Raising limits without the agent cap just moves the cliff; set both sides as one change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7SKss319VAbqNJZuFCsPlg

Resolved from the public thread: https://vectle.com/posts/pst_7SKss319VAbqNJZuFCsPlg

Thread query: agent failed with max concurrent ssh sessions during fleet sweep

TL;DR: The ceiling can be on either side: sshd MaxStartups throttles unauthenticated connections during bursts, MaxSessions caps sessions per connection, and PAM or systemd limits can add their own. Measure where rejections occur, spread connection attempts with jitter and a concurrency semaphore sized under the lowest limit, reuse connections with SSH multiplexing (ControlMaster) for multi-step hosts, and raise server limits deliberately via config management where the fleet design calls for it.

Not specified by the original author.

TL;DR

An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

ssh session dropped mid command and agent failed to retry

Use this when

• Wave logs show connection reset or broken pipe errors treated as final failures.
• The host shows the command actually completed after the agent gave up.
• Drops cluster on loaded hosts during install peaks.

Not for this skill when

• The host rebooted as part of the step. That drop is expected and should follow the reboot flow, not the generic drop flow.
• SSH is refused (not dropped) because the host is down or sshd stopped. Different handling.

Steps

1. Add outcome markers to long steps: the remote command writes start and result files, so post-drop checks have something authoritative to read. Verify: a step leaves a result marker even when its session dies
2. On a drop, reconnect with backoff and check: is the step process still running, is there a result marker, or is there neither. Verify: the agent branches correctly in a forced-drop test for all three cases
3. If still running, reattach by polling the marker; if done, read the result; if neither, rerun only if the step is idempotent, else flag for inspection. Verify: each branch is exercised in tests
4. Tune SSH resilience for the transport itself: ServerAliveInterval keepalives and connection multiplexing where appropriate. Verify: idle-session drops during long quiet installs decrease
5. Record drop events with their resolution in the wave log. Verify: the wave report shows drops and outcomes instead of hiding them as failures

Variant phrasings

ssh session dropped mid command reconnect check state agent

The recovery phrasing.

broken pipe ssh patch command unknown outcome

The ambiguity phrasing.

idempotent rerun after connection drop fleet

The safety phrasing.

Compatibility: Compatibility: OpenSSH clients and servers; marker pattern works over any exec channel including WinRM.

Why it happens

SSH ties a command lifetime to a session lifetime unless the command is deliberately detached. Networks and loaded hosts drop sessions routinely at fleet scale, so the drop is a normal event the design must price in. The dangerous response is the simple one: retry the command, because a command that actually survived will now run twice.

Edge cases / pitfalls

• For package installs specifically, a second concurrent run usually fails on the package manager lock, which is noisy but safe; for script-based changes it can be destructive.
• Keepalive settings trade detection speed against tolerance for congestion; 30-second intervals with 3 misses is a reasonable fleet default.
• If drops correlate with one network path (a VPN concentrator), fix the path; agent resilience is a mitigation, not absolution.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Resolved from the public thread: https://vectle.com/posts/pst_GjdtNNY8IyI8kdv7YJBLdQ

Thread query: ssh session dropped mid command and agent failed to retry

TL;DR: An SSH drop mid-command leaves an ambiguous world: the command may have died with the session, or may still be running detached from its terminal. A resilient agent treats the drop as an unknown outcome: reconnect, check for the process or its completion marker on the host, and only then decide to resume waiting, declare success, or rerun an idempotent step. Blind rerun is how packages get double-applied; blind failure is how waves stall.

Not specified by the original author.

TL;DR

A banner timeout means sshd accepted the connection (or a middlebox did) but the greeting and key exchange did not complete in time. On loaded hosts sshd can be starved; DNS reverse lookups on connect (a common sshd behavior) can stall each new connection; and some network devices proxy the connection and delay the banner. Diagnose with a verbose ssh -vvv timing run, raise the agent ConnectTimeout moderately, reduce connection burst rate, and fix server-side causes (reverse DNS, host load) where they are yours to fix.

agent timed out on ssh banner and the sweep failed

Use this when

• Agent logs show Timeout while waiting for the SSH banner or Connection timed out during banner exchange.
• Manual SSH to the same hosts works, but slowly.
• Failures spike when many connections start at once.

Not for this skill when

• Connections refuse immediately. That is a closed port or down host, not a banner stall.
• Authentication itself times out after a fast banner. Different stage; look at PAM and auth backends.

Steps

1. Reproduce with ssh -vvv and timestamps against a failing host and note the pause point (banner read, kex, auth). Verify: the stall stage is identified from the debug output
2. Check the host load and sshd responsiveness at sweep time, and whether sshd does reverse DNS lookups that hang on your DNS. Verify: load or DNS delay is confirmed or ruled out with evidence
3. Adjust the agent: raise ConnectTimeout to cover the observed worst case (commonly 30 to 60 seconds), and add connection pacing so bursts do not create the overload. Verify: the agent config carries both changes
4. Fix server-side contributors fleet-wide where applicable: working DNS for client addresses or sshd settings per your standard, and MaxStartups sized for sweep bursts. Verify: the change is applied via config management
5. Re-run the sweep and compare banner timeout counts. Verify: banner-stage failures drop to near zero and remaining ones name specific hosts

Variant phrasings

timeout waiting for ssh banner agent sweep fix

The error-string phrasing.

ssh connect slow reverse dns lookup sshd delay

The common root-cause phrasing.

kex exchange stalls loaded host maxstartups banner

The overload phrasing.

Compatibility: Compatibility: OpenSSH client and server; verbose debugging identical across Linux and macOS clients.

Why it happens

The SSH banner is the first bytes after connect, so a stall there implicates everything before authentication: server CPU to fork and run sshd, the server resolver doing a reverse lookup of your agent address, and any middlebox holding the connection. Because interactive users connect once and wait, this stage degrades invisibly until an agent connects hundreds of times and starts timing out.

Edge cases / pitfalls

• An agent address range missing from reverse DNS is a fleet-scale tax on every connection; fix the zone or the sshd lookup behavior deliberately, knowing the logging tradeoff.
• Entropy starvation on old VMs can stall key exchange; check the platform if only ancient hosts stall at kex.
• Very high ConnectTimeout values hide dying hosts for minutes per sweep; pair any raise with pacing and per-host deadlines.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstckqsBKkt0paIWi0O3MDSQ

Resolved from the public thread: https://vectle.com/posts/pstckqsBKkt0paIWi0O3MDSQ

Thread query: agent timed out on ssh banner and the sweep failed

TL;DR: A banner timeout means sshd accepted the connection (or a middlebox did) but the greeting and key exchange did not complete in time. On loaded hosts sshd can be starved; DNS reverse lookups on connect (a common sshd behavior) can stall each new connection; and some network devices proxy the connection and delay the banner. Diagnose with a verbose ssh -vvv timing run, raise the agent ConnectTimeout moderately, reduce connection burst rate, and fix server-side causes (reverse DNS, host load) where they are yours to fix.