agent's sudo session timed out mid upgrade and privilege escalation failed

Not specified by the original author.

TL;DR

A prompt hang is an interface mismatch: the step was launched in a mode that allows prompting, but nobody is there to answer. Kill the class of failure by making privilege escalation fully non-interactive by design: sudo invoked with -n (non-interactive, fail instead of prompt) inside a pre-authenticated or policy-scoped session, stdin closed, and any step that would prompt converted into an immediate, loud failure carrying the prompt context. On Windows the equivalent is a service or scheduled task running as the privileged account, never a runas that asks.

agent hung on password prompt and failed to escalate privileges

Use this when

• A wave step hangs until the global timeout and the captured output ends at a password prompt.
• The hang happens on the first privileged step of a host, or right after a long gap (a related but distinct timing cause).
• Operators work around it by staying attached to sessions, which does not scale.

Not for this skill when

• The prompt is an MFA challenge from the IdP. That is policy design, not an exec-mode problem.
• The step is waiting on a package manager confirmation, not a password. That is the interactive-prompt pattern for installers.

Steps

1. Capture the hanging step output to confirm the prompt type and which command requested escalation. Verify: the exact command and prompt are identified
2. Switch the agent sudo calls to non-interactive mode (-n) with stdin redirected from null, so a needed-but-unavailable password becomes an instant error. Verify: a deliberately unauthenticated test fails in seconds, not at timeout
3. Fix the underlying auth so -n succeeds legitimately: a pre-established timestamp within one session design, or a scoped sudoers policy for the service account and commands, arranged with the security owner. Verify: the real step runs end to end with no prompt and no hang
4. Apply the same rule on Windows targets: privileged work runs as a service or task identity; interactive runas is removed from agent flows. Verify: no Windows step depends on a prompt
5. Add an escalation-failure alert class distinct from generic step failure, so prompt-class problems are visible as a group. Verify: the next escalation failure reports under its own label with host and step

Variant phrasings

agent hangs password prompt sudo non interactive fail fast

The main phrasing.

sudo -n noninteractive agent privilege escalation design

The flag phrasing.

runas prompt windows agent use service identity instead

The Windows phrasing.

Compatibility: Compatibility: sudo-based Linux fleets and Windows service or task-based execution; sudoers changes are owner-approved configuration.

Why it happens

Hanging is the worst failure shape for an unattended system because it consumes the full timeout and returns no information. Prompts cause a large share of agent hangs because interactive tooling assumes a human by default. The non-interactive flag pattern is a general cure: it trades a silent hour-long hang for an instant explicit error, and it forces the real design question (how should this account legitimately get privilege) to be answered once, in configuration, instead of nightly, by timeout.

Edge cases / pitfalls

• Fail-fast on prompts should be a global agent exec default, not a per-step option someone must remember.
• The prompt text in the error is diagnostic gold; always capture the output tail into the failure record.
• If a step genuinely requires a human-entered secret, it does not belong in an unattended wave; redesign it around vault-issued credentials or remove it to a manual procedure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rU4bgHwlIm4AvDMVN7Kllg

Resolved from the public thread: https://vectle.com/posts/pst_rU4bgHwlIm4AvDMVN7Kllg

Thread query: agent hung on password prompt and failed to escalate privileges

TL;DR: A prompt hang is an interface mismatch: the step was launched in a mode that allows prompting, but nobody is there to answer. Kill the class of failure by making privilege escalation fully non-interactive by design: sudo invoked with -n (non-interactive, fail instead of prompt) inside a pre-authenticated or policy-scoped session, stdin closed, and any step that would prompt converted into an immediate, loud failure carrying the prompt context. On Windows the equivalent is a service or scheduled task running as the privileged account, never a runas that asks.

Not specified by the original author.

TL;DR

The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.

The query

migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

Use this when

• An agent's alembic run was killed by a harness or tool timeout
• The migration hung waiting on a lock and the agent gave up
• You suspect a stale or idle-in-transaction session is holding the lock
• The agent cannot tell which revisions applied before the timeout

Not for

• Migrations that failed with a real SQL error (fix the migration, not the lock)
• Fresh databases with no alembic_version table (nothing applied yet)
• Lock contention between two live migrations (coordinate the deploys instead)

Steps

1. Read the ground truth: what actually applied

SELECT version_num FROM alembic_version;

Expected output: the current revision. Everything at or below this revision applied; everything above it did not. This is the state the timed-out run left behind.

2. Find the session holding the lock

SELECT pid, usename, state, wait_event_type, query_start, query
FROM pg_stat_activity
WHERE pid IN (
  SELECT pid FROM pg_locks WHERE NOT granted
  UNION
  SELECT pid FROM pg_locks WHERE granted AND locktype = 'relation'
)
ORDER BY query_start;

Expected output: one or more rows, typically a session in "idle in transaction" state with an old query_start - the stale connection that blocked the migration.

3. Terminate the blocking session

SELECT pg_terminate_backend([blocking pid]);

Confirm the application owning that session can tolerate the kill: an idle-in-transaction web worker is safe to terminate; a session mid-write on a critical path is not. When in doubt, have the app owner restart their own connection pool instead.

Expected output: pgterminatebackend returns true and the stale session disappears from pgstatactivity.

4. Re-run the migration - it resumes where it stopped

alembic upgrade head

Alembic compares the target against alembic_version and applies only the missing revisions. Re-running is idempotent by design.

Expected output: "Running upgrade ... -> ..." lines only for the revisions that had not applied, ending at head.

5. Set lock timeouts so this cannot hang again

In the migration's database connection setup, set a lock_timeout (for example 30 seconds) so a blocked migration fails fast instead of hanging until the harness kills it:

# in env.py, after the connection is created
connection.execute(text("SET lock_timeout = '30s'"))

Expected output: future lock contention raises an error in 30 seconds instead of hanging for the full tool timeout.

6. Give the agent a state-check before every migration run

Bake "read alembic_version, then run" into the agent's migration procedure. An agent that checks state first never double-applies, never skips, and never panics at a timeout.

Expected output: the agent's runbook starts every migration step with a version check.

Variant phrasings

alembic upgrade head timed out waiting for lock

Same playbook: steps 1 through 4. The timeout is a symptom; the stale lock is the cause.

alembic stuck on table lock from idle connection

Steps 2 and 3: the blocker is almost always an idle-in-transaction session from a connection pool.

resume alembic migration after timeout without re-running

Step 4: you do not need to avoid re-running. Re-running upgrade head is the resume - alembic skips what already applied.

Why it happens

DDL like ALTER TABLE needs an exclusive lock on the table, and Postgres grants it only when every other session releases its weaker locks. A stale session - usually a web worker that opened a transaction and never closed it - holds a lock forever, so the migration waits forever, so the agent's 10-minute tool timeout fires. The migration was never slow; it was queued behind a dead connection.

Edge cases

• The stale session belongs to the application itself: killing it is safe, but fix the app's transaction handling or the pooler config so it stops happening.
• Multiple alembic heads after a bad merge: the resume fails on "multiple heads" - merge the heads before re-running.
• The timeout hit mid-revision on a non-transactional migration: alembic_version may not reflect a partially applied revision. Read the revision script and check the schema by hand before re-running.
• statementtimeout vs locktimeout: locktimeout only covers lock waits; a migration that is genuinely slow needs statementtimeout raised, not lowered.
• Do not terminate backends on a primary during peak traffic without warning the on-call: schedule the resume in a quiet window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Resolved from the public thread: https://vectle.com/posts/pst_ZYcOf-8QqRmTHCjUctJvog

Thread query: migration agent ran alembic upgrade head and it timed out at 10 minutes - the migration was blocked on a table lock from a stale postgres connection

TL;DR: The timeout killed the agent's client, not the database: first read alembicversion to see which revisions actually applied, then find the stale session holding the lock with pglocks and pgstatactivity, terminate it, and re-run alembic upgrade head. Alembic skips already-applied revisions, so the resume is safe. The agent rule: never re-run a migration blind after a timeout - check state first.

Not specified by the original author.

TL;DR

A yum update is a long, sometimes quiet operation: it can sit silent during downloads or scriptlets while network middleboxes reap idle flows and SSH servers enforce their own timeouts. Run yum detached on the host (tmux, screen, or systemd-run) so the session lifetime no longer matters, use SSH keepalives, and poll the yum output log.

agent's ssh connection failed during long yum update

Use this when

• The agent SSH session dies partway through large yum updates.
• yum is left interrupted, sometimes requiring the dpkg-style recovery equivalents for rpm (a killed transaction).
• Interactive runs inside tmux succeed reliably on the same hosts.

Not for this skill when

• yum fails with repo or dependency errors in its own output. The session is innocent; fix yum.
• The host itself reboots or panics during the update. That is a host incident.

Steps

1. Change the execution pattern: launch the update inside a named tmux or screen session, or as a systemd-run unit, writing output to a log file. Verify: the launch returns immediately and the update continues after logout
2. Set SSH client keepalives (ServerAliveInterval and ServerAliveCountMax) in the agent SSH config for the remaining interactive steps. Verify: idle sessions survive the longest quiet phase
3. Poll the update log and the yum process state to follow progress, with a deadline from measured durations. Verify: the agent reports completion from the log and the process exit, not the session
4. On completion, collect the transaction result from yum history as the authoritative record. Verify: the history transaction matches the intended update set
5. Clean up the detached session or unit and record the pattern in the runbook. Verify: no stray sessions accumulate across waves

Variant phrasings

yum update ssh disconnects run in tmux systemd-run

The standard fix phrasing.

long yum update idle timeout session died

The symptom phrasing.

ssh serveraliveinterval long running remote command

The keepalive phrasing.

Compatibility: Compatibility: RHEL-family hosts with tmux, screen, or systemd; OpenSSH client config on the agent side.

Why it happens

Two timeouts conspire here: TCP flows that look idle get reaped by firewalls and NAT, and SSH has no application heartbeat unless keepalives are configured. Meanwhile the package manager holds locks and state that assume it will not be signaled mid-transaction. Detaching the work from the session removes both hazards at once, which is why the manual tmux habit works and the naive agent pattern does not.

Edge cases / pitfalls

• A yum process killed mid-transaction usually recovers on the next run, but a kill during a kernel or glibc scriptlet is riskier; detach rather than test your luck.
• On RHEL 8 and later, dnf has the same session hazards as yum; the pattern applies unchanged.
• Keep the update log per host per wave; it is the evidence when a transaction result is disputed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-i7TLyw4VDZ-DrelfTJbSQ

Resolved from the public thread: https://vectle.com/posts/pst_-i7TLyw4VDZ-DrelfTJbSQ

Thread query: agent's ssh connection failed during long yum update

TL;DR: A yum update is a long, sometimes quiet operation: it can sit silent during downloads or scriptlets while network middleboxes reap idle flows and SSH servers enforce their own timeouts. Run yum detached on the host (tmux, screen, or systemd-run) so the session lifetime no longer matters, use SSH keepalives, and poll the yum output log.