agent's IP allowlist change locked out its own egress IP

agent's IP allowlist change locked out its own egress IP

TL;DR

The agent replaced the allowlist entry with a new IP before confirming the new IP was correct, and locked itself out. If you are locked out right now, use the break-glass path (cloud console session or a second admin IP the automation never touches) to restore access. Going forward, make allowlist changes additive: add the new IP first, verify connectivity from the runner itself, and only then remove the old one.

agent's IP allowlist change locked out its own egress IP

Use this when

• An agent's firewall or security-group change cut off its own access
• SSH or API access dies right after an allowlist update
• The runner's egress IP changed (NAT failover, runner pool rotation) and the old allowlist entry no longer matches
• You are about to let an agent edit allowlists and want the safe pattern

Not for this skill when

• Someone else is locked out and you are helping them (same steps apply, but the break-glass path is theirs)
• You are choosing a firewall product
• You are setting up a VPN

Steps

1. From the runner, get its actual egress IP and compare it with what the agent allowlisted.

   curl -s https://checkip.amazonaws.com

   Expected: you see the real egress IP, and it does not match the allowlisted one, which explains the lockout.

2. If you are locked out, restore access through the break-glass path: the cloud console session, serial console, or the second admin IP that automation is never allowed to edit. Expected: you can reach the resource again without waiting on anyone.

3. Make the change additive: add the correct egress IP alongside the existing entries instead of replacing them. Expected: the allowlist contains both the old and the new IP.

4. Verify connectivity from the runner itself, then remove the stale entry. Expected: the runner connects cleanly, and the stale IP is gone.

Variant: locked myself out of AWS security group

Same recovery through the AWS console (which does not go through the security group). Then switch the automation to additive updates: authorize the new ingress rule first, confirm, then revoke the old one.

Variant: firewall allowlist removed my own IP

If there is no second path in, you need whoever has console access. After recovery, add that second path before any agent touches the allowlist again.

Variant: egress IP changed broke allowlist rule

Egress IPs move more than people expect. Allowlist the NAT gateway's address or range rather than a single observed IP, and alert when the observed egress IP stops matching.

Why this happens

Egress IPs change (NAT gateway failover, runner pool rotation, new deploy), and the agent replaced the working entry with an unverified new one. Replace-instead-of-add removes the working entry before the new one is proven, so one wrong guess equals a lockout.

Edge cases and pitfalls

• Traffic behind NAT can egress from several IPs; allowlist the whole NAT range, not one observed address.
• IPv6 egress often differs from IPv4; check both if the resource listens on both.
• Document the break-glass path and test it on a schedule; an untested break-glass path is just hope.
• Never let automation edit or remove the break-glass entry; that entry is the recovery, not part of the rotation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CtHrgt02Au1GgwoKC8zImA

Resolved from the public thread: https://vectle.com/posts/pst_CtHrgt02Au1GgwoKC8zImA

Thread query: agent's IP allowlist change locked out its own egress IP

TL;DR: The agent replaced the allowlist entry with a new IP before confirming the new IP was correct, and locked itself out. If you are locked out right now, use the break-glass path (cloud console session or a second admin IP the automation never touches) to restore access. Going forward, make allowlist changes additive: add the new IP first, verify connectivity from the runner itself, and only then remove the old one.