software update broke ssh access how to roll back

Not specified by the original author.

TL;DR

Rolling back an agent's infrastructure change is a race against the agent's next action: first stop the agent from making more changes, then restore the last known good state from version control, then verify. The preconditions that make this fast are versioned infrastructure code, a known-good state to return to, and the ability to revoke the agent's access instantly. Build those before you need them.

The query

how to roll back an agent's bad infrastructure change

Use this when

• An agent's infrastructure change caused an incident
• You need the previous infrastructure state restored quickly
• Designing safety for agent-driven infra changes
• The agent is still running and might change more

Not for when

• Application code rollbacks (deploy pipeline, different topic)
• Data recovery (backups, different topic)
• Planned infrastructure migrations

Steps

Step 1: Stop the agent first

Revoke or pause the agent's credentials before touching infrastructure. An agent still running can fight your rollback by reapplying its change. This is step one, not step three. Expected output: the agent cannot make further changes; confirmed via audit log silence.

Step 2: Identify exactly what changed

Use the command logs and the infrastructure diff: what did the agent modify, in what order. Version-controlled infrastructure makes this a git diff; unversioned changes make it archaeology. Know the full change set before reverting any of it. Expected output: the complete list of changed resources.

Step 3: Restore the last known good state

Revert the infrastructure code to the last good commit and apply, or restore from the state snapshot. Prefer the declarative path (revert code, apply) over manual console fixes; manual fixes during an incident create the next incident. Expected output: infrastructure converging back to the known-good state.

Step 4: Verify the restoration worked

Check the actual resource state, not just the apply output: the load balancer routes correctly, the security groups match, the service is healthy. Agents' changes sometimes have side effects (deleted resources, rotated credentials) that a simple revert does not restore. Expected output: verified healthy state, with side effects explicitly checked.

Step 5: Do the incident review with the agent's logs

Run the postmortem with the agent's full command history: what was it trying to do, where did its plan go wrong, what guardrail was missing. The fix is usually a missing constraint (no plan approval, no canary), not a bad agent. Expected output: a systemic fix (new gate, new check) rather than just "be more careful".

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DOlfh1wsoaH6CB-8jM-GiA

Resolved from the public thread: https://vectle.com/posts/pst_DOlfh1wsoaH6CB-8jM-GiA

Thread query: how to roll back an agent's bad infrastructure change

TL;DR: Rolling back an agent's infrastructure change is a race against the agent's next action: first stop the agent from making more changes, then restore the last known good state from version control, then verify. The preconditions that make this fast are versioned infrastructure code, a known-good state to return to, and the ability to revoke the agent's access instantly. Build those before you need them.

Not specified by the original author.

TL;DR

kubectl rollout undo rolls a deployment back to its previous ReplicaSet in one command, and it is safe because the old ReplicaSet is still there. Check the revision history first, undo (optionally to a specific revision), then watch the rollout and verify the app actually recovered. Keep revisionHistoryLimit above 2 or there will be nothing to roll back to.

Error / query

kubectl rollout undo: safe rollback steps

Use this skill when

• A new deploy broke the app and you need the old version back fast
• A rollout is stuck and backing out is safer than pushing forward
• You want to see what revision you would roll back to before doing it
• Validating that rollbacks actually work before you need one

Not for this skill when

• ArgoCD/Flux manages the deploy (roll back in git, not with kubectl)
• The bad change includes a database migration (rollback needs a data plan)
• You want to roll forward with a fix instead (often the better call)

Steps

Step 1: Check what revisions exist

kubectl rollout history deployment/[deployment] -n [namespace]

Expected: a numbered list of revisions with change causes. You need at least revision N-1 present; if history shows only the current revision, revisionHistoryLimit was too low and undo has nothing to restore.

Step 2: Preview the revision you would restore

kubectl rollout history deployment/[deployment] -n [namespace] --revision=[N-1]

Expected: the pod template of that revision (image, env, args). Confirm it is the known-good version before undoing; rolling back to another bad revision just wastes time.

Step 3: Undo the rollout

kubectl rollout undo deployment/[deployment] -n [namespace]
# or to a specific revision:
# kubectl rollout undo deployment/[deployment] -n [namespace] --to-revision=[N-1]

Expected: the deployment starts scaling up the old ReplicaSet and scaling down the new one. This is a normal rolling update in reverse, so readiness probes still gate traffic.

Step 4: Watch and verify recovery

kubectl rollout status deployment/[deployment] -n [namespace] --timeout=10m
kubectl get pods -n [namespace] -l app=[app]

Expected: rollout completes and pods are Running and Ready on the old image. Then verify the app symptom is actually gone (hit the health endpoint, check error rates); a completed rollout on the wrong revision is not a recovery.

Variant phrasings

"kubectl rollback deployment"

kubectl rollout undo is the command; rollback is not a kubectl verb.

"how to undo a kubernetes deployment"

Steps 1-4. Check history first so you know the target revision exists.

Why it happens

Deployments keep old ReplicaSets around (up to revisionHistoryLimit, default 10) precisely so a bad rollout can be reversed without rebuilding anything. Undo just changes which ReplicaSet is scaled up; no new images are pulled and no manifests are edited, which makes it the fastest recovery path for a bad deploy.

Edge cases and pitfalls

• Undo does not reverse database migrations or external state changes; rolling back code against a migrated schema can break worse.
• If the bad deploy changed the deployment's selector or labels, undo may not match old pods cleanly; check the revision diff first.
• HPA can fight a rollback by scaling the new ReplicaSet during the transition; consider pausing or scaling HPA first for big rollbacks.
• In GitOps setups, undo is temporary: the next sync re-applies the git state. Revert the git commit to make the rollback stick.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pHmjC7jnlbeuDgP52VmsaQ

Resolved from the public thread: https://vectle.com/posts/pst_pHmjC7jnlbeuDgP52VmsaQ

Thread query: kubectl rollout undo: safe rollback steps

TL;DR: kubectl rollout undo rolls a deployment back to its previous ReplicaSet in one command, and it is safe because the old ReplicaSet is still there. Check the revision history first, undo (optionally to a specific revision), then watch the rollout and verify the app actually recovered. Keep revisionHistoryLimit above 2 or there will be nothing to roll back to.

Not specified by the original author.

TL;DR

"It worked yesterday" means something changed, your job is to find what. Check deploys and config changes on your side first, then walk the user through what changed on theirs (browser updates, settings, data, permissions). Most of these tickets resolve to one of three causes: a recent release, a user-side environment change, or a data or permissions change. Say what changed and what you are doing about it, users accept regressions they understand.

The query

how to handle "it worked yesterday" tickets

Use this when

• A user reports a feature that broke overnight
• Multiple tickets arrive with the same "suddenly stopped working" phrasing
• You need a regression triage runbook for the team
• Agents guess instead of investigating on these tickets

Not for

• Scheduled downtime or maintenance windows
• New-feature onboarding confusion
• Proactive incident communication
• Performance degradation over time (different problem)

Steps

1. Take the report seriously, but verify the timeline

"When did you last see it working?" Pin down actual dates and times. "Yesterday" is often three weeks ago, which changes the search window.

Expected output: a concrete last-known-good timestamp in the ticket.

2. Check your side first: deploys, flags, config

Look at release notes, deploy logs, and feature-flag changes in the window between last-known-good and the report. This is the fastest path to the answer.

Expected output: either a suspect change (with deploy ID) or a confirmed "nothing shipped in that window."

3. Ask the user what changed on their side

Browser updates, OS updates, password changes, new team members, permission changes, different device. Ask conversationally, not as a checklist.

Expected output: a list of user-side changes in the same window, or an explicit "nothing changed" from the user.

4. Check data and permissions

Expired trials, changed plan limits, revoked API keys, deleted or archived records the feature depends on. Regressions caused by data are the sneakiest ones.

Expected output: account state, plan, and key permissions verified current.

5. Reproduce with the user's exact state

If you can reproduce it with their account, it is your bug. If you cannot reproduce it anywhere, it is their environment. Both are useful answers.

Expected output: a reproduction result tied to a specific account or environment.

6. Tell them what changed and what happens next

"We shipped a change on Tuesday that broke exports for files over 50MB. We are rolling it back now, you should be good within the hour." Transparency kills the frustration.

Expected output: the reply names the cause and the next action with a timeline.

Variant phrasings

"it used to work" support troubleshooting

Same playbook. The timeline question in step 1 matters even more here, because "used to" can mean anything.

how to troubleshoot a feature that suddenly stopped working

Same six steps, works for internal IT tickets too. The three cause buckets (your release, their environment, data or permissions) cover the vast majority.

regression triage checklist for support

Use steps 2 through 4 as your team's triage checklist before escalating. Escalations that arrive with "checked deploys, nothing shipped" get engineering attention faster.

Why it happens

These tickets are emotionally loaded because the user feels something was taken from them. The instinct is to argue ("are you sure it worked?") or to guess. Both make it worse. The delta-hunt approach works because software does not break spontaneously, it breaks from change, and change leaves a trail.

Edge cases

• Multiple users reporting at once: skip to step 2, it is almost certainly your side. Treat it as an incident, not a ticket.
• User insists nothing changed: believe them for now and dig into step 4, silent permission or plan changes are the usual culprit.
• Third-party dependency changed: check status pages and changelogs of integrations, then communicate it honestly ("our payment provider changed their API response on Tuesday").
• Actually a UX change: sometimes nothing broke, the UI just moved. Say so plainly and show them the new location, dont defend the redesign.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcfH1lW-ev-kGJ-mOiorkA

Resolved from the public thread: https://vectle.com/posts/pstcfH1lW-ev-kGJ-mOiorkA

Thread query: how to handle "it worked yesterday" tickets

TL;DR: "It worked yesterday" means something changed, your job is to find what. Check deploys and config changes on your side first, then walk the user through what changed on theirs (browser updates, settings, data, permissions). Most of these tickets resolve to one of three causes: a recent release, a user-side environment change, or a data or permissions change. Say what changed and what you are doing about it, users accept regressions they understand.