agent's column rename broke three microservices it didnt know were reading the old name
Recovery when a column rename broke microservices that were still reading the old name: restoring compatibility immediately, then doing the rename properly with dual-write and phased reader migration. Use when a rename broke unknown readers. Not for renames with no live readers.
TL;DR
The rename was atomic in the database but not in the system: services you did not know about still query the old name. Restore compatibility first (add the old name back as an alias or reverse the rename), then inventory every reader, migrate them in phases with both names present, and only drop the old name when no reader uses it. Renames are a coordination problem, not a DDL problem.
The query
agent's column rename broke three microservices it didnt know were reading the old nameUse this when
- A column rename broke microservices reading the old name
- Unknown readers broke after a schema rename
- Need a safe column-rename procedure across services
- An agent renamed a column without inventorying readers
Not for
- Renaming a column no live code reads (just rename it)
- Renaming with all readers already migrated (drop the old name)
- Application-level renames with no database involved (a code refactor)
Steps
1. Restore compatibility immediately
Add the old name back. On postgres, a view or a generated column aliasing the new name; simplest is often to reverse the rename and start over:
ALTER TABLE [table] RENAME COLUMN [new_name] TO [old_name];Expected output: the broken services recover as soon as the old name resolves again. This is triage: the rename attempt is abandoned, not fixed forward.
2. Inventory every reader of the column
Search every service's code, dashboards, ETL jobs, and ad-hoc queries for the old column name. Include the ones you think do not exist: the incident proves your inventory was incomplete.
Expected output: a written list of every reader, owned by a team or a person. "Unknown" is not an entry: find them.
3. Dual-write / dual-read phase
Rename properly this time: add the new column, write both columns from the writers (or backfill the new from the old and keep them in sync with a trigger), and migrate readers one by one to the new name. Keep the old name working throughout.
Expected output: every reader on the list reads the new name, with the old name still present as a safety net.
4. Verify zero old-name traffic, then drop
Monitor query logs for the old column name until traffic is zero for a full cycle, then drop it:
ALTER TABLE [table] DROP COLUMN [old_name];Expected output: the old name is gone and nothing breaks, because nothing was using it.
5. Make reader-inventory mandatory for renames
The agent's rename procedure starts with the inventory step and cannot proceed with an incomplete list. Renames without a reader list are rejected at review.
Expected output: the next rename is a phased project with named owners, not a single DDL statement.
Tool compatibility
postgres, mysql, and any relational database. The phased-rename discipline is database-agnostic.
Variant phrasings
column rename broke microservices old name
Step 1: restore the old name first. Compatibility now, proper rename later.
safe column rename across microservices
Steps 2 through 4: inventory readers, dual-phase, verify zero traffic, drop.
rename column unknown readers
Step 2 is the lesson: the incident means the reader inventory was incomplete. Rebuild it from logs, not memory.
Why it happens
A database rename is instantaneous and total, but the system around it changes on deploy schedules: services deploy independently, dashboards are edited by hand, ETL runs on cron. The rename has to be compatible with the slowest reader, not the fastest writer. Expand/contract (both names live, readers migrate, old name dies last) is the only pattern that respects that.
Edge cases
- Views aliasing the old name can hide performance cliffs (the view may not use indexes the same way): prefer the real column during the transition, views only as emergency triage.
- ORMs cache column metadata: a service can break on the rename even after its code is updated, until it restarts and refreshes metadata.
- Some renames are better as "add new, deprecate old" permanently: if the old name is in external contracts, never drop it.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_txPHJ4JZCugLH6FKv9tLKQ