Approval workflow immutable submitted revision optimistic concurrency transaction outbox notifications publish

TL;DR: never edit a submitted revision in place. Split the design into three parts: (1) the workflow record, which is mutable and holds only status and the current revision number, (2) revision rows, which are immutable content snapshots that are inserted once at submit and never updated, and (3) outbox rows, which hold the notifications to publish. Do the submit in one EF Core transaction - insert the frozen revision, move the workflow status with an optimistic concurrency check on a rowversion value, and insert the outbox rows - and let a background publisher send notifications only after that transaction commits. An approver then always reads and approves the exact revision id captured at submit, so later edits create a new revision instead of changing the snapshot under review.

Verbatim problem (the search that created the source thread):

Approval workflow immutable submitted revision optimistic concurrency transaction outbox notifications published snapshot remains unchanged ASP.NET Core EF Core

Symptoms this fixes: the author edits a document after submitting it and the approver unknowingly approves different content, two editors save at once and the second save silently overwrites the first, a notification goes out for a submission whose database transaction later rolled back, the same approval notification is delivered twice, or code throws DbUpdateConcurrencyException and the request retries by overwriting the newer data.

Steps

1. Split workflow state from revision content

Model two entities, not one editable document row.

• ApprovalRequest: Id, Status (Draft, Submitted, Approved, Rejected), CurrentRevisionNumber, SubmittedRevisionId, and a concurrency value (step 3). Only status fields and pointers change on this row.
• ApprovalRevision: Id, RequestId, RevisionNumber, a snapshot of every content field that approval covers (or one snapshot payload column holding the serialized content), ContentHash, CreatedBy, CreatedAt, and SubmittedAt. Once a revision row is submitted it has no update path: give its content properties private setters, create it in one factory method, and never attach a modified copy later.

Store the content hash at submit time (for example a SHA-256 hash over the canonical snapshot fields). The hash is the cheap proof in step 6 that the snapshot an approver saw is the snapshot that was approved.

Expected output: editing after submit is impossible through the normal model because there is nothing to edit - the edit screen loads the latest revision, copies its values into a new draft revision, and saves that as a new row. Success check: the database shows revision 1 unchanged after an edit, and a new revision 2 row holding the edited values.

2. Enforce immutability in SaveChanges, not just in the UI

UI rules get bypassed by background jobs, imports and future endpoints. Add a guard in the DbContext: override SaveChanges and SaveChangesAsync, inspect the change tracker for ApprovalRevision entries in the Modified or Deleted state whose original SubmittedAt value is already set, and throw an InvalidOperationException before anything is written. New revision inserts and changes to ApprovalRequest stay allowed.

Also block the bypass routes in code review: bulk update and delete calls and raw SQL must never target submitted revision rows, because they skip the change tracker entirely. If the database supports it, add a trigger or a restricted grant as a second layer, but keep the SaveChanges guard as the portable rule every code path shares.

Expected output: a test that loads a submitted revision, changes a content field and saves throws the guard exception, and the stored row is byte for byte unchanged afterwards. Success check: the same test against a draft revision that was never submitted either inserts a new revision or is rejected by your draft flow - it never updates the submitted row.

3. Add optimistic concurrency to the workflow row

On SQL Server, add a byte array property named RowVersion, configure it as a row version in OnModelCreating (IsRowVersion), and let the database generate a new value on every update. On PostgreSQL or SQLite, use an unsigned integer or a Guid Version property configured as a concurrency token, and set a fresh value yourself on every save of the workflow row. Either way the rule is the same: every update carries the value the caller originally read, and the update only succeeds when the stored value still matches.

Catch DbUpdateConcurrencyException around SaveChanges on submit, approve and reject. On a conflict, do not merge and do not retry by overwriting: reload the request, return a conflict response (HTTP 409 in a web API) and make the caller re-read the current status and revision before acting again. Last write wins is exactly the failure this design exists to prevent.

Expected output: two concurrent submits that both read version 1 produce one success and one DbUpdateConcurrencyException, surfaced as a conflict, with only one submitted revision and one set of outbox rows in the database. Success check: no test scenario ends with the second writer silently replacing the first writer data.

4. Submit in a single transaction, with the outbox write inside it

The submit operation is one unit of work on one DbContext, committed by one SaveChanges call, wrapped in an explicit transaction when the outbox or status logic spans more than one SaveChanges call:

1. Load the ApprovalRequest with its current concurrency value.
2. Insert the new ApprovalRevision row with SubmittedAt set and its ContentHash computed.
3. Update the request: Status becomes Submitted, SubmittedRevisionId points at the new revision, CurrentRevisionNumber is set to the new revision number.
4. Insert one OutboxMessage row per notification to publish (for example ApprovalSubmitted for the approver group), holding AggregateId (the request id), MessageType, a payload with RequestId, RevisionId, RevisionNumber and ContentHash, OccurredAt, and a null ProcessedAt.
5. Commit. If any part fails, including the concurrency check in step 3, everything rolls back together: no revision, no status change, no outbox row.

Never send email, push notifications or broker messages from inside this transaction. A send before commit is the classic dual write bug: the notification can escape for a submission that never committed, and a committed submission can lose its notification when the send fails after commit. The outbox row is the notification intent, stored atomically with the state it describes.

Expected output: after a successful submit the database holds, from one commit, the frozen revision, the updated status and the pending outbox rows. Success check: forcing a failure at the last write (for example a duplicate key in a test) leaves none of the three behind.

5. Publish notifications from the outbox in a separate background step

Run a background service that loops on a timer: read a small batch of outbox rows where ProcessedAt is null, ordered by OccurredAt, publish each one to the broker or notification service, and only then mark that row processed in its own short transaction. If publishing fails, leave ProcessedAt null, record the attempt count and the error, and let the next pass retry. Add a maximum attempt count with an alert, so a poison message does not loop forever unnoticed.

This delivery is at least once by design: a crash after publishing but before marking processed will publish the same message again on the next pass. That is correct and preferable to losing messages, and it puts the burden where step 6 handles it - on idempotent consumption.

Preserve order per aggregate when it matters: publish outbox rows for the same request in OccurredAt order and do not let a later message for that request overtake an earlier failed one.

Expected output: notifications appear shortly after commit, never before it, and a broker outage only delays them - the outbox rows wait in the database instead of being lost. Success check: stopping the publisher, submitting, then starting it delivers exactly the pending notifications for that submission.

6. Make consumption and approval idempotent, and pin approval to one revision

Every consumer dedupes on the outbox message Id: keep a processed messages table (or a unique index on the notification log) keyed by that Id, insert the key in the same transaction as the consumer effect, and skip the work when the key already exists. A duplicate delivery then produces no duplicate email and no duplicate side effect.

Approval itself records what was approved: the approve call takes the RevisionId the approver was shown, checks that it still equals SubmittedRevisionId on the request, recomputes or compares the ContentHash, and writes ApprovedRevisionId and ApprovedContentHash on the decision. If the author has since created or submitted a newer revision, the old approval does not carry over - the request goes back through review for the new revision. This is the guarantee behind snapshot remains unchanged: approval is a statement about one immutable revision, never about whatever the document happens to contain today.

Expected output: delivering the same outbox message twice sends one notification, and approving revision 1 after revision 2 was submitted fails the pin check instead of approving revision 2 by accident. Success check: the approval record and the submitted revision agree on both revision id and content hash.

7. Prove the whole flow with tests

Pin these cases before shipping:

1. Immutability: submit revision 1, edit, and confirm a new revision row appears while revision 1 fields and hash are unchanged when read back fresh from the database.
2. Concurrency: run two submits against the same original concurrency value; expect one commit and one conflict, one submitted revision, and one set of outbox rows.
3. Atomicity: inject a failure during submit and confirm no revision, no status change and no outbox row survive.
4. No early notification: assert no publish call happens while the submit transaction is still open, and that a rolled back submit publishes nothing ever.
5. Retry and dedupe: make the publisher crash after publishing once, let it redeliver, and confirm the consumer effect happened once.
6. Approval pinning: submit, capture the revision id and hash, create a newer draft, and confirm approving the captured revision still refers to the original snapshot, and that the newer draft is not marked approved.

Expected output: all six pass. Success check: removing the outbox and sending directly from the submit handler makes tests 3, 4 and 5 fail, which proves the outbox is carrying the guarantee.

When this applies

• Documents, requests, expenses, changes or content go through a submit then approve or reject flow, and approval must bind to the exact content that was submitted.
• More than one person or service can edit or act on the same request, so lost updates are a real risk.
• Notifications or downstream events must match committed state exactly - never sent for rolled back work, never lost after a commit.
• Audit or compliance needs a durable record of what was submitted, when, and what was approved.

When this does not apply

• A single editor, no approval step and no audit need: a plain status column and a concurrency token on one table is enough, and revisions plus an outbox are overhead.
• Pure event streaming with no relational state to keep atomic with the events: use the platform native log instead of a relational outbox table.
• Fire and forget notifications whose loss or duplication is harmless (for example a cosmetic badge): send them directly and skip the outbox.
• Long running human workflows spanning many systems with compensation steps: that is a saga or workflow engine problem; this pattern covers the submit and notify core inside one service boundary.

Variant: DbUpdateConcurrencyException on submit or approve

This exception means the concurrency value in the database no longer matches the value the caller loaded - someone else saved first. Reload the request, show the caller the current status and submitted revision, and let them decide again. Never catch it and resave the stale entity by overwriting the stored concurrency value; that converts a detected conflict into a silent lost update.

Variant: snapshot changed after submit

If approvers report seeing different content than the author submitted, the usual cause is an edit path that updates the submitted row in place, or a read path that loads the latest revision instead of SubmittedRevisionId. Fix the write side with steps 1 and 2, fix the read side by always loading the revision by the pinned id, and compare ContentHash on the approval screen during diagnosis - a hash mismatch proves the snapshot moved.

Variant: notification sent but the submission rolled back

That ordering only happens when the send sits inside or before the transaction commit (the dual write problem). Move the send out entirely: write the outbox row in step 4, publish in step 5. There is no safe version of sending first and compensating later, because the notification has already reached people.

Variant: duplicate approval notifications

Duplicates come from at least once outbox delivery meeting a consumer with no dedupe. Keep the delivery, add the consumer side key from step 6. Trying to get exactly once delivery from the publisher instead is the wrong lever - a crash between publish and mark processed makes duplicates unavoidable, so consumption must tolerate them.

Variant: approvals applying to the wrong version

The approval call accepted no revision identifier, so it approved whatever was current at click time. Require the RevisionId (and ideally the hash) the approver rendered, compare both against the pinned submitted revision in step 6, and return a conflict when they differ.

Why it happens

Three separate gaps combine into the usual approval bugs. First, one mutable row cannot be both the working draft and the evidence of what was submitted, so any in place edit rewrites history. Second, without a concurrency value checked on write, the database has no way to know a save was based on stale data, so it happily applies it. Third, a database commit and a message send are two writes with no shared transaction, so any code that does both directly will eventually do one without the other. Immutable revision rows close the first gap, optimistic concurrency closes the second, and writing the message into the same table and transaction as the state - the transactional outbox - closes the third.

Edge cases and pitfalls

• Provider differences for the concurrency value: SQL Server generates rowversion values itself and ignores values you set; PostgreSQL needs a token you maintain (for example the xmin system column or your own version column incremented on each save). Test the conflict path on the real provider, not only on an in memory one, which often skips concurrency checks.
• In memory and SQLite test providers can hide concurrency and transaction behaviour that differs in production. Keep at least one integration test suite on the production database engine.
• Outbox payload drift: the payload is a contract for consumers. Version it or keep it additive; changing a field meaning breaks consumers that replay old messages.
• Outbox growth: processed rows accumulate. Archive or delete processed rows on a schedule, after consumers no longer need them for dedupe replay, and index the publisher query on ProcessedAt and OccurredAt.
• Hash canonicalization: compute ContentHash over a stable field order and stable formatting (dates, decimals, casing), or the same content hashes differently and the pin check in step 6 false alarms.
• Clock and ordering: OccurredAt orders messages approximately, not perfectly across machines. When strict per aggregate order matters, add a per aggregate sequence number assigned inside the submit transaction.
• Status is not content: allow status transitions on the workflow row freely, but treat any code that copies new content values into an existing revision row as a defect, even when a product request asks for a quick correction - corrections are new revisions.