pytest "sqlalchemy.exc.OperationalError: database is locked" in parallel CI runs: how to fix

SQLite lets one writer in at a time, so parallel xdist workers queue up and time out on the lock. Either give every worker its own database file, or switch CI to a real database server. A single shared SQLite file under parallel writes will always do this.

pytest "sqlalchemy.exc.OperationalError: database is locked" in parallel CI runs: how to fix

Steps

1. Confirm the setup: check that the test database is SQLite and that all xdist workers share a single database file path. Serial runs passing while parallel runs lock is the signature.

   Expected: You can state the database backend and prove every worker uses the same file.

2. Pick the fix that fits the suite: per-worker database files (each worker gets its own SQLite file, usually keyed by worker id) or a server database in CI (Postgres or MySQL via a service container).

   Expected: You have a chosen strategy and a reason: isolation versus realism.

3. For per-worker files, make the test database path include the xdist worker id so workers never share a file. Each worker migrates and seeds its own copy.

   Expected: Workers create separate database files and the lock errors disappear.

4. For the server-database route, point CI at the service container and keep SQLite for local runs. Make sure the CI job waits for the database to be ready before pytest starts.

   Expected: The parallel run passes against the shared server database.

5. As a complement either way, enable WAL mode and set a busy timeout on SQLite connections so brief contention waits instead of failing instantly.

   Expected: Short lock contention retries instead of raising, and the suite is green under -n4 and up.

Use this when

• database is locked errors appear only under xdist
• parallel workers share one SQLite file
• CI runs tests in parallel against SQLite

Not for this skill when

• the lock happens in serial runs too (look for an uncommitted transaction)
• the database is Postgres or MySQL (locks there mean something else)
• a single test holds a write lock across the whole suite (fix the test)

Variant phrasings

• sqlalchemy database is locked parallel pytest
• OperationalError database is locked xdist sqlite
• pytest parallel database is locked how to fix

Why it happens

SQLite's default journal mode allows a single writer: when one connection holds the write lock, every other writer blocks until the busy timeout expires, then raises OperationalError. One pytest process writing to SQLite is fine. Four xdist workers hammering the same file turn every write into a lock race, and the losers raise. The database is not broken and the tests are not wrong; the architecture assumes a concurrency level SQLite was never given.

Edge cases

• Per-worker files multiply disk usage and migration time; for huge suites the server database is cheaper.
• In-memory SQLite per worker is the fastest option but it cannot be inspected after a failure; use file-based when you need to debug.
• If only a few tests write, an alternative is running the writers serially in one worker and the readers in parallel.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpBrPpSus4Dr0HC1csT6Nw