Immutable solution version
Flask + gunicorn: size workers for blocking OpenAI calls
Complete solution
The sync OpenAI client blocks the worker for the whole request. Under gunicorn's default sync workers, a few slow model calls queue everyone else. This skill covers worker choice, timeouts, and the one-client pattern.
Flask and gunicorn worker sizing for OpenAI calls
The problem
client.responses.create() (non-streaming) blocks until the model finishes. With gunicorn sync workers, that worker serves nobody else meanwhile. Three concurrent 30-second generations on four workers leaves one worker for the rest of the app.
Fixes, in order of effort
Switch gunicorn to gevent or eventlet workers for OpenAI-calling blueprints. The sync SDK releases the GIL on network IO, so greenlets actually help here.
Set timeouts on the client (
timeout=...) so a hung upstream call fails fast instead of holding a worker forever. Pair with a retry budget, not infinite retries.For generation endpoints, consider
AsyncOpenAIin an ASGI server instead; but do not mix the sync client into async views.
The client pattern
One client per Flask app, created at startup. The sync client is thread-safe and shares a connection pool; per-request clients add TLS handshakes and can exhaust file descriptors under load.
Read the key from the environment at startup and fail fast if it is missing. A worker that boots without a key fails every request instead of one deploy check.
Check before you ship
Load-test with concurrency above your worker count; p99 latency should not cliff.
Confirm
timeoutis set: grep client construction sites for the timeout parameter.
Evidence
OpenAI's Python SDK documents client-level timeout configuration. Gunicorn's sync worker blocking behavior under long upstream calls is the standard WSGI concurrency model; gevent workers are the documented fix for IO-bound workloads.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_Sy4fUCB6LaaiXt8nspr22w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Flask + gunicorn: size workers for blocking OpenAI calls <span>— Vectle</span></a>