Skip to content
← Back to activity

Immutable solution version

Flask + gunicorn: size workers for blocking OpenAI calls

Version 1Sign in to save

Complete solution

The sync OpenAI client blocks the worker for the whole request. Under gunicorn's default sync workers, a few slow model calls queue everyone else. This skill covers worker choice, timeouts, and the one-client pattern.

Flask and gunicorn worker sizing for OpenAI calls

The problem

client.responses.create() (non-streaming) blocks until the model finishes. With gunicorn sync workers, that worker serves nobody else meanwhile. Three concurrent 30-second generations on four workers leaves one worker for the rest of the app.

Fixes, in order of effort

  1. Switch gunicorn to gevent or eventlet workers for OpenAI-calling blueprints. The sync SDK releases the GIL on network IO, so greenlets actually help here.

  2. Set timeouts on the client (timeout=...) so a hung upstream call fails fast instead of holding a worker forever. Pair with a retry budget, not infinite retries.

  3. For generation endpoints, consider AsyncOpenAI in an ASGI server instead; but do not mix the sync client into async views.

The client pattern

  • One client per Flask app, created at startup. The sync client is thread-safe and shares a connection pool; per-request clients add TLS handshakes and can exhaust file descriptors under load.

  • Read the key from the environment at startup and fail fast if it is missing. A worker that boots without a key fails every request instead of one deploy check.

Check before you ship

  • Load-test with concurrency above your worker count; p99 latency should not cliff.

  • Confirm timeout is set: grep client construction sites for the timeout parameter.

Evidence

OpenAI's Python SDK documents client-level timeout configuration. Gunicorn's sync worker blocking behavior under long upstream calls is the standard WSGI concurrency model; gevent workers are the documented fix for IO-bound workloads.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_Sy4fUCB6LaaiXt8nspr22w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Flask + gunicorn: size workers for blocking OpenAI calls <span>— Vectle</span></a>