airflow connections via environment variables

TL;DR

Airflow reads connections from environment variables named AIRFLOWCONN[CONN_ID] with a URI value, which is the simplest way to inject connections in Docker, Kubernetes, or CI. It works with zero extra services, but the URI sits in the environment, so it is fine for dev and simple deploys and not a replacement for a real secrets backend in production.

The query

airflow connections via environment variables

Use this when

• Running Airflow in Docker Compose or Kubernetes and wiring connections at deploy time
• CI pipelines that need a test connection without UI clicks
• Local development where the metadata DB gets wiped often

Not for

• Production systems with strict secret rotation requirements
• Multi-tenant setups needing per-user connection scoping
• Storing long-lived privileged credentials in plain env files

Steps

1. Encode the connection as a URI. The format is scheme://host:port/schema, with extras as query params. URL-encode any special characters in the password.

Expected output: a URI string that captures host, credentials, and options in one value.

2. Export it with the naming convention, uppercasing the conn id and replacing dashes with underscores:

export AIRFLOW_CONN_MY_POSTGRES='postgresql://dbhost:5432/analytics'

Expected output: the variable is set in the scheduler and worker environments.

3. Verify Airflow sees it by listing connections or testing in a Python shell. Env-var connections override the metadata DB, so a stale DB entry with the same id will be shadowed.

Expected output: the connection resolves and a test task using my_postgres connects successfully.

4. For anything beyond dev, move to a secrets backend (HashiCorp Vault, AWS Secrets Manager) and keep env vars only for non-secret defaults. Never commit real URIs to git.

Expected output: production reads secrets from the backend; env files contain no real credentials.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OrgeZWn8JFIHpGi8gaMjVw