duckdb "IO Error: Could not open file" s3 credentials chain

TL;DR

DuckDB cannot resolve S3 access for the path you gave it. Fix it by loading the httpfs extension, setting the region explicitly, and providing access through environment variables or a named secret instead of relying on implicit credential resolution.

IO Error: Could not open file

When to use

• S3 reads fail from a new machine, container, or CI runner
• The same query works locally but not in production
• The bucket region is ambiguous or cross-region

When not to use

• The object genuinely does not exist (check the key spelling)
• The bucket name is wrong (thats a 404-style problem, not access)
• Local file reads fail (thats a filesystem path problem)

Steps

1. Load the httpfs extension, which provides S3 support:

INSTALL httpfs;
LOAD httpfs;

Expected output: both statements succeed. Without httpfs, S3 paths are treated as local paths and the open fails.

2. Set the region explicitly. Implicit region detection fails often, especially with new buckets:

import os
os.environ["AWS_REGION"] = "us-east-1"

Expected output: requests route to the right region. A wrong region produces access-looking errors on objects that exist.

3. Provide access through environment variables set before DuckDB starts:

export AWS_ACCESS_KEY_ID=[YOUR-KEY]
export AWS_SECRET_ACCESS_KEY=[YOUR-SECRET]

Expected output: DuckDB picks up the standard chain automatically. Setting them after the connection is open often has no effect.

4. Or declare a named secret inside DuckDB for full control:

CREATE SECRET my_s3 (
    TYPE S3,
    REGION 'us-east-1'
);

Expected output: the secret is registered and S3 reads use it. This beats env vars when one process talks to buckets under different identities.

Variant phrasings

Could not open file with s3:// path but the https scheme works

The httpfs settings differ per scheme. Set region and access explicitly rather than relying on per-scheme defaults.

IO Error only on some objects in the bucket

Those objects may be in a different region or under a key prefix with different permissions. Check per-object, not per-bucket.

Why it happens

DuckDB resolves S3 access through a chain: explicit secrets, then environment variables, then instance metadata, then anonymous. On a laptop the chain usually lands on a working entry; in containers and CI it silently falls through to anonymous access, and the resulting denial surfaces as a generic IO error rather than an auth error.

Edge cases

• Instance metadata credentials expire. Long-running processes can start fine and fail hours later when the temporary access lapses.
• Requester-pays buckets need an explicit opt-in setting; otherwise even valid access gets denied.
• Object keys with special characters need URL encoding. The file exists but the unencoded path 404s.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJMrY8HnxngYSBfY_6vCuw