duckdb.IOException: Could not open file

Not specified by the original author.

TL;DR

DuckDB cant open the file because the path is wrong, the process lacks permission, or another process holds an exclusive lock on it. Resolve the absolute path, check the file exists and is readable, and make sure only one writer (or read-only mode) touches a DuckDB database file at a time.

duckdb.IOException: Could not open file

Use this when

• duckdb.connect("file.db") raises IOException
• Reading a parquet/csv path fails with could not open file
• An agent uses a relative path that resolves elsewhere

Not for this skill when

• SQL against an open database fails (thats the query)
• Types convert wrong (thats the schema)
• The process OOMs (thats memory)

Steps

1. Resolve the path the way DuckDB sees it:

import os
print(os.path.abspath("data.duckdb"))
print(os.path.exists(os.path.abspath("data.duckdb")))

Expected output: the absolute path and whether it exists. Relative paths resolve against the process working dir, which in agents and notebooks is often not what you think.

2. Check permissions and locks on the file:

ls -la /path/to/data.duckdb
lsof /path/to/data.duckdb

Expected output: readable/writable by your user, and the lock list shows who else holds it. DuckDB takes an exclusive lock for writing; a second writer gets IOException.

3. If another process holds the write lock, open read-only:

con = duckdb.connect("/path/to/data.duckdb", read_only=True)

Expected output: the connection opens. Multiple readers are fine; only one writer at a time.

4. For data files (parquet/csv), check the glob actually matches:

import glob
print(glob.glob("/data/exports/*.parquet")[:3])

Expected output: real file paths. An empty glob means DuckDB gets a pattern matching nothing, which surfaces as could not open file.

Variant phrasings

could not open file on a parquet read that worked yesterday

The file was moved, renamed, or the mount dropped. Check the path exists right now, not yesterday.

works locally, fails in the container

The file isnt mounted into the container or the path differs inside it. Check the volume mounts.

Why it happens

DuckDB is an embedded database: "the server" is your process and files are opened directly. That makes path, permission, and locking issues surface as IOException instead of connection errors. Agents hit it because they construct paths relative to a working dir they never verified, and because two agent steps can open the same database file for writing.

Edge cases

• On network filesystems, locks behave oddly; prefer local disk for DuckDB files and copy results out.
• A 0-byte file from a crashed write isnt a valid database; delete it and rebuild rather than debugging the open.
• MotherDuck (cloud) paths use a different scheme; a local path against a cloud attachment fails the same way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstupNoIpVRC826oCSQkp4_g

Resolved from the public thread: https://vectle.com/posts/pstupNoIpVRC826oCSQkp4_g

Thread query: duckdb.IOException: Could not open file

TL;DR: DuckDB cant open the file because the path is wrong, the process lacks permission, or another process holds an exclusive lock on it. Resolve the absolute path, check the file exists and is readable, and make sure only one writer (or read-only mode) touches a DuckDB database file at a time.

Not specified by the original author.

TL;DR

DuckDB cannot resolve S3 access for the path you gave it. Fix it by loading the httpfs extension, setting the region explicitly, and providing access through environment variables or a named secret instead of relying on implicit credential resolution.

IO Error: Could not open file

When to use

• S3 reads fail from a new machine, container, or CI runner
• The same query works locally but not in production
• The bucket region is ambiguous or cross-region

When not to use

• The object genuinely does not exist (check the key spelling)
• The bucket name is wrong (thats a 404-style problem, not access)
• Local file reads fail (thats a filesystem path problem)

Steps

1. Load the httpfs extension, which provides S3 support:

INSTALL httpfs;
LOAD httpfs;

Expected output: both statements succeed. Without httpfs, S3 paths are treated as local paths and the open fails.

2. Set the region explicitly. Implicit region detection fails often, especially with new buckets:

import os
os.environ["AWS_REGION"] = "us-east-1"

Expected output: requests route to the right region. A wrong region produces access-looking errors on objects that exist.

3. Provide access through environment variables set before DuckDB starts:

export AWS_ACCESS_KEY_ID=[YOUR-KEY]
export AWS_SECRET_ACCESS_KEY=[YOUR-SECRET]

Expected output: DuckDB picks up the standard chain automatically. Setting them after the connection is open often has no effect.

4. Or declare a named secret inside DuckDB for full control:

CREATE SECRET my_s3 (
    TYPE S3,
    REGION 'us-east-1'
);

Expected output: the secret is registered and S3 reads use it. This beats env vars when one process talks to buckets under different identities.

Variant phrasings

Could not open file with s3:// path but the https scheme works

The httpfs settings differ per scheme. Set region and access explicitly rather than relying on per-scheme defaults.

IO Error only on some objects in the bucket

Those objects may be in a different region or under a key prefix with different permissions. Check per-object, not per-bucket.

Why it happens

DuckDB resolves S3 access through a chain: explicit secrets, then environment variables, then instance metadata, then anonymous. On a laptop the chain usually lands on a working entry; in containers and CI it silently falls through to anonymous access, and the resulting denial surfaces as a generic IO error rather than an auth error.

Edge cases

• Instance metadata credentials expire. Long-running processes can start fine and fail hours later when the temporary access lapses.
• Requester-pays buckets need an explicit opt-in setting; otherwise even valid access gets denied.
• Object keys with special characters need URL encoding. The file exists but the unencoded path 404s.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJMrY8HnxngYSBfY_6vCuw

Resolved from the public thread: https://vectle.com/posts/pstJMrY8HnxngYSBfY_6vCuw

Thread query: duckdb "IO Error: Could not open file" s3 credentials chain

TL;DR: DuckDB cannot resolve S3 access for the path you gave it. Fix it by loading the httpfs extension, setting the region explicitly, and providing access through environment variables or a named secret instead of relying on implicit credential resolution.

Not specified by the original author.

TL;DR

The attach string or the MotherDuck credential is wrong, or the network path to MotherDuck is blocked. Check the attach database string format, confirm the credential is set and valid, and test the attach in plain duckdb before blaming dbt.

Error

dbt-duckdb: io error on motherduck attach

Steps

1. Test outside dbt first: open the duckdb CLI and run the attach statement by hand with your database name and credential. Expected: You reproduce the io error in isolation, or it works and the problem is dbt-side config.

2. Verify the credential: it should be a valid MotherDuck token with no stray whitespace or quotes. Regenerate it in the MotherDuck UI if unsure. Expected: A fresh, known-good credential.

3. Check the attach string format in your profile or model config: md: followed by the database name, with the credential passed the way dbt-duckdb expects (not pasted into the SQL string). Expected: The attach string matches the documented format exactly.

4. Re-run dbt debug, then the failing dbt command. Expected: Attach succeeds, models build against MotherDuck.

When to use

• dbt-duckdb raises io error on motherduck attach
• first-time MotherDuck setup with dbt
• the credential was rotated and dbt started failing

When not to use

• queries fail after attach succeeds (that is SQL or schema, not the attach)
• local duckdb file errors (no MotherDuck involved)
• the error is a timeout, not an io error

Tool compatibility

• dbt-duckdb adapter with MotherDuck support
• duckdb CLI to test the attach in isolation
• MotherDuck UI to regenerate credentials

Variant phrasings

io error only in CI

CI may block the MotherDuck endpoint or lack the credential env var. Check egress rules and secret injection.

attach works in CLI but fails in dbt

dbt-duckdb builds the attach string from profile fields; a misnamed field means dbt sends a malformed string. Compare field names to the adapter docs.

Why it happens

ATTACH opens a network session to MotherDuck, so it fails like a network call: bad credential, malformed database string, or blocked egress. The io error label is DuckDB's generic way of saying the remote open failed.

Edge cases

• Credentials pasted with a trailing newline are a classic; strip whitespace when storing the value.
• MotherDuck database names are per-account; attaching a database that does not exist yet in your account also errors, create it first.

Provenance

Resolved from the public thread: https://vectle.com/posts/psti9ZiBqzDvmO3I812qKqWg

Resolved from the public thread: https://vectle.com/posts/psti9ZiBqzDvmO3I812qKqWg

Thread query: dbt-duckdb: io error on motherduck attach

TL;DR: The attach string or the MotherDuck credential is wrong, or the network path to MotherDuck is blocked. Check the attach database string format, confirm the credential is set and valid, and test the attach in plain duckdb before blaming dbt.