duckdb s3 parquet read with secrets

TL;DR

DuckDB reads S3 Parquet directly with read_parquet('s3://bucket/path/*.parquet'), but it needs AWS credentials, which you provide with CREATE SECRET using the s3 type and the right region. Never paste keys into the SQL string: CREATE SECRET keeps them out of query history and logs, and lets agents share read-only credentials safely.

The query

duckdb s3 parquet read with secrets

Use this when

• DuckDB fails to read from S3 with an access or credential error
• you are setting up a pipeline or agent that reads Parquet from S3
• credentials keep leaking into query logs and you want them out

Not for

• reading local Parquet files, which need no secrets at all
• writing to S3 or other stores like GCS and Azure, which use different secret types

Steps

1. Create the secret with the s3 type, scoped to your region: CREATE SECRET (TYPE s3, KEY_ID '...', SECRET '...', REGION 'us-east-1'). Use a read-only IAM role where possible.

Expected output: CREATE SECRET succeeds and subsequent S3 reads authenticate.

1. Verify with a cheap listing-style query, like reading one small file's schema, before scanning the whole dataset.

Expected output: The schema comes back without an auth error.

1. Load credentials from environment config at runtime instead of hardcoding them in the SQL file.

Expected output: No credential value appears in any checked-in file or query log.

1. Scope the secret narrowly: a dedicated IAM user or role with GetObject on only the buckets the job needs.

Expected output: The credential has the least privilege that still lets the job run.

1. Use glob patterns (s3://bucket/prefix/*.parquet) to read partitioned datasets as one table.

Expected output: One read_parquet call returns the full partitioned dataset.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DGpGprrg4SSz2j6xA7Z8-w