S3 / object storage¶

Whenever source.location starts with s3://, DataPress needs a [dataset.s3] block (or environment-provided credentials) to talk to the bucket. The same shape works for AWS, MinIO, Cloudflare R2, Backblaze B2, Wasabi, and any other S3-compatible service.

Reference¶

Field	Default	Notes
`region`	`us-east-1`	Falls back to `AWS_REGION` env, then `us-east-1`.
`endpoint`	(unset)	Custom S3 endpoint (MinIO, R2, Wasabi, Backblaze, …).
`addressing_style`	`virtual`	`virtual` = `https://bucket.host`, `path` = `https://host/bucket` (MinIO).
`allow_http`	`false`	Must be `true` if `endpoint` is `http://...`.
`partitioning`	`auto`	Hive partition discovery: `auto` (detect from path), `hive` (force on), `none` (force off).
`endpoint_bucket_in_host`	`auto`	Whether to fold the bucket into the endpoint host: `auto` (follows `addressing_style`), `true`, `false`.
`access_key_id`, `secret_access_key`, `session_token`	(unset)	Inline creds. Discouraged for prod — use env vars instead.

AWS S3 — default credential chain¶

Most production deployments use IAM roles or env vars. DataPress reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, and AWS_REGION automatically when not set inline.

[[dataset]]
name = "logs"

[dataset.source]
kind     = "parquet"
location = "s3://my-bucket/logs/2024/"

[dataset.s3]
region = "eu-west-1"

MinIO / R2 / Wasabi / Backblaze¶

Non-AWS providers usually need a custom endpoint. MinIO additionally needs addressing_style = "path". Plain-HTTP endpoints require allow_http = true.

[[dataset]]
name = "warehouse"

[dataset.source]
kind     = "parquet"
location = "s3://warehouse/exports/"

[dataset.s3]
region           = "us-east-1"
endpoint         = "http://minio.local:9000"
addressing_style = "path"
allow_http       = true

Delta on S3¶

Same shape as parquet on S3 — just flip the kind.

[[dataset]]
name = "events_delta"

[dataset.source]
kind     = "delta"
location = "s3://my-bucket/events_delta/"

[dataset.s3]
region = "us-east-1"

Partitioning & endpoint host alignment¶

The DuckDB and DataFusion backends historically required slightly different S3 configs. These two options let you align them (or override the defaults) so the same location works on either backend:

partitioning controls Hive partition discovery (e.g. year=2024/month=01/…). With auto (default) each backend infers partition columns from the path layout. Set hive to force partition columns on, or none to disable them.
endpoint_bucket_in_host controls whether the bucket name is folded into the endpoint hostname. With auto (default) this follows addressing_style (virtual → bucket in host, path → not). Use true / false to override when a provider needs a specific form.

With these defaults a plain prefix such as s3://my-bucket/logs/ behaves identically on both backends: DataPress lists the prefix recursively and discovers Hive partitions automatically — you no longer need to hand-write *.parquet globs for DuckDB or put the bucket in the endpoint host for DataFusion.

Inline credentials (discouraged)¶

[dataset.s3]
region            = "us-east-1"
access_key_id     = "AKIA..."
secret_access_key = "..."
# session_token   = "..."   # optional, for STS creds

Avoid checking these into version control.

Per-dataset env vars¶

For multi-tenant setups, scope credentials to one dataset by prefixing the standard AWS env-var names with ${DATASET_NAME_UPPERCASE}_. Non-alphanumeric chars become _.

For a dataset named sales.eu-1 (prefix → SALES_EU_1):

export SALES_EU_1_AWS_ACCESS_KEY_ID=AKIA...
export SALES_EU_1_AWS_SECRET_ACCESS_KEY=...

Credential precedence¶

Highest → lowest:

Per-dataset env vars: ${PREFIX}_AWS_ACCESS_KEY_ID, ${PREFIX}_AWS_SECRET_ACCESS_KEY, ${PREFIX}_AWS_SESSION_TOKEN, ${PREFIX}_AWS_REGION.
Inline [dataset.s3] keys.
Plain AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION.
The backend's default credential chain (~/.aws/credentials, IMDS, …).

Python: dynamic credentials provider

The Python S3Config binding accepts a credentials_provider — a zero-argument callable returning an HMACKeyPair. It runs once when DataPress(...) is constructed, is cached indefinitely, and overrides any inline access_key_id / secret_access_key. See Python › Configuration.