Skip to content

Configuration

Every DataPress instance reads a single TOML file at startup. By convention it's called datasets.toml; override with the DATASETS_CONFIG environment variable.

It has one [server] block and one [[dataset]] block per table you want to expose.

[server]                  # optional; defaults shown below
backend = "datafusion"    # or "duckdb"
listen  = "127.0.0.1"
port    = 8080

[[dataset]]               # one block per dataset
name = "..."
# source, s3, index, lazy follow

Pages

  • Server settings — listen, port, workers, prefix, compression, body limits, timeouts, graceful shutdown.
  • Datasets — source, lazy, parquet vs delta, local files, directories, globs.
  • S3 / object storage — credentials, endpoints, addressing styles, per-dataset env overrides.
  • Indexing — DataFusion equality-index policy.
  • Explorer UI — the built-in dataset browser, API query tab, and DuckDB-WASM terminal, with an optional OIDC "Authorize" button ([explorer] / [explorer.oauth2]).
  • Swagger UI — embedded interactive API docs and optional OIDC "Authorize" button ([swagger] / [swagger.oauth2]).
  • Prometheus metrics — the /metrics scrape endpoint ([metrics]).
  • Documentation site — enabling the embedded MkDocs site.
  • Authentication — OIDC / OAuth2 bearer validation and scope-based authorization ([auth]).

Optional feature blocks

A few features are opt-in and configured in their own block:

  • [explorer] — the dataset explorer UI. Enabled by default when the explorer feature is compiled in. Set enabled = false to suppress it at runtime.
  • [swagger] — the embedded Swagger UI and OpenAPI spec. Enabled by default when the swagger feature is compiled in.
  • [swagger.oauth2] — drives the Swagger UI's "Authorize" button through an OIDC Authorization Code + PKCE flow. See Swagger UI and Authentication › Swagger UI SSO.
  • [metrics] — the Prometheus metrics endpoint (/metrics). Disabled by default; requires the metrics feature.
  • [sql] — the raw SQL endpoint (POST /api/v1/sql). Disabled by default; set enabled = true to expose it.
  • [auth] — OIDC / OAuth2 authentication with scope-based authorization. Disabled by default; requires the auth feature. Set enabled = true to enforce bearer tokens.

Examples

A minimal public server (no auth):

[server]
backend = "datafusion"
listen  = "0.0.0.0"
port    = 8080

[[dataset]]
name = "sales"
source.kind     = "parquet"
source.location = "./data/sales.parquet"

The same server with OIDC scope-based authorization — reads require the datasets:read scope and reloads require datasets:reload:

[server]
backend = "datafusion"
listen  = "0.0.0.0"
port    = 8080

[auth]
enabled              = true
issuer               = "https://login.microsoftonline.com/<tenant-id>/v2.0"
audience             = "api://datapress"
read_scopes          = ["datasets:read"]
reload_scopes        = ["datasets:reload"]
anonymous_read       = false
admin_token_fallback = false

[[dataset]]
name = "sales"
source.kind     = "parquet"
source.location = "./data/sales.parquet"

To keep reads public but still protect reloads, set anonymous_read = true and leave read_scopes empty:

[auth]
enabled        = true
issuer         = "https://login.microsoftonline.com/<tenant-id>/v2.0"
audience       = "api://datapress"
anonymous_read = true
reload_scopes  = ["datasets:reload"]

See Authentication for the full field reference, the equivalent Python AuthConfig, and a runnable Keycloak example.