Python configuration¶

`DataPressConfig`¶

from datap_rs.datapress import DataPressConfig

cfg = DataPressConfig(
    backend="datafusion",         # "datafusion" | "duckdb"
    listen="0.0.0.0",
    port=8000,
    workers=8,
    prefix="",                    # e.g. "/datapress" behind a proxy
    compress=True,
    max_body_bytes=1_048_576,     # 413 above this
    max_page_size=100_000,        # clamp query page_size above this
    force_lazy_above_mb=0,        # >0: force lazy for datasets larger than this (MiB)
    request_timeout_ms=30_000,    # 504 above this; 0 disables
    shutdown_timeout_secs=30,     # SIGTERM/SIGINT grace period
    server_environment="production",  # Explorer navbar badge label; None = no badge
    server_environment_color="danger",  # Bootstrap colour for the badge (see table below)
    swagger_enabled=True,
    swagger_path="/docs",
)

Every kwarg mirrors the TOML [server] block. See Configuration › Server for full semantics.

Environment badge¶

server_environment and server_environment_color mirror the TOML environment / environment_color keys. When server_environment is set, the Explorer navbar shows a coloured badge next to the logo. If server_environment_color is omitted the colour is inferred from the name (production/prod → red, staging/stage/uat → yellow, development/dev/local → green, anything else → grey). Set server_environment_color to any Bootstrap colour to override it:

name	hex	color
primary	`#0d6efd`	🟦
secondary	`#6c757d`	⬜
success	`#198754`	🟩
info	`#0dcaf0`	🟦
warning	`#ffc107`	🟨
danger	`#dc3545`	🟥
light	`#f8f9fa`	⬜
dark	`#212529`	⬛

Raw SQL endpoint¶

sql_enabled mirrors the TOML [sql] block and exposes POST /api/v1/sql for read-only SQL over a single dataset. It is disabled by default.

cfg = DataPressConfig(
    backend="datafusion",
    port=8000,
    sql_enabled=True,        # exposes POST /api/v1/sql (default False)
    sql_max_rows=100_000,    # hard cap on rows per query (default 100_000)
)

See Querying › Raw SQL for the request/response shape and the validation rules.

DataFusion performance tuning¶

Five optional kwargs mirror the TOML [datafusion] block and tune the DataFusion backend's parquet scan and object-store listing cache. They are ignored by the DuckDB backend and are all off by default:

cfg = DataPressConfig(
    backend="datafusion",
    port=8000,
    datafusion_pushdown_filters=True,        # decode-time row filtering
    datafusion_reorder_filters=True,         # reorder predicates by selectivity
    datafusion_list_files_cache=True,        # cache S3/object-store LISTs
    datafusion_list_files_cache_mb=64,       # listing-cache budget (MiB)
    datafusion_list_files_cache_ttl_secs=60, # 0 = never expire
)

datafusion_pushdown_filters pushes row-level predicates into the parquet decoder so rows failing a filter are never materialised. Best for selective filters over large row groups.
datafusion_reorder_filters reorders those predicates by selectivity — only effective together with datafusion_pushdown_filters.
datafusion_list_files_cache caches object-store file listings so repeated lazy queries reuse LIST results — the dominant per-query cost on S3. *_mb bounds the cache; *_ttl_secs bounds how long before newly written files become visible (0 = never expire). This cache does not help delta sources (their file list comes from the transaction log).

See Configuration › DataFusion performance tuning for the equivalent TOML knobs and their defaults.

PostgreSQL wire protocol (pgwire)¶

Seven pgwire_* kwargs mirror the TOML [server.pgwire] block. When enabled, DataPress starts a PostgreSQL wire-protocol listener alongside the HTTP API so psql, JDBC/ODBC drivers, and BI tools can query your datasets directly. It is DataFusion-only, read-only, and off by default, and requires a wheel built with the pgwire Cargo feature:

cfg = DataPressConfig(
    backend="datafusion",
    port=8000,
    pgwire_enabled=True,
    pgwire_listen="127.0.0.1",   # loopback-only unless password + TLS are set
    pgwire_port=5432,
    pgwire_username="datapress",
    pgwire_password="change-me",  # required for any non-loopback bind
)

Authentication is cleartext password, so the same safety rules as the TOML config apply and are validated when the config is built:

A loopback bind may omit pgwire_password.
Any non-loopback bind (e.g. pgwire_listen="0.0.0.0") requires both pgwire_password and TLS (pgwire_tls_cert + pgwire_tls_key).
pgwire_tls_cert and pgwire_tls_key must be set together.

cfg = DataPressConfig(
    backend="datafusion",
    pgwire_enabled=True,
    pgwire_listen="0.0.0.0",
    pgwire_username="datapress",
    pgwire_password="change-me",
    pgwire_tls_cert="/etc/datapress/pg.crt",
    pgwire_tls_key="/etc/datapress/pg.key",
)

A pgwire_enabled=True config on a wheel built without the pgwire feature logs a warning and is otherwise a no-op. See Clients › PostgreSQL (pgwire) for connecting psql and BI tools, and Configuration › PostgreSQL wire protocol for the equivalent TOML block.

Swagger UI OAuth2 / OIDC¶

AuthConfig protects the API. The swagger_oauth2_* fields only make the Swagger UI's Authorize button log in and attach a bearer token to "Try it out" requests.

from datap_rs.datapress import AuthConfig, DataPressConfig

cfg = DataPressConfig(
    backend="duckdb",
    port=8000,
    swagger_enabled=True,
    swagger_path="/docs",
    swagger_oauth2_issuer="https://issuer.example.com",
    swagger_oauth2_client_id="datapress-swagger",
    swagger_oauth2_scopes=["openid", "profile", "datasets:read"],
    swagger_oauth2_pkce=True,
)

auth = AuthConfig(
    enabled=True,
    issuer="https://issuer.example.com",
    audience="datapress-api",
    read_scopes=["datasets:read"],
    reload_scopes=["datasets:reload"],
)

swagger_oauth2_pkce=True enables the browser-safe Authorization Code with PKCE flow in Swagger UI. PKCE adds a one-time code_verifier / code_challenge pair so the UI can obtain a token without embedding a client secret in the page. Keep it enabled for public Swagger UI clients unless your identity provider explicitly cannot support PKCE.

Register this redirect URI with your IdP:

http://localhost:8000/docs/oauth2-redirect.html

For production, use your public HTTPS origin instead.

For machine-to-machine access, use the OAuth2 client credentials grant outside Swagger UI. Request a token from your IdP's token endpoint with grant_type=client_credentials, then call DataPress with the returned access token:

headers = {"Authorization": f"Bearer {access_token}"}

Do not put a client secret in Swagger UI or other browser code. If you want to try a client-credentials token from the docs page, obtain the token separately and paste it into Swagger UI's bearer authorization dialog.

The built-in Swagger UI configuration is intended for interactive OIDC login. It discovers the provider from swagger_oauth2_issuer, requests the configured scopes, and sends the access token on API calls; it does not store or exchange client secrets.

Explorer OAuth2 / OIDC¶

The explorer UI's API Query tab can offer the same interactive login. The explorer_oauth2_* fields mirror the Swagger ones and only drive the UI — attaching a bearer token to the requests the API Query tab makes. AuthConfig is still what enforces tokens on the API.

from datap_rs.datapress import DataPressConfig

cfg = DataPressConfig(
    backend="duckdb",
    port=8000,
    explorer_enabled=True,
    explorer_path="/explore",
    explorer_oauth2_issuer="https://issuer.example.com",
    explorer_oauth2_client_id="datapress-explorer",
    explorer_oauth2_scopes=["openid", "profile", "datasets:read"],
    explorer_oauth2_pkce=True,
)

Register this redirect URI with your IdP (matching explorer_path):

http://localhost:8000/explore/oauth2-redirect.html

For production, use your public HTTPS origin instead. The endpoints are discovered from explorer_oauth2_issuer at startup; if discovery fails the explorer is served without the Authorize button rather than a broken dialog. As with Swagger UI, keep explorer_oauth2_pkce=True for public browser clients and never put a client secret in browser code.

`DatasetConfig`¶

from datap_rs.datapress import DatasetConfig

ds = DatasetConfig(
    name="accidents",
    source="data/accidents.parquet",   # file, dir, glob, or s3://...
    format="parquet",                  # "parquet" | "delta"
    mode="auto",                       # eq-index policy
    description="US accidents 2016-2023",
    lazy=False,
    # DataFusion eq-index policy:
    # mode="list", index_columns=["State","Severity"],
    # index_max_cardinality=100_000,
)

kwarg	Meaning
`name`	URL slug + table name. Required.
`source`	Local path, glob, or `s3://...` URL. Required.
`format`	`"parquet"` (default) or `"delta"`.
`mode`	DataFusion eq-index policy: `"auto"` (default), `"none"`, `"list"`.
`index_columns`	Required when `mode="list"`.
`index_max_cardinality`	Auto-mode cardinality cap. Default 100_000.
`lazy`	Stream from disk instead of materialising (parquet + delta).
`description`	Free-form metadata; surfaced by `/api/v1/datasets`.
`s3`	`S3Config` — only for `s3://` sources.
`projection_include` / `projection_exclude`	Access control: hide columns everywhere. Set one, not both.
`predicate_include` / `predicate_exclude`	Access control: block columns from filters. Set one, not both.

Column access control

projection_* hides columns from every read surface (query, schema, sample, SQL, parquet export); predicate_* keeps a column visible but blocks it from where/having filters. Use *_include for an allowlist or *_exclude for a denylist — never both for the same filter. See Column access control for full enforcement semantics.

ds = DatasetConfig(
    name="people",
    source="data/people.parquet",
    projection_exclude=["ssn", "internal_notes"],
    predicate_exclude=["email"],
)

Empty datasets are skipped

If a dataset's source resolves to no files at startup (an empty directory, a non-matching glob, or an empty s3:// prefix), DataPress(...) logs a warning and skips just that dataset instead of raising — the server still boots with the remaining datasets. An empty Delta table is not skipped: it registers as a 0-row dataset. See Configuration › Datasets.

`S3Config`¶

from datap_rs.datapress import S3Config

s3 = S3Config(
    region="us-east-1",
    endpoint="http://localhost:9000",   # MinIO / R2 / Wasabi / Backblaze
    addressing_style="path",            # or "virtual"
    allow_http=True,                    # only for non-https endpoints
    access_key_id=None,
    secret_access_key=None,
    session_token=None,
    partitioning="auto",                # "auto" | "hive" | "none"
    endpoint_bucket_in_host="auto",     # "auto" | "true" | "false"
)

Credentials fall back to the standard AWS env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION) when not set inline. See Configuration › S3 for the full precedence chain and per-dataset env var overrides.

partitioning controls Hive partition discovery (auto detects key=value/ paths, hive forces it on, none off) and endpoint_bucket_in_host controls whether the bucket is folded into the endpoint hostname (auto follows addressing_style). The defaults make a plain s3://bucket/prefix/ source behave identically on the DuckDB and DataFusion backends.

Dynamic credentials with `credentials_provider`¶

Instead of hard-coding access_key_id / secret_access_key, pass a zero-argument callable that returns an HMACKeyPair. DataPress invokes it once while DataPress(...) is being constructed and caches the resolved keys indefinitely. This is handy when credentials live in a secrets manager (Vault, AWS Secrets Manager, …) or are minted on demand.

from datap_rs.datapress import HMACKeyPair, S3Config

def fetch_creds() -> HMACKeyPair:
    # e.g. read from Vault / AWS Secrets Manager / your own broker
    secret = my_secrets_client.get("datapress/s3")
    return HMACKeyPair(
        access_key=secret["access_key_id"],
        secret_key=secret["secret_access_key"],
    )

s3 = S3Config(
    region="us-east-1",
    endpoint="http://localhost:9000",
    allow_http=True,
    credentials_provider=fetch_creds,   # overrides static creds below
    access_key_id="ignored-when-provider-set",
    secret_access_key="ignored-when-provider-set",
)

When credentials_provider is set it takes precedence over any inline access_key_id / secret_access_key (those are ignored). The callable must return an HMACKeyPair with both keys non-empty, otherwise construction raises ValueError. The secret is redacted in the object's repr:

>>> HMACKeyPair("AKIA...", "super-secret")
HMACKeyPair(access_key="AKIA...", secret_key='***')

`AuthConfig`¶

Available when the wheel is built with the auth Cargo feature (maturin build --release --features auth). Mirrors the TOML [auth] block — see Operations › Authentication for field semantics and the full validation pipeline.

from datap_rs.datapress import AuthConfig

auth = AuthConfig(
    enabled=True,
    issuer="https://issuer.example.com",
    audience="datapress-api",
    read_scopes=["datasets:read"],
    reload_scopes=["datasets:reload"],
    anonymous_read=False,
    algorithms=["RS256"],
    leeway_secs=60,
    jwks_refresh_secs=3600,
    tenant_claim="",                   # e.g. "/tenant"
    allowed_tenants=[],
    admin_token_fallback=True,
    start_degraded=True,
)

kwarg	Default	Meaning
`enabled`	`False`	Master switch. When `False` all other fields are ignored.
`issuer`	`""`	Provider issuer URL; use discovery `issuer` / `iss`.
`audience`	`""`	Expected `aud` claim. Empty = skip audience check.
`read_scopes`	`[]`	Scopes required for `GET` endpoints.
`reload_scopes`	`[]`	Scopes required for reload / admin endpoints.
`anonymous_read`	`False`	Allow unauthenticated `GET` requests.
`algorithms`	`["RS256"]`	Permitted JWT signing algorithms.
`leeway_secs`	`60`	Clock-skew tolerance for `exp` / `nbf`.
`jwks_refresh_secs`	`3600`	Background JWKS refresh interval.
`tenant_claim`	`""`	JSON Pointer (`/tenant_id`, `/realm_access/roles/0`, …).
`allowed_tenants`	`[]`	Allow-list. Requires `tenant_claim`.
`admin_token_fallback`	`True`	Honour the legacy `X-Admin-Token` header on reload endpoints.
`start_degraded`	`True`	Boot even if JWKS fetch fails; all requests rejected until it recovers.

Pass it to DataPress(...) via the auth= keyword:

from datap_rs.datapress import DataPress, DataPressConfig, DatasetConfig, AuthConfig

dp = DataPress(
    DataPressConfig(backend="datafusion", listen="0.0.0.0", port=8000),
    [DatasetConfig(name="accidents", source="data/accidents.parquet")],
    auth=AuthConfig(
        enabled=True,
        issuer="https://issuer.example.com",
        audience="datapress-api",
        read_scopes=["datasets:read"],
        reload_scopes=["datasets:reload"],
    ),
)
import asyncio; asyncio.run(dp.run())

Validation rules (raised as ValueError):

enabled=True requires a non-empty issuer.
issuer is the exact issuer from your provider's discovery document or JWT iss claim. Keycloak commonly uses /realms/<realm>, but providers such as Auth0, Okta, Entra ID, and Zitadel use different paths.
allowed_tenants requires tenant_claim to be set.
tenant_claim must be a JSON Pointer starting with /.

To spin up a local OIDC provider for testing, see examples/keycloak/ — one docker compose up and you have a pre-provisioned realm with client datapress-api and the right scopes.

read_scopes and reload_scopes apply to the whole DataPress server instance. To enforce strict per-dataset access from Python, run one server instance per dataset or per access domain and use dataset-named scopes such as datasets:accidents:read and datasets:accidents:reload. See Python › Examples for a complete pattern.