Skip to content

DataPress

Turn Parquet and Delta datasets into fast, typed HTTP APIs without standing up a warehouse, writing a service layer, or moving data out of object storage.

DataPress is an opinionated small and fast data server for teams that already have columnar files and need a dependable way to publish them: JSON for applications, Arrow IPC for analytics clients, health probes for orchestrators, and a Python package for notebooks, jobs, and embedded services. There's even a JDBC client driver and it's compatible with PostgreSQL drivers using the PostgreSQL wire protocol.

pip install datap-rs

Prefer a standalone binary? Install it from crates.io — one datapress command bundles both the DuckDB and DataFusion backends:

cargo install datapress

from datap_rs.datapress import DataPress, DataPressConfig, DatasetConfig

server = DataPress(
    DataPressConfig(backend="duckdb", port=8000),
    datasets=[DatasetConfig(name="events", source="s3://lake/events/*.parquet")],
)
  • Publish data where it lives

Serve local files, S3-compatible buckets, Parquet partitions, and Delta tables through one predictable API.

  • Choose the execution engine

Run the same request model on DuckDB or Apache Arrow + DataFusion and compare them under the same workload.

  • Serve apps and analysts

Return paged JSON for product surfaces or Arrow IPC streams for Python, Polars, DuckDB, notebooks, and downstream pipelines.

  • Operate it like a service

Use readiness/liveness probes, version metadata, graceful shutdown, Prometheus metrics, hot reloads, and optional OIDC/OAuth2 scopes.

What It Does

DataPress exposes configured datasets over a versioned HTTP API:

  • discover datasets and schemas;
  • query columns with predicates, sorting, grouping, aggregation, distinct, limits, and pagination;
  • count matching rows without fetching them;
  • stream large results as Arrow IPC;
  • reload datasets without restarting the server;
  • enforce bearer-token scopes when OIDC/OAuth2 auth is enabled.

The goal is not to replace your lakehouse, database, or BI platform. It is the thin, fast publication layer between columnar data and the people or systems that need to consume it.

Who It Helps

  • Data engineers

Publish curated Parquet or Delta outputs directly from pipelines. Avoid bespoke Flask/FastAPI wrappers, schema hand-coding, and repeated export jobs.

  • Application developers

Query datasets with a stable HTTP contract instead of linking against storage-specific readers or embedding SQL engines into every service.

  • Analytics engineers

Give notebooks and jobs an Arrow-native path for bulk pulls while still supporting lightweight JSON inspection and operational APIs.

  • Business teams

Turn governed files into reusable data products: documented, discoverable, reloadable, and protected by familiar OAuth2 scopes.

Why It Is Fast

DataPress leans on columnar formats and mature query engines instead of serialising everything through an application ORM.

  • Rust + actix-web keeps the HTTP layer compact and predictable.
  • DuckDB is excellent for huge or growing datasets, object storage, Delta, and rich SQL execution \u2014 served eagerly from memory or streamed lazily from source.
  • Apache Arrow + DataFusion gives a pure-Rust path with resident Arrow batches (or lazy streaming) and optional equality indexes for hot point lookups.
  • Arrow IPC avoids JSON overhead when clients need many rows.
  • Projection and predicate pushdown mean clients can ask for the columns and rows they need instead of downloading whole files.

Two engines, one API

DuckDB and DataFusion expose the same HTTP request and response shapes. Pick the engine in config, A/B-test, and switch without rewriting clients.

Why Rust

The headline win is memory density, but the root cause is parallelism. A data server's hot path isn't just the query engine — it's also the glue around every request: predicate parsing, validation, pagination, auth, and JSON / Arrow encoding. In Rust that glue runs on all cores inside one process, so DataPress materialises each dataset (and its indexes) once and hands every worker thread a cheap shared Arc — an 8-worker server holds exactly one copy of the data in RAM.

Python can share a single resident copy too — DuckDB, Arrow, and Polars all release the GIL, so the engine work even parallelises. The catch is the glue: it stays pure-Python and serialises on the GIL. To put that glue on every core, the standard answer is to fork multiple worker processes (gunicorn -w N, several uvicorn workers) — and forking is what gives each process its own address space, loading the dataset once per worker. So Python's memory duplication isn't an inability to share; it's the price of working around the GIL to use all cores. Rust skips the workaround, and the single-copy memory model falls out for free. (Free-threaded Python 3.13+ is starting to chip away at this, but the ecosystem and the C extensions are still catching up.)

That single-process parallelism is the foundation; Rust adds three more reasons that matter just as much:

  • Single static binary. cargo install datapress, Homebrew, winget, or a tiny scratch Docker image — no interpreter, no venv, no dependency resolution at deploy time. The CLI and server ship as one file.
  • Embeddable two ways. The exact same engine compiles into a pip install datap-rs wheel via PyO3 + maturin and is available as a crate other Rust projects can depend on directly. One codebase, four delivery forms: server, CLI, Python wheel, library.
  • Predictable for a long-lived server. No GC pauses, no GIL contention spikes, fearless multithreading, and memory safety without a runtime — exactly what a process that stays up for weeks wants.

One process, many workers, one copy of the data

Every actix-web worker thread serves requests against the same in-memory dataset through a shared Arc. Loading the data twice never happens, no matter how many workers you run:

flowchart TB
    subgraph proc["DataPress · single OS process"]
        direction TB
        subgraph workers["actix-web worker threads"]
            direction LR
            W1["Worker 1"]
            W2["Worker 2"]
            W3["Worker 3"]
            Wn["Worker N"]
        end
        DS["Shared dataset + indexes<br/>(one Arrow copy in RAM)"]
        W1 --> DS
        W2 --> DS
        W3 --> DS
        Wn --> DS
    end
    Storage[("Parquet / Delta<br/>local · S3")] -->|loaded once at startup| DS
    Clients(["HTTP clients<br/>JSON · Arrow IPC"]) --> workers

Contrast that with the multi-process model: N uvicorn workers are N separate address spaces, so the same dataset is resident N times.

Reload on demand — without a restart

A POST to the reload endpoint swaps the resident dataset for a freshly loaded copy atomically (DataFusion uses an ArcSwap double-buffer; DuckDB a transactional replace). In-flight queries finish against the old copy; new queries see the new one. This makes DataPress easy to drive from an external event loop — for example a Python process consuming a Kafka topic that fires whenever a pipeline publishes new files:

sequenceDiagram
    autonumber
    participant Pipe as Data pipeline
    participant Kafka as Kafka topic
    participant Py as Python consumer
    participant DP as DataPress server
    participant Mem as Resident dataset

    Pipe->>Kafka: publish "events refreshed"
    Kafka-->>Py: new message
    Py->>DP: POST /api/v1/datasets/events/reload
    DP->>Mem: load new copy, atomic swap
    Note over Mem: in-flight queries use the old copy — new queries hit the new copy
    DP-->>Py: 200 OK

See Operations › Dataset reload for the backend-specific swap semantics and auth requirements.

Why It Is Easy

DataPress keeps the setup surface intentionally small.

Declare a dataset in TOML:

[server]
backend = "duckdb"
port = 8000

[[dataset]]
name = "events"
source.kind = "parquet"
source.location = "s3://lake/events/*.parquet"

Or launch the same server from Python:

from datap_rs.datapress import DataPress, DataPressConfig, DatasetConfig

cfg = DataPressConfig(backend="duckdb", port=8000)
ds = DatasetConfig(name="events", source="s3://lake/events/*.parquet")
await DataPress(cfg, datasets=[ds]).run()

Then query it over HTTP:

curl -s -X POST http://localhost:8000/api/v1/datasets/events/query \
  -H 'Content-Type: application/json' \
  -d '{
    "columns": ["event_id", "country", "amount"],
    "predicates": [{ "col": "country", "op": "eq", "val": "NL" }],
    "page_size": 1000
  }'

No generated service code. No schema structs to keep in sync. No separate API for every dataset. Configure the data, start the server, and query it.

Production Shape

  • Versioned routes under /api/v1.
  • /healthz, /readyz, and /version for deployment automation.
  • Graceful shutdown on SIGTERM and SIGINT.
  • Optional Prometheus metrics.
  • Optional OIDC/OAuth2 bearer validation with read and reload scopes.
  • Hot reload endpoints for publishing refreshed datasets.
  • Built-in MkDocs and Swagger UI embedding for self-documenting deployments.

Where It Fits

DataPress works well when you need to expose data products, operational analytics, customer-facing slices, internal tools, data science extracts, or pipeline outputs without turning every use case into a custom backend project.

It is especially useful when your source of truth is already Parquet or Delta, your consumers speak HTTP, and you want the option to serve both small interactive requests and larger Arrow-native pulls from the same deployment.

Start Here

Install, configure your first dataset, run a backend, and query it with curl.

Learn every server, dataset, S3, auth, metrics, and docs setting.

Use predicates, projection, pagination, aggregation, JSON, and Arrow IPC.

Choose between DuckDB and Arrow + DataFusion for your workload.

Install datap-rs, launch a server from Python, and call it from scripts or notebooks.

Run it with probes, reloads, logging, metrics, graceful shutdown, and OIDC/OAuth2.