Build a data warehouse on a volume

This guide is for developers who want analytical queries without running a database server. A volume holds Parquet files, DuckDB reads them inside a function, and an agent calls that function as a tool. Prerequisites: CYCLS_API_KEY, and Docker if you want to run locally first.

Architecture

One writer, many readers. That split matters, because a volume is shared files with no locking: concurrent writes to the same file are last write wins.

Do not put a DuckDB database file (.duckdb) on a shared volume and write to it from more than one deployment. DuckDB expects exclusive access to a database file, and the volume gives no locking. Write immutable Parquet files instead, one per partition, and let readers open them read-only.

1. Create the volume and load a file

cycls volume create lake
cycls volume put lake ./events-2026-09-01.csv raw/2026-09-01.csv
cycls volume ls lake raw/

2. Write the ingest function

warehouse.py

import cycls

lake = cycls.Volume("lake")
image = cycls.Image().pip("duckdb")


@cycls.function(image=image, volumes={"/lake": lake}, memory="2Gi")
def ingest(day: str):
    """Convert one day of raw CSV into a Parquet partition."""
    import pathlib
    import re
    import duckdb

    if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", day):
        raise ValueError(f"day must be YYYY-MM-DD, got {day!r}")

    raw = f"/lake/raw/{day}.csv"
    if not pathlib.Path(raw).exists():
        return {"day": day, "status": "missing"}

    pathlib.Path("/lake/events").mkdir(parents=True, exist_ok=True)
    out = f"/lake/events/{day}.parquet"

    con = duckdb.connect()
    con.execute(
        f"COPY (SELECT * FROM read_csv_auto('{raw}')) "
        f"TO '{out}' (FORMAT PARQUET, COMPRESSION ZSTD)"
    )
    rows = con.execute("SELECT count(*) FROM read_parquet(?)", [out]).fetchone()[0]
    return {"day": day, "rows": rows, "path": out}

Run it against one day:

cycls run warehouse.py::ingest --day 2026-09-01
# {'day': '2026-09-01', 'rows': 18422, 'path': '/lake/events/2026-09-01.parquet'}

COPY ... TO takes a literal path, not a bound parameter, so the date is validated before it reaches the SQL string. Writing one file per day makes a rerun idempotent: the same input overwrites the same partition, which is what makes at-least-once scheduling safe.

3. Backfill with a fan-out

@cycls.local_entrypoint
def main():
    days = [f"2026-09-{d:02d}" for d in range(1, 31)]
    for result in ingest.map(days):
        print(result)

cycls run warehouse.py

map runs one call per item across autoscaled instances and returns results in input order.

4. Query it

@cycls.function(image=image, volumes={"/lake": lake.read_only()}, memory="4Gi")
def query(sql: str, limit: int = 200):
    """Run read-only SQL over every Parquet partition."""
    import duckdb

    con = duckdb.connect()
    con.execute("CREATE VIEW events AS SELECT * FROM read_parquet('/lake/events/*.parquet')")
    con.execute("SET enable_external_access = false")

    rel = con.execute(f"SELECT * FROM ({sql}) LIMIT {int(limit)}")
    columns = [d[0] for d in rel.description]
    return {"columns": columns, "rows": [list(r) for r in rel.fetchall()]}

cycls run warehouse.py::query --sql "SELECT country, count(*) c FROM events GROUP BY 1 ORDER BY c DESC"

Two safeguards are worth keeping. lake.read_only() means this deployment cannot modify the warehouse even if the SQL tries. SET enable_external_access = false stops DuckDB from reading URLs or local paths outside what is already attached.

query runs arbitrary SQL. Only expose it to callers you trust, or to an agent whose output you are willing to treat as user input. The read-only mount limits the damage to reads, and the external access setting blocks file and network reads from inside SQL.

5. Schedule the ingest

import datetime


@cycls.function(
    image=image,
    volumes={"/lake": lake},
    schedule=cycls.Cron("0 2 * * *", timezone="Asia/Riyadh"),
)
def nightly():
    day = (datetime.date.today() - datetime.timedelta(days=1)).isoformat()
    return ingest.remote(day)

cycls deploy warehouse.py

A scheduled function is called with no arguments, so nightly computes the day itself and delegates to ingest. A rerun overwrites the same partition, which is what makes at-least-once delivery harmless here.

6. Give an agent access

Deploy query as a named endpoint, then call it from a tool handler.

analyst.py

import asyncio

import cycls

TOOLS = [
    {
        "name": "warehouse_sql",
        "description": (
            "Run read-only SQL over the events warehouse. "
            "One table: events(day DATE, country TEXT, product TEXT, amount DOUBLE)."
        ),
        "inputSchema": {
            "type": "object",
            "properties": {"sql": {"type": "string"}},
            "required": ["sql"],
        },
    }
]

run_sql = cycls.remote("query")


async def warehouse_sql(args):
    result = await asyncio.to_thread(run_sql, args["sql"])
    if not result["rows"]:
        return "No rows."
    header = " | ".join(result["columns"])
    body = "\n".join(" | ".join(str(c) for c in row) for row in result["rows"][:50])
    return f"{header}\n{body}"


llm = (
    cycls.LLM()
    .model("anthropic/claude-sonnet-4-6")
    .system(
        "You answer questions about sales using the warehouse_sql tool. "
        "Write one query, read the result, then answer in plain language."
    )
    .tools(TOOLS)
    .on("warehouse_sql", warehouse_sql, label=lambda i: i["sql"][:60])
)


@cycls.agent(
    image=cycls.Image().copy(".providers.env", ".env"),
    web=cycls.Web().auth(cycls.Clerk()).title("Analyst"),
    volumes={"/workspace": cycls.Volume("analyst-chats")},
)
async def analyst(context):
    async for ev in llm.run(context=context):
        yield ev

cycls deploy analyst.py

The agent image contains no data and no DuckDB. It calls the deployed query endpoint, which keeps the chat container small and the warehouse in one place.

7. Serve a dashboard from the same data

dashboard.py

import cycls

lake = cycls.Volume("lake")


@cycls.app(image=cycls.Image().pip("duckdb"), volumes={"/lake": lake.read_only()})
def dashboard():
    from fastapi import FastAPI
    import duckdb

    app = FastAPI()
    con = duckdb.connect()
    con.execute("CREATE VIEW events AS SELECT * FROM read_parquet('/lake/events/*.parquet')")

    @app.get("/revenue")
    async def revenue():
        rows = con.execute(
            "SELECT day, sum(amount) AS total FROM events GROUP BY 1 ORDER BY 1"
        ).fetchall()
        return [{"day": str(d), "total": t} for d, t in rows]

    return app

Three deployments now share one volume: ingest writes, query and dashboard read. Deleting any of them leaves the data untouched.

Operational notes

Concern	What to do
Partition size	aim for 100MB to 1GB per Parquet file. Smaller files make the glob slow
Schema changes	write a new column into new partitions, and use `union_by_name=true` in `read_parquet` when reading across a change
Visibility	a file written by one deployment appears to others within seconds, not instantly
Memory	DuckDB spills to disk, so set `memory` on the query function to match your largest scan
Cost	idle deployments scale to zero. Volume storage is billed on what is stored
Backfills	`map()` over a date list, and keep partitions one file per day so reruns are safe

Scheduled reports

Turn a nightly query into a file users can open.

Get started

Functions

Apps

Agents

Interface and users

Ship it

Guides

Reference

Resources

For LLMs

Build a data warehouse on a volume

Architecture

1. Create the volume and load a file

2. Write the ingest function

3. Backfill with a fan-out

4. Query it

5. Schedule the ingest

6. Give an agent access

7. Serve a dashboard from the same data

Operational notes

Next

Scheduled reports

​Architecture

​1. Create the volume and load a file

​2. Write the ingest function

​3. Backfill with a fan-out

​4. Query it

​5. Schedule the ingest

​6. Give an agent access

​7. Serve a dashboard from the same data

​Operational notes

​Next

Scheduled reports

Architecture

1. Create the volume and load a file

2. Write the ingest function

3. Backfill with a fan-out

4. Query it

5. Schedule the ingest

6. Give an agent access

7. Serve a dashboard from the same data

Operational notes

Next