> ## Documentation Index > Fetch the complete documentation index at: https://docs.cycls.com/llms.txt > Use this file to discover all available pages before exploring further. # Build a data warehouse on a volume > Store Parquet on a Cycls volume, query it with DuckDB from a function, schedule the ingest, and let an agent ask questions in SQL. This guide is for developers who want analytical queries without running a database server. A [volume](/build/volumes) holds Parquet files, DuckDB reads them inside a function, and an agent calls that function as a tool. **Prerequisites:** `CYCLS_API_KEY`, and Docker if you want to run locally first. ## Architecture ```mermaid theme={null} flowchart LR S[Source API or CSV] --> I["ingest()
scheduled function"] I -->|writes Parquet| V[("volume: lake
/lake/events/*.parquet")] V -->|read only mount| Q["query()
function"] Q --> A[Agent tool] Q --> D[Dashboard app] ``` One writer, many readers. That split matters, because a volume is shared files with no locking: concurrent writes to the same file are last write wins. Do not put a DuckDB database file (`.duckdb`) on a shared volume and write to it from more than one deployment. DuckDB expects exclusive access to a database file, and the volume gives no locking. Write immutable Parquet files instead, one per partition, and let readers open them read-only. ## 1. Create the volume and load a file ```bash theme={null} cycls volume create lake cycls volume put lake ./events-2026-09-01.csv raw/2026-09-01.csv cycls volume ls lake raw/ ``` ## 2. Write the ingest function ```python warehouse.py theme={null} import cycls lake = cycls.Volume("lake") image = cycls.Image().pip("duckdb") @cycls.function(image=image, volumes={"/lake": lake}, memory="2Gi") def ingest(day: str): """Convert one day of raw CSV into a Parquet partition.""" import pathlib import re import duckdb if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", day): raise ValueError(f"day must be YYYY-MM-DD, got {day!r}") raw = f"/lake/raw/{day}.csv" if not pathlib.Path(raw).exists(): return {"day": day, "status": "missing"} pathlib.Path("/lake/events").mkdir(parents=True, exist_ok=True) out = f"/lake/events/{day}.parquet" con = duckdb.connect() con.execute( f"COPY (SELECT * FROM read_csv_auto('{raw}')) " f"TO '{out}' (FORMAT PARQUET, COMPRESSION ZSTD)" ) rows = con.execute("SELECT count(*) FROM read_parquet(?)", [out]).fetchone()[0] return {"day": day, "rows": rows, "path": out} ``` Run it against one day: ```bash theme={null} cycls run warehouse.py::ingest --day 2026-09-01 # {'day': '2026-09-01', 'rows': 18422, 'path': '/lake/events/2026-09-01.parquet'} ``` `COPY ... TO` takes a literal path, not a bound parameter, so the date is validated before it reaches the SQL string. Writing one file per day makes a rerun idempotent: the same input overwrites the same partition, which is what makes at-least-once scheduling safe. ## 3. Backfill with a fan-out ```python theme={null} @cycls.local_entrypoint def main(): days = [f"2026-09-{d:02d}" for d in range(1, 31)] for result in ingest.map(days): print(result) ``` ```bash theme={null} cycls run warehouse.py ``` `map` runs one call per item across autoscaled instances and returns results in input order. ## 4. Query it ```python theme={null} @cycls.function(image=image, volumes={"/lake": lake.read_only()}, memory="4Gi") def query(sql: str, limit: int = 200): """Run read-only SQL over every Parquet partition.""" import duckdb con = duckdb.connect() con.execute("CREATE VIEW events AS SELECT * FROM read_parquet('/lake/events/*.parquet')") con.execute("SET enable_external_access = false") rel = con.execute(f"SELECT * FROM ({sql}) LIMIT {int(limit)}") columns = [d[0] for d in rel.description] return {"columns": columns, "rows": [list(r) for r in rel.fetchall()]} ``` ```bash theme={null} cycls run warehouse.py::query --sql "SELECT country, count(*) c FROM events GROUP BY 1 ORDER BY c DESC" ``` Two safeguards are worth keeping. `lake.read_only()` means this deployment cannot modify the warehouse even if the SQL tries. `SET enable_external_access = false` stops DuckDB from reading URLs or local paths outside what is already attached. `query` runs arbitrary SQL. Only expose it to callers you trust, or to an agent whose output you are willing to treat as user input. The read-only mount limits the damage to reads, and the external access setting blocks file and network reads from inside SQL. ## 5. Schedule the ingest ```python theme={null} import datetime @cycls.function( image=image, volumes={"/lake": lake}, schedule=cycls.Cron("0 2 * * *", timezone="Asia/Riyadh"), ) def nightly(): day = (datetime.date.today() - datetime.timedelta(days=1)).isoformat() return ingest.remote(day) ``` ```bash theme={null} cycls deploy warehouse.py ``` A scheduled function is called with no arguments, so `nightly` computes the day itself and delegates to `ingest`. A rerun overwrites the same partition, which is what makes at-least-once delivery harmless here. ## 6. Give an agent access Deploy `query` as a named endpoint, then call it from a tool handler. ```python analyst.py theme={null} import asyncio import cycls TOOLS = [ { "name": "warehouse_sql", "description": ( "Run read-only SQL over the events warehouse. " "One table: events(day DATE, country TEXT, product TEXT, amount DOUBLE)." ), "inputSchema": { "type": "object", "properties": {"sql": {"type": "string"}}, "required": ["sql"], }, } ] run_sql = cycls.remote("query") async def warehouse_sql(args): result = await asyncio.to_thread(run_sql, args["sql"]) if not result["rows"]: return "No rows." header = " | ".join(result["columns"]) body = "\n".join(" | ".join(str(c) for c in row) for row in result["rows"][:50]) return f"{header}\n{body}" llm = ( cycls.LLM() .model("anthropic/claude-sonnet-4-6") .system( "You answer questions about sales using the warehouse_sql tool. " "Write one query, read the result, then answer in plain language." ) .tools(TOOLS) .on("warehouse_sql", warehouse_sql, label=lambda i: i["sql"][:60]) ) @cycls.agent( image=cycls.Image().copy(".providers.env", ".env"), web=cycls.Web().auth(cycls.Clerk()).title("Analyst"), volumes={"/workspace": cycls.Volume("analyst-chats")}, ) async def analyst(context): async for ev in llm.run(context=context): yield ev ``` ```bash theme={null} cycls deploy analyst.py ``` The agent image contains no data and no DuckDB. It calls the deployed `query` endpoint, which keeps the chat container small and the warehouse in one place. ## 7. Serve a dashboard from the same data ```python dashboard.py theme={null} import cycls lake = cycls.Volume("lake") @cycls.app(image=cycls.Image().pip("duckdb"), volumes={"/lake": lake.read_only()}) def dashboard(): from fastapi import FastAPI import duckdb app = FastAPI() con = duckdb.connect() con.execute("CREATE VIEW events AS SELECT * FROM read_parquet('/lake/events/*.parquet')") @app.get("/revenue") async def revenue(): rows = con.execute( "SELECT day, sum(amount) AS total FROM events GROUP BY 1 ORDER BY 1" ).fetchall() return [{"day": str(d), "total": t} for d, t in rows] return app ``` Three deployments now share one volume: `ingest` writes, `query` and `dashboard` read. Deleting any of them leaves the data untouched. ## Operational notes | Concern | What to do | | -------------- | ------------------------------------------------------------------------------------------------------------------- | | Partition size | aim for 100MB to 1GB per Parquet file. Smaller files make the glob slow | | Schema changes | write a new column into new partitions, and use `union_by_name=true` in `read_parquet` when reading across a change | | Visibility | a file written by one deployment appears to others within seconds, not instantly | | Memory | DuckDB spills to disk, so set `memory` on the query function to match your largest scan | | Cost | idle deployments scale to zero. Volume storage is billed on what is stored | | Backfills | `map()` over a date list, and keep partitions one file per day so reruns are safe | ## Next Turn a nightly query into a file users can open.