dbt in your DAGs

A Leoflow DAG is a dag.py plus a leoflow.yaml. dbt_group("name") puts a dbt project inside one: Leoflow reads dbt’s own manifest.json at compile time and turns each node (seed, model, snapshot, test) into a task in the same graph as your Python and Bash tasks, executed pod-per-task against your warehouse — no Apache Airflow in the control plane, and no Cosmos at runtime.

If your DAG is only models, there is a shortcut that skips the dag.py entirely — see If your DAG is only models. Start here either way: the shortcut’s exit cost is real, and this section is what you will need the day you add one Python task.

1. Put a dbt project in your DAG

Author a dag.py and call dbt_group("<name>") where the models belong. The group is a task like any other: operators before it, operators after it, one graph. Configure the project under dbt_groups: in leoflow.yaml.

Embed the dbt project between your operators with dbt_group("<name>"):

# sales/dag.py
from leoflow import dbt_group
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import DAG

def extract(): ...
def notify(): ...

with DAG("sales", schedule="@daily"):
    pull   = PythonOperator(task_id="extract", python_callable=extract)
    models = dbt_group("transform")          # the dbt project, expanded
    ping   = PythonOperator(task_id="notify", python_callable=notify)

    pull >> models >> ping

# sales/leoflow.yaml
# The schedule lives in dag.py's DAG(schedule=…) — there is no top-level
# schedule: key, and leoflow.yaml rejects one.
dag_id: sales
dependencies:
  - dbt-postgres==1.9.*       # the adapter; the base image ships no dbt
dbt_groups:
  transform:                  # the name passed to dbt_group()
    project: ./transform
    granularity: level
    connection: warehouse_pg  # managed connection (see §3)

At compile, the operators and the dbt project become one dag.json. The dbt tasks are namespaced under the group (transform__stg, transform__orders, …), the group’s roots depend on extract, and notify depends on the group’s leaves:

extract → transform__level_0 → transform__level_1 → transform__level_2 → notify

2. Granularity — split vs fused

granularity controls how dbt nodes are packed into pods. It is a knob, not a fixed “one model = one pod”.

The folder key is the first segment, not the containing folder: marts/finance and marts/sales both become one marts task. A model with no folder at all falls back to its resource-type plural (models, seeds), so a project with a folder named models and a model at the root of models/ gets both in a single task — the compile warns on stderr when that happens, naming the members. Use granularity: node if you need them apart.

node is split (Leoflow’s scheduler parallelizes across pods, one model per pod). level/folder are fused — a group runs as a single dbt build --select <members> invocation.

Fused is parallel, not sequential

A fused group runs in one pod, but dbt’s own engine parallelizes the group’s independent models up to its threads setting, respecting the internal DAG. Measured on three independent models (3s each):

So fused trades per-model isolation and grid granularity for far fewer pod startups, while keeping in-pod parallelism. Rule of thumb:

• Lite (dev loop): node — cheap local subprocess, per-model visibility.
• Pro (production): level/folder — fewer pods at scale; node when you want strict per-model isolation and can afford the pods.

The fused trade-off is the same one Cosmos faces; the difference is Leoflow exposes it as a single declarative knob.

3. The warehouse connection

dbt needs a profiles.yml. Leoflow resolves it for you — pick the one that fits:

Zero-config local (Lite → duckdb)

On Lite, a dbt project with no connection: and no profiles.yml of its own just runs — against an embedded duckdb file (leoflow_local.duckdb, in the project) with no setup at all:

$ leoflow lite            # write models, hit Trigger — that's it

Leoflow generates the duckdb profile transparently at both compile (dbt parse) and run time, in the task’s working dir — never touching your global ~/.dbt. It’s the ideal way to develop and test transformations before wiring a real warehouse. Add a connection: (below) or a project profiles.yml at any time and that wins instead — the default only kicks in when there’s nothing configured.

Managed connection (recommended for Pro)

connections: [warehouse_pg, reporting]
variables: [env]

Set connection: to a Leoflow connection id. Leoflow delivers the connection to the pod (encrypted at rest, decrypted in-pod) and the runtime generates profiles.yml before dbt runs — no credential is ever baked into the image.

dbt:
  project: .
  connection: warehouse_pg

# create the connection once (UI, or the API)
$ curl -X POST .../api/v2/connections -d '{
    "connection_id":"warehouse_pg","conn_type":"postgres",
    "host":"db.internal","port":5432,"login":"etl","password":"…","schema":"transform"}'

The compiled command becomes:

python -m leoflow_runtime --dbt-profile warehouse_pg <profile> && dbt run --select …

Bring your own profiles.yml

Omit connection: and ship a profiles.yml in the project (it is baked into the image). Simple for Lite; you own the credential delivery.

Use one or the other — a connection: makes Leoflow generate the profile; without it, your baked profiles.yml is used.

Adapters: Postgres, Snowflake, BigQuery, Databricks (the official dbt-databricks adapter, not the community one), and duckdb (embedded, for zero-server local dev) are supported — Leoflow maps the managed connection to each adapter’s profile. Declare the adapter package (dbt-snowflake, dbt-bigquery, dbt-databricks, …) as a dependency so it lands in the image.

Each cloud adapter supports modern, service-account auth — Leoflow’s recommended mode for automation — alongside the legacy password/key-file mode. Everything is driven by the connection’s extra, so nothing secret is baked into the image, and the connection form surfaces these fields with inline help:

The recommended mode wins when its fields are present; otherwise the legacy mode is used. Per-warehouse setup — required fields (account/warehouse, http_path, …), example payloads, and precedence — lives in the connection reference: Snowflake, BigQuery, Databricks.

4. Failure isolation & the build parse-gate

A syntax error in one model does not blow up production. dbt parses the whole project on every invocation, so a compilation error in any model would, in naive setups, break every task. Leoflow stops that at the build parse-gate:

leoflow compile runs dbt parse on your machine — both editions, before it writes anything. A broken project never produces a dag.json and, on Pro, never produces an image: nothing deploys. So a syntax error fails loudly and early, never at 5am.

The baked manifest

Leoflow compiles from dbt parse’s target/manifest.json. leoflow compile runs dbt parse on your machine — both editions, not inside the image build — and the resulting manifest is copied into the DAG image alongside the project. Pin a pre-built one with dbt.manifest to skip the parse (see No local dbt).

Which dbt runs the parse is not the same in both editions, and the difference decides whether you need dbt installed on your host at all:

• Under leoflow dev the per-DAG venv is provisioned from dependencies: before the project is compiled, and the parse uses that venv’s dbt. So a Lite project that declares dbt-core and its adapter in dependencies: compiles with no dbt on your PATH — the version that parses your models is the same one that will run them.
• Running leoflow compile by hand provisions nothing. It falls back to the dbt on your PATH, and fails with a message naming both ways out if there is none: install dbt-core plus your adapter, or pin dbt.manifest.

The practical consequence is the reverse of what people usually assume: you do not need to pip install dbt-core to develop a dbt DAG in Lite. Declaring it in dependencies: is enough, and is better, because the host and the task then cannot disagree about the dbt version.

dbt’s Slim CI (--select state:modified+ --defer --state) is not offered as a turnkey recipe: there is no supported way to export the deployed manifest to diff against, and the compiler emits node/level/folder selectors only. If you drive dbt yourself in CI you can supply your own prior manifest.json as --state today.

Run-time errors (a model that compiles but fails against the warehouse) are isolated by granularity:

• node: the failing model fails its own pod; independent models succeed; only its downstream subtree is blocked.
• fused: dbt still materializes the group’s good models, but the group task fails as a unit (coarser blast radius).

Retrying a fused group re-runs the whole group

A fused group is one dbt build --select <members> task. If it fails mid-way, dbt keeps the models it already built — but retrying the task re-runs the entire group from scratch, including the models that already succeeded. dbt is not resumed from its failure point here (that would need dbt retry, which reads the previous run’s target/run_results.json — an artifact Leoflow does not yet persist across pod attempts). On warehouses billed per compute-second, retrying a mostly-green group re-bills the green models.

This is the flip side of the fused trade-off. If retry efficiency matters more than pod count for a given DAG, use granularity: node: each model is its own task, so a retry re-runs only the failed model (and its blocked downstream), exactly like Airflow’s per-task retry. Choose per DAG:

• Expensive warehouse + flaky sources → node — pay in pods, save on recompute.
• Cheap/idempotent models at scale → level/folder — pay a little recompute on the rare retry, save on pod startups.

Planned: resumable fused retries — persisting run_results.json so dbt retry skips the already-built models (#569).

5. Where config lives (two YAML worlds)

They never overlap: dbt_project.yml never mentions schedules/pods; leoflow.yaml never mentions SQL. Author your models in your dbt tooling (VS Code + dbt Power User, dbt Cloud IDE); Leoflow only adds orchestration and packing.

6. Adapters and auth

Leoflow generates each warehouse’s profiles.yml from your managed connection:, so the credential never enters the image or the repository.

Each emitted profile is checked in CI against the real adapter’s own credential parser — field names, alias resolution, required fields, and every auth mode above — so a profile Leoflow generates is one the adapter accepts.

That is not the same as a query succeeding against your account. Postgres and duckdb are the only adapters exercised against a live warehouse in CI; Snowflake, BigQuery and Databricks are contract-tested and hand-verified, because live-query coverage needs real accounts and CI secrets.

If your DAG is only models

A DAG whose tasks are only dbt models has no dag.py — and must not carry one, even an empty placeholder: compile refuses the pair. Declare the project under a top-level dbt: block and the shape comes from dbt’s ref()/source() graph. It is one file fewer to start with.

You write dbt the way you always do, and add one leoflow.yaml:

sales/                         # the DAG = a dbt project + leoflow.yaml
├── leoflow.yaml               # the only Leoflow file
├── dbt_project.yml            # dbt
├── profiles.yml               # dbt (or use a managed connection — see below)
├── seeds/raw_orders.csv
└── models/
    ├── staging/stg_orders.sql #  select … from {{ ref('raw_orders') }}
    └── marts/orders.sql       #  select … from {{ ref('stg_orders') }}

# leoflow.yaml
schema_version: "1.0"
dag_id: sales
owner: data-team
dependencies:
  - dbt-postgres==1.9.*        # the adapter; the base image ships no dbt
dbt:
  project: .                   # dir containing dbt_project.yml
  granularity: node            # node | level | folder  (see §2)
  schedule: "@daily"           # optional; empty = on-demand (Lite dev loop)

Compile it like any DAG:

$ leoflow compile ./sales --image registry.example.com/sales:v1
Compiled ./sales -> dag.json (image registry.example.com/sales:v1, version 9f3a2c1)

leoflow compile reads the dbt manifest and emits one task per node:

You never write task dependencies — {{ ref('stg_orders') }} is the edge.

The manifest comes from dbt parse, which leoflow compile runs on your machine. Set dbt.manifest: target/manifest.json to point at a pre-built one instead.

No local dbt

No dbt on this machine, or no wheel of your adapter for your Python? Pre-build the manifest in a container and pin it with dbt.manifest.

leoflow compile/--build shells out to dbt parse on your machine (the runtime never needs dbt — only compile-time manifest generation does). If your host has no dbt installed, or your adapter has no prebuilt wheel for your Python (a common one: dbt-databricks publishes wheels for 3.10–3.12, not the Python 3.9 that ships as the default python3 on some LTS distros/older macOS), dbt parse fails or the adapter refuses to install — before Leoflow ever gets involved.

Escape hatch: generate the manifest in a throwaway container with a Python dbt-databricks actually supports, then point dbt.manifest: at the result.

dbt parse does not open a warehouse connection, so a profiles.yml with placeholder values (matching the profile name in dbt_project.yml) is enough — no real credentials need to enter the container:

$ docker run --rm -v "$PWD":/proj -w /proj python:3.11-slim bash -c "
    pip install --no-cache-dir dbt-databricks &&
    dbt parse --profiles-dir . "
$ ls target/manifest.json     # now sitting in your project dir

# leoflow.yaml
dbt:
  project: .
  manifest: target/manifest.json   # pinned — leoflow compile skips `dbt parse` entirely
  connection: warehouse_databricks

A pinned manifest is used as-is (see loadDbtManifest — no local dbt required at all from that point on); regenerate it in the container whenever you change models. This is the same trick CI runners use when their base image doesn’t carry a compatible Python for every adapter — see Python on the runner for the CI-side version of this problem.

Reference

leoflow.yaml dbt: (whole-DAG) and each dbt_groups: entry (embedded) accept:

Cosmos at a glance