DAG authoring

Author a DAG: dexaflow.yaml plus dag.py compiled to one immutable artifact.

A Dexaflow DAG is two files in a project directory, compiled into an immutable artifact (dag.json + a container image, versioned together — ADR 0003).

dags/my_pipeline/
  dag.py         # real Apache Airflow SDK 3.2.x code
  dexaflow.yaml   # Dexaflow deploy config (not an Airflow file)

Scaffold one with dexaflow init dags/my_pipeline.

Workspace layout (multi-DAG)

Lite-specific. The hot-reload watcher exists only in dexaflow lite — the developer-mode loop. At runtime Cluster has no “workspace”: every DAG ships as its own image-and-dag.json pair, built by CI and registered via dexaflow push dag.json, and the control plane never sees a directory of siblings. See The development → deploy lifecycle.

The same discovery does run at build time in Cluster, in one place: dexaflow build [workspace] walks a directory of projects and builds each with its own registry:. That is a convenience for the person at the keyboard — it produces the same one-image-per-DAG artifacts and changes nothing about how Cluster runs them.

dexaflow lite watches a workspace that can hold many DAGs as sibling subdirectories. The default workspace is ~/dexaflow/ (set by dexaflow setup).

~/dexaflow/                      # workspace root
  recurring_print/               # one DAG project per subdir
    dexaflow.yaml
    dag.py
  recurring_parallel/
    dexaflow.yaml
    dag.py
  ml/
    train/                       # nested subdirs are scanned too
      dexaflow.yaml
      dag.py

Recommended layout (best practice, not enforced): name the subdirectory the same as the dag_id. The binding is by dag_id (yaml field, or the subdir basename when no yaml is present — see below), but matching names make the workspace navigable by humans and grep-friendly. ~/dexaflow/sales_etl/ for a DAG named sales_etl.

Discovery rules

dexaflow lite — and dexaflow build, which shares the same discovery — walks the workspace and treats every subdirectory containing a dag.py (with or without a dexaflow.yaml) as a project. The scan:

  • Goes at most 5 levels deep from the workspace root. A DAG at <ws>/a/b/c/d/dag.py is the deepest valid case. Deeper paths are skipped.
  • Skips .git, __pycache__, .venv, venv, and any hidden directory (.*). That list mirrors the exclude_paths default so the scan and an image build agree on what counts as noise — but discovery has its own copy and never reads your exclude_paths, so changing that value does not change what the scan walks.
  • Fails loud on a duplicate dag_id: if two subdirs resolve to the same id, lite refuses to compile any of them and prints both paths so you can rename one. There is no last-write-wins.

Every compile log line names the resolved config source — either the absolute path of the dexaflow.yaml that was loaded, or auto-defaults: <subdir> when none exists. This is the one line to grep when “which config did it pick up?” is the debugging question.

dexaflow.yaml is optional

A subdir with just a dag.py is a valid project. Dexaflow synthesizes a config with dag_id = <subdir-basename> and every other field filled from the schema defaults (see Configuration → Defaults). Add a dexaflow.yaml when you need to pin a Python version, declare dependencies, set per-task overrides, or change the dag_id to something other than the subdir name.

dag.py — the Airflow dialect

dag.py is real Airflow SDK 3.2.x code, imported by the Python parser via the Airflow DagBag. TaskFlow and classic operators both work:

from airflow.sdk import DAG, task

@task
def extract() -> dict:
    return {"rows": 100}

@task
def transform(data: dict) -> dict:
    return {"rows": data["rows"], "doubled": data["rows"] * 2}

with DAG("my_pipeline", schedule="@daily", catchup=False, tags=["etl"]):
    transform(extract())

A DAG starts paused the first time it is deployed to Cluster, as in Airflow, so its schedule waits for an unpause. Pass is_paused_upon_creation=False to DAG(...) or @dag(...) to have it start active (True keeps it paused on a server configured otherwise). It only matters on that first deploy; see New DAGs start paused.

from airflow.sdk import … is the canonical Airflow 3 spelling and what we recommend. The deprecated-but-still-valid Airflow 3 aliases resolve to the same objects, so a migrated DAG compiles unchanged:

  • from airflow.decorators import task, dag → the airflow.sdk decorators.
  • from airflow import DAG → airflow.sdk.DAG.

The names a DAG file usually imports only for its structure or its type hints also compile from 0.5.11: the exceptions in airflow.exceptions and airflow.sdk.exceptions, TriggerRule, WeightRule, BaseSensorOperator, PokeReturnValue, BaseHook, Connection, Context, ParamsDict, Metadata, ObjectStoragePath, get_parsing_context, literal, and the pre-3 modules airflow.models.dag, airflow.models.baseoperator, airflow.hooks.base and airflow.utils.trigger_rule. import pendulum works for pendulum.datetime(..., tz=...), pendulum.duration(...) and pendulum.timezone(...), so start_date can be written as in the Airflow docs. get_current_context(), BaseHook.get_connection() and Connection.get() work inside a task body; called at module level they are refused, because no run exists while the file is compiled. op.output is the task’s return value, as in Airflow.

A module the compile cannot import is reported as missing from the Python that compiles the DAG, not as a Dexaflow limitation: import it inside the task body, which runs only in the task image, or install it where dexaflow compile runs.

The core airflow.operators.* operators, however, were removed from Airflow in 3.0 and relocated to apache-airflow-providers-standard. Importing one (e.g. from airflow.operators.bash import BashOperator) fails the compile with a message pointing at the replacement — use from airflow.providers.standard.operators.bash import BashOperator instead.

Dependencies between tasks

Tasks are wired as on Airflow, and every form below compiles to the same depends_on lists in dag.json:

  • a >> b, b << a, and lists on either side: a >> [b, c] >> d, [a, b] >> c.
  • a.set_downstream(b) and b.set_upstream(a).
  • chain(a, [b, c], [d, e], f) links each element to the next: a task next to a list links to every task in it, and two adjacent lists, which must have the same length, link pairwise (b >> d, c >> e).
  • cross_downstream([a, b], [c, d]) makes every task of the first list upstream of every task of the second.
  • Label("text") on an edge (a >> Label("ok") >> b, also inside chain) wires the edge; the label text is not kept, so the graph view shows the edge without it.

chain, cross_downstream, Label and Variable import from airflow.sdk. The older spellings (airflow.models.baseoperator, airflow.utils.edgemodifier, airflow.models) resolve to the same objects.

For a start or end marker, EmptyOperator from airflow.providers.standard.operators.empty compiles to a bash task that runs true.

Generating tasks in a loop

A DAG file is plain Python, so tasks built in a for loop compile, edges included:

TABLES = ["orders", "customers", "items"]

@task
def load(table: str) -> None: ...

with DAG("load_tables", schedule="@daily"):
    for table in TABLES:
        extract = BashOperator(task_id=f"extract_{table}", bash_command=f"extract {table}")
        extract >> load.override(task_id=f"load_{table}")(table)

The loop can read its list from a file next to dag.py (a JSON or YAML config, for example), with three limits:

  • The task list is fixed at compile. dexaflow compile runs the loop once and writes the resulting tasks to dag.json, which is immutable. A change to the list, in the code or in the config file, takes effect on the next deploy, not on the next run. A list read from a database, an API or a Variable while the file is parsed is not supported (see Variable.get below).
  • One DAG per file. A loop that creates several DAGs in one file (for country in COUNTRIES: with DAG(f"etl_{country}"): ...) is refused unless dexaflow.yaml names the one to build with dag_id. Give each DAG its own directory, sharing the code through a helper module.
  • No fan-out at run time. A number of task instances decided by an upstream result (.expand(), .partial()) is dynamic task mapping, which is refused (see below). Return the list from one task and process it inside the next one instead.

Task concurrency per run (max_active_tasks)

max_active_tasks on DAG(...) or @dag(...) caps how many task instances of each run of the DAG may be queued or running at once, as in Airflow 3. The other ready tasks of the run wait in scheduled until one of its tasks finishes. Two runs of the DAG each get the full cap, so the DAG as a whole can have up to max_active_runs times max_active_tasks tasks active:

with DAG("fan_out", schedule="@daily", max_active_runs=2, max_active_tasks=4):
    ...  # at most 4 tasks of each run at once, 8 for the DAG

Unset means no cap. The value must be a positive integer; anything else is refused at compile. concurrency, the Airflow 2 name that Airflow 3 removed, is refused with a message naming max_active_tasks. To cap tasks across all runs of a DAG, or across DAGs, use a pool. Before 0.5.10 the compiler dropped max_active_tasks, so a DAG that set it ran without a cap (#2427).

Supported task types

Dexaflow runs the common types on a native fast path and everything else — any of Airflow’s ~1,500 provider operators and poke-mode sensors — through a generic executor, native-first by type. Nothing is silently dropped or mistranslated.

Task typeAirflow operatorWhere it runsNotes
python@task (TaskFlow) or PythonOperator, and @task.bashagent in a pod (Cluster) / subprocess (Lite)The general-purpose escape hatch: any provider library you pip install is callable from inside a @task. A @task.bash function runs in the pod and returns the command, which then runs with bash, as on Airflow.
bashBashOperator, and EmptyOperatoragentExecutes a shell command. An EmptyOperator compiles to a bash task that runs true: it costs one short pod and shows as a BashOperator in the UI.
http_api (removed)——Removed (ADR 0047/0048, #512). HttpOperator now compiles to airflow_operator and runs in a pod, like any other provider operator — declare connectors: [http]. The old inline path ran the request in the control-plane process (an SSRF surface) and is gone.
airflow_operatorany provider operator/sensor (Snowflake, S3, Postgres, BigQuery, …)agent in a podThe generic executor (ADR 0040): the runtime imports the class, instantiates it with your args, and calls execute(). Declare the provider or compile fails.

Also supported:

  • Trigger rules: all_success, all_failed, all_done, one_success, one_failed.
  • XCom: TaskFlow data-flow (transform(extract())) is resolved automatically into typed inputs (xcom_input) — for @task and operator args.
  • Schedule: cron strings and presets (@daily, 0 * * * *).
  • Dependencies: linear ordering via TaskFlow calls or a >> b.
  • .override(...) on a @task: estimate.override(task_id="estimate_03")(3) builds the task with different operator arguments, as in Airflow. It takes the same arguments as @task(...) (task_id, retries, retry_delay, execution_timeout, trigger_rule, pool_slots, …), merged over the decorator’s, and leaves the decorated function unchanged. A repeated task_id gets the usual __1, __2 suffix. Two limits: on_failure_callback is refused in .override(...) (set it on @task(...), where the runtime reads it), and .override(...).expand(...) is refused like any dynamic task mapping.

Connector cookbooks (postgres, mysql, sqlite, redis, http) are python tasks that read a managed Connection (AIRFLOW_CONN_*) — that is the hook-inside-@task style. The airflow_operator type below is the operator-class style; both resolve the same managed connections.

Provider operators & sensors (airflow_operator)

Write the operator exactly as you would in Airflow — Dexaflow captures it and runs it in its own task pod:

from airflow.sdk import DAG, task
from airflow.providers.snowflake.operators.snowflake import SQLExecuteQueryOperator

@task
def build_sql() -> str:
    return "SELECT count(*) FROM events"

with DAG("rollup", schedule="@daily"):
    # an upstream task's output flows straight into an operator arg
    SQLExecuteQueryOperator(task_id="rollup", conn_id="snowflake_default",
                            sql=build_sql())
# dexaflow.yaml — installs apache-airflow-providers-snowflake into the image
connectors: [snowflake]      # or: dependencies: [apache-airflow-providers-snowflake]

How it works, and the Phase-A limits:

  • The operator runs in its own task pod; its hooks resolve connections from AIRFLOW_CONN_* exactly as inside a @task.
  • Constructor args must be JSON-serializable or an upstream task’s output (wired as XCom, like sql=build_sql() above). A non-serializable arg — a callable, a datetime, an arbitrary object — is a loud compile error; move that logic into a @task.
  • Literal args are for small constants. A task’s literal args (@task call_args and operator operator_args) ride as a single environment variable at dispatch, which POSIX caps at ~128 KiB; dexaflow compile rejects a payload over 100 KiB with a clear error naming the task. For large data, pass a Connection or an external-storage reference (S3/GCS) and fetch it inside the task — never a big dict/list literal.
  • The provider must be declared in dexaflow.yaml (connectors: or dependencies:). If it isn’t, dexaflow compile fails and prints the exact line to add — no surprise ModuleNotFoundError in the pod.
  • Sensors run in poke mode by default (the pod holds until the condition is met), or mode="reschedule" (ADR 0040 Phase B, #380/#389) — the pod is released between pokes and the scheduler re-dispatches the sensor, preserving its try_number, once its next-poke time passes.
  • Jinja templating renders the operator’s template_fields with the run context ({{ ds }}, {{ logical_date }}, {{ macros.ds_add(ds, 1) }}, {{ params.x }}, {{ var.value.x }}). A field that fails to render fails the task attempt before execute(), with the field and the Jinja error in the task log, as on Airflow (ADR 0084). For a context key Dexaflow does not provide yet, compute the value in a @task and pass it in.
  • A native task type (bash/python) always wins when it matches, so BashOperator/HttpOperator/PythonOperator keep their fast path.

Not supported — dexaflow compile rejects these

Each entry names the workaround and, where support is planned, the tracking issue.

Control flow and structure

  • Branching (BranchPythonOperator, @task.branch) and short-circuit (@task.short_circuit, ShortCircuitOperator): refused; the scheduler does not model skipped downstream paths yet. Planned in #787 (ADR 0064).
  • A task skipping itself (raise AirflowSkipException): the task fails instead of ending skipped. Planned with branching (#2347).
  • Trigger rules other than all_success, all_failed, all_done, one_success, one_failed (for example none_failed_min_one_success, the usual join after a branch): refused. Planned in #2346.
  • Task groups (TaskGroup, @task_group): refused at compile, with this advice in the message. Flatten the group into plain tasks and prefix the ids by hand (for example extract_load for extract.load). The dbt integration’s own dbt_group() is the one group construct that works. Planned as a grouping and UI construct, with no change to execution, in #2338.
  • Dynamic task mapping (.expand / .partial): refused. Static fan-in (collecting a list of task calls into a downstream task) works; dynamic fan-out, expanding one task into N instances at run time, does not.
  • Setup and teardown tasks (as_setup(), as_teardown(), @setup, @teardown): refused at compile; express the cleanup as a downstream task with trigger_rule="all_done". Not planned beyond the refusal.
  • chain_linear(): not available; use chain(), cross_downstream() or >> with lists. chain(), cross_downstream() and edge Labels compile (see Dependencies between tasks).
  • EmptyOperator from airflow.sdk: refused, because Airflow has no EmptyOperator there either. Import it from airflow.providers.standard.operators.empty.

Dependencies between DAGs and scheduling

  • Cross-DAG operators (ExternalTaskSensor, ExternalTaskMarker, TriggerDagRunOperator) and LatestOnlyOperator: refused at compile, subclasses included, because they need the Airflow metadata database, which a task pod does not have. See Operators that need the Airflow metadata database. Planned as waits the scheduler evaluates itself, with no pod (ADR 0103, #2340, #2348).
  • Assets (schedule=[Asset(...)], @asset, inlets=, outlets=): not implemented; the import, or the argument, is refused at compile with a pointer to #2352, where a design is tracked.
  • schedule=timedelta(...) and schedule="@continuous": refused at compile from 0.5.11. Before, the DAG compiled with no schedule and only ran when triggered. Use a cron string or a preset, for example "*/30 * * * *" for every 30 minutes.
  • end_date, on the DAG, on a task or in default_args: refused at compile from 0.5.11; it was dropped and the DAG kept running. Pause the DAG when it should stop.
  • Timetables (CronTriggerTimetable, EventsTimetable, custom ones): refused. Use a cron string or a preset such as @daily. CronTriggerTimetable is planned in #2351.
  • depends_on_past=True, wait_for_downstream=True and a priority_weight other than 1: refused at compile, on a task or in the DAG’s default_args, because the scheduler does not honor them and the run would proceed as if they were not set. Their default values (False, False, 1) still compile. depends_on_past and wait_for_downstream are planned in #2350.
  • max_active_tis_per_dag and max_active_tis_per_dagrun: refused at compile from 0.5.10, on a task or in default_args, for the same reason: the cap was dropped and the task ran without it. Unset (None) still compiles. Planned in #2428.
  • dagrun_timeout and max_consecutive_failed_dag_runs on the DAG: refused at compile from 0.5.10. A run never timed out and the DAG never paused after failed runs, so the DAG behaved as if they were unset. Their defaults (None, 0) still compile. Use execution_timeout on the tasks to bound a run.
  • More DAG arguments Dexaflow does not honor, refused at compile from 0.5.11 when set to anything but their default: on_failure_callback and on_success_callback on the DAG (put on_failure_callback in default_args so every task runs it, or use an alerts: block), sla_miss_callback, fail_fast, deadline, access_control (access follows Dexaflow’s roles), allowed_run_types, and the template options render_template_as_native_obj, user_defined_macros, user_defined_filters, template_searchpath, template_undefined and jinja_environment_kwargs. Each was dropped without a word before.
  • concurrency= on the DAG: refused at compile; Airflow 3 removed it. Use max_active_tasks, a cap per DAG run.

Operators and sensors

  • Deferrable operators and sensors (deferrable=True): refused at compile; Dexaflow has no triggerer (ADR 0016). Pass deferrable=False for a synchronous poke, or use mode="reschedule" on a sensor for a long wait.
  • Time sensors (TimeDeltaSensor, DateTimeSensor, TimeSensor, WaitSensor): an operator argument that is a timedelta or a datetime (other than retry_delay and execution_timeout) is refused, because the value cannot be carried in dag.json, so TimeDeltaSensor does not compile; the others run as ordinary sensors when their arguments are plain values, one pod per poke. Planned as scheduler waits with no pod in #2345.
  • PythonSensor(python_callable=...) and @task.sensor: refused. Poll the condition inside a @task, or use a provider sensor in mode="reschedule". Planned in #2339.
  • on_failure_callback on @task.bash: refused. The command replaces the Python process, so nothing is left to run the callback; use a @task that runs the command with subprocess, or a downstream task with trigger_rule="one_failed".
  • Virtualenv operators (PythonVirtualenvOperator, ExternalPythonOperator, @task.virtualenv, @task.external_python): refused, by design. Each DAG already ships as its own image with its own dependencies; declare them in dexaflow.yaml and use @task.
  • KubernetesPodOperator (and KubernetesJobOperator, GKEStartPodOperator, EksPodOperator, @task.kubernetes, subclasses included): refused, by design. Every Dexaflow task already runs in its own pod, and an author-controlled pod spec is the escalation surface the pod-per-task model exists to contain. Before 0.5.11 it compiled once the provider was declared, and then failed in the pod or ran outside admission.
  • DockerOperator, DockerSwarmOperator and @task.docker: refused from 0.5.11; a task pod has no Docker daemon. Run the work as a task of the DAG image.
  • Human in the loop operators (HITLOperator, ApprovalOperator, HITLBranchOperator, HITLEntryOperator): refused from 0.5.11. They need the Airflow API server and a triggerer.
  • A subclass of a provider operator (class AuditedSQL(SQLExecuteQueryOperator)): refused from 0.5.11. The pod ran the parent class by its class path, so every override was lost. Call the provider operator from a @task, or put the extra logic in a task before or after it.
  • A custom operator class written in the project (a BaseOperator or BaseSensorOperator subclass): refused. Move the logic of its execute into a @task.
  • @task.pyspark, @task.run_if and @task.skip_if: refused, each with its own message. Create the Spark session inside a @task; check a condition at the start of the task and return early.

Python authoring

  • Variable.get (or set) called at module level: refused. The compiler has no Variable store, and a value read while parsing would be frozen into dag.json. Importing Variable at the top of the file is fine; call Variable.get inside the task, or use {{ var.value.<key> }} in a templated field.
  • Indexing a task’s output (result["key"], the multiple_outputs=True pattern): refused. Pass the whole result to the downstream task and read the key there.
  • Task arguments Dexaflow does not honor, refused at compile from 0.5.11 when set on a task or in default_args to anything but their default: email (use an alerts: block; email_on_failure and email_on_retry do nothing without it and still compile), retry_exponential_backoff and max_retry_delay (planned in #2454), retry_policy, sla (removed in Airflow 3), executor_config, resources and queue (set size: or resources: in dexaflow.yaml), executor, run_as_user, weight_rule, pre_execute, post_execute, on_execute_callback and on_skipped_callback. Each was dropped without a word before.
  • default_args in dag.py: retries, retry_delay, execution_timeout, pool, pool_slots, trigger_rule (from 0.5.11) and on_failure_callback are applied to every task that does not set its own (the callback was dropped before 0.5.10). start_date there is the DAG’s start date when the DAG sets none. Every argument above that Dexaflow refuses is refused there too, and owner and the doc* arguments are accepted and ignored.

Run parameters and their schemas

params= on the DAG declares the parameters a run accepts, as in Airflow. A bare value is a default; a Param adds a JSON Schema built from its keyword arguments, and a Param without a default is required:

from airflow.sdk import DAG, Param

with DAG(
    "my_pipeline",
    params={
        "limit": Param(100, type="integer", minimum=1),
        "region": Param(type="string", enum=["us", "eu"]),
    },
):
    ...

The control plane checks the schemas when the DAG is registered and validates a run’s conf against them when the run is triggered. A task’s xcom_schema in dag.json works the same way for the value the task pushes. These schemas may $ref only into themselves (#/$defs/...) and the standard JSON Schema meta-schemas. Since 0.5.1 any other reference, such as a file:// URL or a relative path, is refused with a schema error at registration, at trigger or at the push, so inline what a schema needs instead of pointing at a file.

A task sees params ({{ params.x }}, context["params"]) the way Airflow builds them: the DAG’s declared defaults, then the task’s own params=, then the run’s conf, each one winning over the one before for the same key. Since 0.5.5 this holds for every run: a scheduled run, which carries no conf, still gets the DAG defaults, and an operator’s params= is part of its template context. Since 0.5.5 the same holds for the params= of a BashOperator and of @task(params=...), which the compiler now carries in dag.json. A task param must be JSON-serializable, or dexaflow compile fails naming it. context["dag_run"].conf is the run’s conf alone, without the defaults. A manual trigger stores the declared defaults in the run’s conf, as the Airflow UI’s trigger form does, so on such a run a DAG default wins over a task param of the same name.

Clearing a task

Clearing a task (the UI’s Clear, or POST /api/v2/dags/{dag_id}/clearTaskInstances) runs it again as a new try with a full retry budget, as Apache Airflow does:

  • the budget is the retries of the DAG version the re-run executes, so a task with retries: 3 gets three retries after a clear, whatever it spent before;
  • its infra re-placements are reset, so it can survive a lost agent again;
  • the XCom values of the attempts it clears are deleted, so a downstream task cannot read a value the new attempt did not write;
  • on_failure_callback fires only on the final attempt.

The clear is one transaction: if the XCom values cannot be deleted, nothing is cleared and the request fails. Before 0.5.1 a clear kept the spent retries, and the UI could show “try 4 of 3”.

Clearing onto the current version

By default a clear re-runs the version the run was created with. To re-run on the DAG’s current version, tick “Run with latest bundle version” in the Clear dialog, or send run_on_latest_version: true. As in Airflow, this moves the run only when the run has finished (success or failed) and none of its task instances is queued or running. The run then takes the current version’s tasks in the same transaction:

  • a task the version adds gets a task instance and runs in the re-opened run;
  • a task the version dropped keeps its task instance, history and logs, in the new state removed. It is not run, and it does not count toward the run’s outcome;
  • a removed task whose task a later version declares again runs again as a new try.

A clear that names a task the current version dropped resets nothing for it. See ADR 0086 for the reasoning.

Clearing a run that is still running

A run that is queued or running, or a finished run that still has a task instance queued or running (marked failed or success while that task ran), keeps its DAG version. Asking for the current version does not move it:

  • the cleared tasks re-run on the run’s own version, with that version’s retries, as a clear without the option would;
  • tasks already running finish on the image they started with;
  • the response says the option was not applied, and why. Both clear routes, dry run included, add run_on_latest_version_not_applied, one entry per run with its dag_run_id and a reason such as the run is running: only a finished run moves to the current version. The run’s version in the UI stays the one it had. The Clear dialog of the Airflow UI that Dexaflow ships does not display this field, so read it from the API when you need the reason.

To run the cleared tasks on the current version, first get the run to a finished state with no task queued or running, then clear again with the option:

  1. wait for the run to finish, or mark it failed or success (the run page’s Mark run as, or PATCH /api/v2/dags/{dag_id}/dagRuns/{dag_run_id} with a state);
  2. if you marked it while a task was still running, wait for that task to end;
  3. clear again with run_on_latest_version: true.

Or trigger a new run, which always uses the current version.

For example, with etl still running on version 3 while version 4 is current:

curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  "$DEXAFLOW_URL/api/v2/dags/etl/dagRuns/$RUN_ID/clear" \
  -d '{"dry_run": true, "run_on_latest_version": true}' | jq .run_on_latest_version_not_applied
[
  {
    "dag_run_id": "manual__2026-10-16T09:00:00+00:00",
    "reason": "the run is running: only a finished run moves to the current version"
  }
]

Once the run has finished, the same dry run answers no run_on_latest_version_not_applied, and the clear with "dry_run": false moves the run to version 4.

dexaflow.yaml — deploy config

These are Dexaflow concerns, not Airflow operator attributes (you cannot invent kwargs on an operator — the parser imports real Airflow and would raise).

schema_version: "1.0"
dag_id: my_pipeline
description: Daily ETL.
owner: data-eng
tags: [etl]
python_version: "3.11"
dependencies:           # pip packages baked into the image
  - pandas==2.1.0
defaults:               # DAG-level defaults (applied to every task)
  retries: 1
  retry_delay_seconds: 30
staging:                # opt-in shared per-run RWX volume (ADR 0022)
  enabled: false
tasks:                  # per-task overrides, keyed by task_id (ADR 0023)
  transform:
    retries: 3
    resources:
      requests: { cpu: "2", memory: 4Gi }

Binding + override layers (ADR 0023)

Config binds to the DAG by dag_id and to tasks by task_id. Three layers, most specific wins:

task override (tasks.<id>)  >  DAG default (defaults)  >  platform default (server)
  • tasks.<id> is merged at compile time onto the task in dag.json.
  • Platform defaults are applied at dispatch time, filling gaps the artifact left empty (keeps the artifact portable across clusters), one resource dimension at a time: a task that declares only ephemeral_storage, or only cpu, still gets the platform default for the dimensions it left out. A dimension the task sets at all (a request or a limit) is entirely the task’s (#802).
  • staging is DAG-level only — one RWX volume is shared atomically by the whole run, so it cannot be per-task.

Task size (size, ADR 0066)

size is how many slots of its pool a task takes while it is queued or running: Airflow’s pool_slots. It defaults to 1, so a pool of 8 slots runs eight tasks of size 1, or two of size 4. A task that does not fit waits until enough slots free up; it never fails for it. Smaller tasks cannot keep a task of more than one slot out forever: once it has waited past scheduler.pool_starvation_threshold (60s by default), the pool holds new admissions until it fits. Tasks of size 1 never reserve a pool.

The Pools screen and /api/v2/pools count slots the same way: two tasks of size 4 running in a pool of 8 show 8 occupied slots and 0 open, and each state’s count (running_slots, queued_slots, scheduled_slots, deferred_slots) is the sum of its tasks’ sizes. A task’s size and pool are recorded when its run starts, and a clear takes both from the version the re-run executes, so a task a newer version moved to another pool is charged there once it is cleared onto that version. The API’s task instances report the same pool and pool_slots.

defaults:
  size: 1          # every task that sets none
tasks:
  train:
    size: 4        # this task takes 4 slots

@task(pool_slots=4) or an operator’s pool_slots=4 in dag.py sets the same thing. Most specific wins: tasks.<id>.size > pool_slots in dag.py > defaults.size > 1. The size counts only where pools are on, which is the cluster edition by default; see Pools for naming a pool and reading how full it is.

When the operator sets a resource unit (executor.unit, for example 250m CPU and 512Mi memory per slot), the size also sizes the pod: a task of size 4 gets 1 CPU and 2Gi wherever it does not set cpu or memory itself, and a task whose own resources ask for more than size x unit is refused when the DAG is registered, with the size it would need.

Guardrails (fail loudly, never silently)

  • A tasks: entry naming a task_id absent from the DAG → compile error.
  • A duplicate task_id key in the YAML → parse error.
  • Across a monorepo, a duplicate dag_id is a CI-gate concern (one image per DAG).
  • A task’s execution.labels or execution.annotations key under the leoflow.io/ prefix → validation error. That prefix belongs to Dexaflow’s own pod metadata, which other components identify task and warm-worker pods by; use a prefix of your own (for example team.example.com/owner). A DAG version registered before this rule still runs, with those keys dropped from its pods.

The development → deploy lifecycle

flowchart LR
  I[dexaflow init] --> D[dexaflow lite<br/>hot-reload loop]
  D -->|save & iterate| D
  D --> G[git push]
  G --> CI[CI: dexaflow compile --build --push]
  CI --> REG[dexaflow push dag.json]
  REG --> PROD[(control plane<br/>immutable artifact)]

1 · Develop (fast, isolated)

dexaflow init dags/my_pipeline          # scaffold dag.py + dexaflow.yaml
dexaflow lite dags/my_pipeline           # cluster-mode: real pods on an isolated k3d
# or: dexaflow lite --executor=subprocess dags/my_pipeline   (fastest, host venv)

Open http://localhost:8088 — the UI is marked Dexaflow Lite (silver edition badge); log in with the admin password generated by dexaflow setup (or run dexaflow lite reset-password if you misplaced it). Edit dag.py/dexaflow.yaml and save; Dexaflow recompiles, re-runs the guardrails, and re-registers. A bad binding prints an error in the terminal immediately:

✗ dexaflow.yaml tasks: unknown task_id "transfrom"; the DAG defines [extract load transform]

Lite is fully isolated from Demo/Cluster (own database, cluster, and ports): no split brain. See Operating modes.

2 · Deploy (authoritative, immutable)

On git push, CI compiles + builds + pushes the artifact. The same parser, overlay, and guardrails run as a gate, so what you tested in Lite is what ships:

dexaflow compile dags/my_pipeline --image ghcr.io/org/my_pipeline:$GIT_SHA --build --push -o dag.json
dexaflow push dag.json

Without -o the artifact lands next to the project (dags/my_pipeline/dag.json), not in the directory you ran the command from. The pipelines below pass -o so the two lines agree wherever the job happens to run.

Full, copy-pasteable pipelines for GitHub Actions, GitLab CI, Google Cloud Build/Run, and generic runners are in CI/CD & deploy examples.

What ends up in the image

--build bakes the build context into the image, and with the default project: "." that context is your whole DAG directory. The image is pushed to a registry and pulled by every pod that runs the DAG, so anything in it is shared with everyone who can pull it.

exclude_paths in dexaflow.yaml decides what stays out. During the build it is materialized as a .dockerignore in the context — merged with your own if you have one, and removed afterwards, so the workspace is unchanged when the build ends. Your rules come first and Dexaflow’s last, which means exclude_paths has the final word: a ! re-include in your .dockerignore cannot silently defeat an exclusion you declared in dexaflow.yaml.

Each entry is written out in the forms Docker honours rather than verbatim. .dockerignore is not .gitignore: a pattern with no slash matches only at the context root, so __pycache__ alone would leave every nested one in the image. Dexaflow emits p, **/p and p/** for a bare name — the last of those is also what lets exclude_paths beat an earlier ! rule, which re-stating the plain pattern does not do.

For a dbt project, two things a host-side dbt parse leaves behind are excluded automatically, scoped to each project directory: logs/ and .user.yml. Both matter beyond image size — .user.yml is dbt’s anonymous-usage cookie identifying your machine, and it is read by the in-pod dbt, so every pod from that image reports as you; logs/dbt.log carries absolute paths from the build host.

Nothing else in a dbt project is excluded, deliberately. target/ holds the manifest dbt.manifest points at, dbt_packages/ is where dbt deps installs, and profiles.yml may be a BYO profile you point DBT_PROFILES_DIR at — each is a real input in a configuration people use, so excluding them would break working projects.

Credentials are not excluded for you either. A .env can be a legitimate input — a DAG calling load_dotenv() reads it at run time — and so can a BYO profiles.yml, so dropping either silently would break that project far from the cause. Instead the build warns:

warning: .env is in the build context and will be baked into the image, which is
pushed to a registry and pulled by every pod that runs this DAG. If it holds
credentials, add ".env" to exclude_paths in dexaflow.yaml.

Act on it or declare it deliberately; the warning stops once the file is in exclude_paths.


See also: Concepts & glossary · Operating modes · HTTP API · ADR 0023 — binding & overrides.