CI/CD & deploy examples

Deploying a Dexaflow DAG is the same everywhere because a DAG is an immutable artifact — a dag.json + a container image, versioned together (ADR 0003).

By hand or for a team without a pipeline, one command does it all:

dexaflow auth login --server "$DEXAFLOW_SERVER"   # once; stores the token
dexaflow deploy --yes                            # compile → build → push → register

dexaflow deploy is the pipeline-less promotion — it cross-builds for the cluster, pins the image by digest, and registers the artifact. --yes skips the confirmation prompt (use it in automation).

In a pipeline, the same three boundaries are explicit so each can be cached, gated, and audited independently:

flowchart LR
  E[edit dag.py + dexaflow.yaml] --> C[dexaflow compile --build]
  C --> P[push image → registry]
  P --> R[dexaflow push dag.json → control plane]

1. dexaflow compile --build — parse dag.py, overlay dexaflow.yaml, run the guardrails (unknown task_id, unsupported operator, duplicate keys), and build the DAG image.
2. push the image to your registry, tagged by git SHA (immutable).
3. dexaflow push dag.json — register the artifact with the control plane.

Your repo of DAGs (CI is the recommended path)

In Pro you keep your DAGs in a Git repository — one directory per DAG — and CI turns each into an artifact when it changes. This is the recommended path: a push triggers the pipeline, no one builds by hand. Each DAG is just two files — no Dockerfile, no requirements.txt:

my-dags/                      # your Git repo
├── .github/workflows/
│   └── deploy-dag.yml        # the CI below (the recommended path)
└── dags/
    ├── my_pipeline/
    │   ├── dag.py            # the DAG (TaskFlow / operators)
    │   └── dexaflow.yaml      # id, python_version, dependencies, registry
    └── another_pipeline/
        ├── dag.py
        └── dexaflow.yaml

dexaflow compile --build synthesizes the image from dexaflow.yaml — FROM the published Dexaflow base, your deps/connectors installed, your DAG copied in. No Dockerfile to maintain. (Ship your own Dockerfile only if you want full control; it is then used verbatim. Our examples ship one so you can docker build and inspect them by hand.)

The built image is your artifact — push it wherever you like (Docker Hub, ECR, Artifact Registry, ACR, GHCR, a private registry), via the registry: block in dexaflow.yaml or --image. The only image Dexaflow owns is the base your DAG layers on.

The mental model: a push that touches dags/my_pipeline/** triggers CI for that DAG only (the paths: filter), which compiles → builds (from yaml) → pushes the image to your registry → registers dag.json. The control plane runs the new version on the next trigger. One DAG per pipeline keeps blast radius small: a broken another_pipeline never blocks my_pipeline.

dexaflow build my-dags/           # build every project under my-dags/
dexaflow build my-dags/ --push    # ...and push each to its registry

Prerequisites

• The dexaflow CLI on the runner (download the release binary, or go install).
• Python 3.11+ on the runner (dexaflow compile invokes the stdlib-only parser shim — ADR 0024 — to turn dag.py into dag.json). See Python on the runner below.
• A container registry your cluster can pull from.
• DEXAFLOW_SERVER (control plane URL) and DEXAFLOW_TOKEN (a push token) as CI secrets.

Python on the runner

The dexaflow compile step needs Python 3.11, 3.12, or 3.13 to parse dag.py. Bring your own Python on the runner — do not rely on dexaflow setup to download a managed CPython in CI (that path is designed for first-touch on a developer laptop, not for build pipelines, where it adds ~50 MB to every run and bypasses your runner’s pin/caching).

The recommended path on each runner type:

Older Python (≤3.10) fails the compile cleanly — dexaflow compile errors out with the version requirement, not a confusing traceback. Newer Python (3.14+) is accepted by the upper end of the detection range; the range is bumped per release once the parser shim is re-verified against it.

One more step: dexaflow setup extracts the parser

After the dexaflow binary lands on the runner and Python is in scope, run dexaflow setup ONCE per runner. The CLI ships the parser source embedded; setup extracts it under ~/.dexaflow/pysrc/parser/ and writes a config file pointing the compile command at the chosen interpreter. Without this step dexaflow compile fails with No module named leoflow_parser (the runner’s Python has no idea where the parser lives).

The snippets below all show dexaflow setup as the step after the install, before dexaflow compile. The follow-up to make this implicit (auto-bootstrap on first compile, or embed the parser execution inside the Go binary) is tracked separately; for now, calling it explicitly is the recommended path because it’s the operation that decides whether managed CPython is downloaded, and that’s a step CI operators should consciously opt into.

Examples

• GitHub Actions
• GitLab CI
• Google Cloud Build + Cloud Run

name: Deploy DAG
on:
  push:
    branches: [main]
    paths: ["dags/my_pipeline/**"]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5     # BYO Python — see #python-on-the-runner
        with: { python-version: '3.12' }
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Install dexaflow
        run: curl -fsSL https://raw.githubusercontent.com/dexadata/dexaflow/main/install.sh | DEXAFLOW_VERSION=<VERSION> DEXAFLOW_NO_SETUP=1 DEXAFLOW_INSTALL_DIR=/usr/local/bin sh
      - name: Bootstrap the parser (uses the BYO Python from above)
        run: dexaflow setup
      - name: Compile + build + push image
        run: |
          IMAGE=ghcr.io/${{ github.repository }}/my_pipeline:${{ github.sha }}
          dexaflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
      - name: Register with the control plane
        env: { DEXAFLOW_TOKEN: ${{ secrets.DEXAFLOW_TOKEN }} }
        run: dexaflow push dag.json --server ${{ secrets.DEXAFLOW_SERVER }}

deploy_dag:
  # Default docker:27 is Alpine-based; install python3 before dexaflow compile.
  # Alternative: a custom base image that bakes Python+Docker together.
  # See #python-on-the-runner for the rationale.
  image: docker:27
  services: [docker:27-dind]
  before_script:
    - apk add --no-cache python3   # 3.12 on Alpine 3.20+; see #python-on-the-runner
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      changes: ["dags/my_pipeline/**/*"]
  variables:
    IMAGE: $CI_REGISTRY_IMAGE/my_pipeline:$CI_COMMIT_SHA
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - wget -qO- https://raw.githubusercontent.com/dexadata/dexaflow/main/install.sh | DEXAFLOW_VERSION=<VERSION> DEXAFLOW_NO_SETUP=1 DEXAFLOW_INSTALL_DIR=/usr/local/bin sh
    - dexaflow setup        # extracts the parser into ~/.dexaflow/ using the python3 from before_script
    - dexaflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
    - dexaflow push dag.json --server "$DEXAFLOW_SERVER"   # DEXAFLOW_TOKEN from CI vars

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - |
        # BYO Python — Cloud Builders' docker image is Debian; install python3.
        # See #python-on-the-runner for the rationale.
        apt-get update -qq && apt-get install -y --no-install-recommends python3
        curl -fsSL https://raw.githubusercontent.com/dexadata/dexaflow/main/install.sh | DEXAFLOW_VERSION=<VERSION> DEXAFLOW_NO_SETUP=1 DEXAFLOW_INSTALL_DIR=/usr/bin sh
        dexaflow setup    # extracts the parser into ~/.dexaflow/ using the python3 just installed
        IMAGE="$_REGION-docker.pkg.dev/$PROJECT_ID/dags/my_pipeline:$SHORT_SHA"
        dexaflow compile dags/my_pipeline --image "$$IMAGE" --build --push -o dag.json
        dexaflow push dag.json --server "$_DEXAFLOW_SERVER"
substitutions:
  _REGION: us-central1
  _DEXAFLOW_SERVER: https://leoflow.run.app
options: { logging: CLOUD_LOGGING_ONLY }

• Generic / Makefile

dexaflow setup    # one-shot per runner: extracts the parser into ~/.dexaflow/
IMAGE="$REGISTRY/my_pipeline:$(git rev-parse --short HEAD)"
dexaflow compile dags/my_pipeline --image "$IMAGE" --build --push -o dag.json
dexaflow push dag.json --server "$DEXAFLOW_SERVER" --token "$DEXAFLOW_TOKEN"

Control-plane deployment (Helm chart, in validation)

Deploying the control plane itself (Helm chart, published dexaflow-server/ dexaflow-migrate images, TLS on the agent channel, keyless cloud auth) is the Pro track. One command installs the chart with auto-generated TLS and no cert-manager — from its published OCI artifact (helm install dexaflow oci://ghcr.io/dexadata/charts/dexaflow --version <VERSION>), or from source on main for the bleeding edge (helm install lf ./helm/dexaflow …). See Install Pro. The chart is installable today and in validation — see the Helm chart, the reproducible Kubernetes test setup (the deploy/k8s recipe is cloud-portable — it runs unchanged on EKS / GKE / AKS), Operating modes, and the Roadmap. The product proves itself in Lite first.

See also: DAG authoring · Operating modes.