Troubleshooting & observability

Symptoms grouped by where they surface. Start with the diagnostics — most issues are one dexaflow doctor away from a clear cause. New here? The Quickstart and Installation guides cover a clean first run; this page is where you land when one goes wrong.

First things to run

dexaflow doctor                          # host check (OS, python, docker, k3d, kubectl, recommended tier); with a target, the deploy preflight too
dexaflow version                         # version + commit + build date
tail -f /tmp/leoflow-lite.log           # the live boot log when you ran `dexaflow lite` via lite-redeploy
journalctl -u dexaflow-server -f         # Pro / systemd hosts

dexaflow --version
dexaflow-server --version
dexaflow-agent --version
dexaflow-mcp --version

Install & setup

The migration Job

On Pro, helm install and helm upgrade run the migrations in a pre-install/pre-upgrade hook Job. When it fails, Helm prints only failed pre-install: resource Job/<namespace>/<release>-migrate not ready ... Job Failed; the reason is in the pod’s log:

kubectl -n <namespace> logs -l app.kubernetes.io/component=migrate --tail=50

The dexaflow-migrate image prints two lines: what happened and what to do, then the driver’s own error (driver error: ...), which is what older images printed alone and what the rows below quote. Each failure a retry cannot fix exits with its own code, and on Kubernetes 1.31 or newer the Job fails on the first such pod (podFailurePolicy) instead of retrying it three times.

Exit code 2 is a command line the image cannot run, which only a changed migrations.image or hand-written args produce.

A control plane that is up but not ready

A Pro control plane that finds its database schema absent or behind does not exit. It stays up, /healthz answers 200 so Kubernetes does not restart it, and /readyz answers 503 with the reason and what to run. It serves nothing else meanwhile: every other request, the API and the UI included, gets the same 503, so nothing is written to the database. It checks again on its own, after 1 second at first, doubling up to every 30 seconds, and starts serving without a restart once the schema is current. This is what happens after the database was emptied, restored into a fresh instance or repointed, or when a rollout runs before its migration Job.

kubectl -n <namespace> get pods -l app.kubernetes.io/name=dexaflow   # Running, 0/1 ready
kubectl -n <namespace> logs deploy/<release> --tail=20 | grep 'database schema'

The log has one line per change of reason, database schema is not ready; serving only the probes and metrics and checking again, with reason and action fields, and database schema is current; starting when it moves on. On a chart install the action names this release’s own command, with its release name, namespace and chart version filled in, or, with migrations.enabled: false, your own migration step and the dexaflow-migrate image of the release. Which case you are in, and what to run for each, is on database lost, restored or repointed.

A pod that was serving when the database changed under it stays up and not ready too, with postgres schema not current on /readyz and the same action at the end of its readiness log line, and returns to ready when the schema is current again. When the database it finds is a replaced one, it exits instead, with code 3 and the database was replaced under this process in its log, so no cache outlives the old database; the restarted pod waits as above with a database replaced reason.

A replaced database

Each Pro release records an installation id: the migration Job writes it into the database (the one-row installation table) and into a ConfigMap named <release>-installation that it creates in the release namespace, and the control plane reads it from the ConfigMap. When the database answers with another id, with none, or with no schema at all, the release does not treat it as a fresh install:

• the migration Job migrates nothing and fails with exit code 8 (database replaced: ...), so helm upgrade fails;
• a control plane booting on it waits, not ready, with database replaced: no schema, no installation id or another installation id on /readyz, and dexaflow_schema_ready{reason="database replaced"} at 0;
• a control plane that was serving exits with code 3 and restarts into that wait.

This is what an emptied database looks like (an emptyDir Postgres moved to another node, a volume lost), as well as a restore from another installation’s backup and a database.url changed to another database. A restore of a backup of this same installation carries the same id and is not a replacement: it is checked as absent, behind or ahead as usual. The log line of the server and of the Job names both ids:

kubectl -n <namespace> get configmap <release>-installation -o jsonpath='{.data.installationId}'
psql "$DATABASE_URL" -c 'SELECT id FROM installation'

What to do depends on which database you want:

Changing installation.id rolls the control plane pods, so they read the new id. The acceptance applies once: the next replacement is refused again, with installation.id still set, until it is changed again. A new id is written only into a database that has none, so a database that belongs to an installation is never re-keyed by mistake.

helm uninstall leaves the ConfigMap in place (the chart does not render it), so reinstalling into the same namespace against an empty database is reported as replaced too. When that is a fresh start on purpose, accept it with a new id as above, or delete the ConfigMap before installing. A backup taken before this installation recorded its id (before 0.5.6) restores without one and is reported as no installation id: accept it with a new id.

On Lite, and on a chart install with migrations.enabled: false, no id is recorded and nothing is compared, unless your own migration step runs the dexaflow-migrate image with -installation-configmap, -installation-namespace and, to accept a replacement, -installation-id.

dexaflow lite boot

Running a DAG

UI / browser

Single sign-on (OIDC)

Reset paths (when in doubt)

dexaflow lite reset-password                             # fresh admin password, signs out old sessions (no sudo)
dexaflow lite --fresh                                     # start a session with nothing registered (DESTRUCTIVE)
dexaflow db reset --yes                                  # drop + recreate the Lite database (DESTRUCTIVE)
dexaflow uninstall                                       # remove ~/.dexaflow (binaries, managed Python, config)
dexaflow uninstall --purge                               # also remove the workspace (your DAGs!)

Logs

Task logs stream from the agent over gRPC to the control plane’s log sink and are served at /api/v2/dags/<dag>/dagRuns/<run>/taskInstances/<task>/logs/<try> (the UI’s drill-down), or from the CLI: dexaflow runs logs <dag_id> prints every task of the latest run, dexaflow runs logs <dag_id> <run_id> every task of that run, and dexaflow runs logs <dag_id> <run_id> <task_id> [--try N] [-f] one attempt of one task. Without a task, -f follows the run to its end. The sink directory is DEXAFLOW_LOGS_DIR (must be writable; dexaflow lite points it at a temp dir).

Control-plane logs are structured slog (JSON by default), one line per HTTP request with a request id — grep <request_id> correlates a UI click to its backend trace.

“the request could not be completed; see the server logs”

API error bodies never carry the underlying failure. A storage error carries the database’s own text — for Postgres, severity: message (SQLSTATE code), with the constraint, table and column names of the schema inside the message — and Dexaflow is multi-tenant, so that text stays server-side (CWE-209). The response says only which kind of failure it was:

The cause is on the control-plane’s request log line, under cause, alongside the request_id the response header X-Request-Id carries:

journalctl -u dexaflow-server -o cat | grep '"request_id":"<id>"' | jq '.cause'
# "upserting dag: ERROR: ... violates foreign key constraint \"dag_versions_dag_id_fkey\" (SQLSTATE 23503)"

kubectl logs deploy/<release> -c leoflow-server | jq 'select(.cause) | {path, status, cause}' lists every request that failed with a server-side cause.

“…: the request could not be completed; see the control-plane logs” in a task log

The agent inside a task pod talks to the control plane over gRPC, and the same rule applies there for the same reason: the pod runs your image and entrypoint, so it is not a place to put the database’s text. A failed agent RPC therefore reads as the step that failed plus a fixed phrase, for example:

fetching task spec: rpc error: code = Internal desc = loading task spec: the request could not be completed; see the control-plane logs

The step name is the diagnostic half and is always there. All sixteen: loading task spec, loading task spec for scope enforcement, recording state, recording reschedule, storing xcom, reading xcom, fetching variables, fetching connections, resolving pod to agent identity, minting agent token, opening log sink for task; logs will not be shipped, writing log line, flushing logs, receiving log line, receiving assignment request, receiving assignment ack. The cause is on the control-plane log line of the same name, carrying the attempt identity:

kubectl logs deploy/<release> -c leoflow-server \
  | jq 'select(.cause) | select(.run == "<run_id>" and .task == "<task_id>") | {msg, try, cause}'
# {"msg":"loading task spec","try":1,"cause":"loading run: ERROR: … (SQLSTATE 42P01)"}

On Lite the same line comes from the service journal rather than a pod — worth knowing because opening log sink against a filesystem sink is a characteristically Lite failure:

journalctl -u dexaflow -o cat \
  | jq 'select(.cause) | select(.run == "<run_id>" and .task == "<task_id>") | {msg, try, cause}'

Three messages are not redacted, because they are Dexaflow’s own words about your DAG rather than an infrastructure failure, and you can act on them:

agent_lost that is not a network fault: a refused agent credential

An agent whose bearer the control plane refuses gets Unauthenticated on every RPC, so its heartbeats stop counting and the attempt is eventually reaped as agent_lost, which looks exactly like a network partition. The control plane writes a WARN line for every refusal, rejected agent bearer token, with a reason that tells the cases apart:

For expired (and not_yet_valid) the line also names the attempt the credential belonged to (ti, run, dag, task, try, epoch) and when it lapsed (expired_at, expired_for), because the token’s signature verified. For every other reason the token’s claims are not trusted, so they are not logged. The token itself is never logged. The line is rate limited to ten per reason per minute, and the first line after a quiet minute reports how many were left out in suppressed; the metric dexaflow_agent_token_rejections_total{reason} counts every refusal.

kubectl logs deploy/<release> -c leoflow-server \
  | jq 'select(.msg == "rejected agent bearer token") | {reason, run, task, try, expired_at, peer}'

Observability

• Metrics: Prometheus at :9090/metrics (scheduler, dispatch, inline runner, undispatchable counters; ADR 0007 has the catalogue).

• Schema readiness: dexaflow_schema_ready is 1 when the database schema is one the binary can serve, and 0 while it is not, with a reason label (schema absent, schema behind, schema dirty, schema unreadable, database replaced; the versions are on /readyz). dexaflow_schema_wait_seconds is how long the process has been waiting, 0 when it is ready. Every role exports both. With the Prometheus Operator, metrics.prometheusRule.enabled renders the alert below; without it, add the same rule to your Prometheus:

  - alert: DexaflowSchemaNotReady
    expr: dexaflow_schema_ready == 0
    for: 5m
    labels:
      severity: warning
  

• Tracing: OpenTelemetry — set DEXAFLOW_OBSERVABILITY_OTEL_ENABLED=true and …_OTEL_ENDPOINT.

• Logs: structured slog (JSON by default), one line per HTTP request with a request id.

Observability ships from the first commit (it is not optional).