Troubleshooting & observability

Symptoms grouped by where they surface. Start with the diagnostics — most issues are one leoflow doctor away from a clear cause. New here? The Quickstart and Installation guides cover a clean first run; this page is where you land when one goes wrong.

First things to run

leoflow doctor                          # host check (OS, python, docker, k3d, kubectl, recommended tier)
leoflow version                         # version + commit + build date
tail -f /tmp/leoflow-lite.log           # the live boot log when you ran `leoflow lite` via lite-redeploy
journalctl -u leoflow-server -f         # Pro / systemd hosts

leoflow --version
leoflow-server --version
leoflow-agent --version
leoflow-mcp --version

Install & setup

leoflow lite boot

Running a DAG

UI / browser

Single sign-on (OIDC)

Reset paths (when in doubt)

leoflow lite reset-password --user admin@leoflow.local  # generate a fresh admin password (no sudo)
leoflow dev --fresh                                     # start a session with nothing registered (DESTRUCTIVE)
leoflow db reset --yes                                  # drop + recreate the Lite database (DESTRUCTIVE)
leoflow uninstall                                       # remove ~/.leoflow (binaries, managed Python, config)
leoflow uninstall --purge                               # also remove the workspace (your DAGs!)

Logs

Task logs stream from the agent over gRPC to the control plane’s log sink and are served at /api/v2/dags/<dag>/dagRuns/<run>/taskInstances/<task>/logs/<try> (the UI’s drill-down), or from the CLI: leoflow runs logs <dag_id> <run_id> <task_id> [--try N] [-f] (landing in v0.4.1). The sink directory is LEOFLOW_LOGS_DIR (must be writable; leoflow lite points it at a temp dir).

Control-plane logs are structured slog (JSON by default), one line per HTTP request with a request id — grep <request_id> correlates a UI click to its backend trace.

“the request could not be completed; see the server logs”

API error bodies never carry the underlying failure. A storage error carries the database’s own text — for Postgres, severity: message (SQLSTATE code), with the constraint, table and column names of the schema inside the message — and Leoflow is multi-tenant, so that text stays server-side (CWE-209). The response says only which kind of failure it was:

The cause is on the control-plane’s request log line, under cause, alongside the request_id the response header X-Request-Id carries:

journalctl -u leoflow-server -o cat | grep '"request_id":"<id>"' | jq '.cause'
# "upserting dag: ERROR: ... violates foreign key constraint \"dag_versions_dag_id_fkey\" (SQLSTATE 23503)"

kubectl logs deploy/leoflow -c leoflow | jq 'select(.cause) | {path, status, cause}' lists every request that failed with a server-side cause.

“…: the request could not be completed; see the control-plane logs” in a task log

The agent inside a task pod talks to the control plane over gRPC, and the same rule applies there for the same reason: the pod runs your image and entrypoint, so it is not a place to put the database’s text. A failed agent RPC therefore reads as the step that failed plus a fixed phrase, for example:

fetching task spec: rpc error: code = Internal desc = loading task spec: the request could not be completed; see the control-plane logs

The step name is the diagnostic half and is always there. All sixteen: loading task spec, loading task spec for scope enforcement, recording state, recording reschedule, storing xcom, reading xcom, fetching variables, fetching connections, resolving pod to agent identity, minting agent token, opening log sink for task; logs will not be shipped, writing log line, flushing logs, receiving log line, receiving assignment request, receiving assignment ack. The cause is on the control-plane log line of the same name, carrying the attempt identity:

kubectl logs deploy/leoflow -c leoflow \
  | jq 'select(.cause) | select(.run == "<run_id>" and .task == "<task_id>") | {msg, try, cause}'
# {"msg":"loading task spec","try":1,"cause":"loading run: ERROR: … (SQLSTATE 42P01)"}

On Lite the same line comes from the service journal rather than a pod — worth knowing because opening log sink against a filesystem sink is a characteristically Lite failure:

journalctl -u leoflow -o cat \
  | jq 'select(.cause) | select(.run == "<run_id>" and .task == "<task_id>") | {msg, try, cause}'

Three messages are not redacted, because they are Leoflow’s own words about your DAG rather than an infrastructure failure, and you can act on them:

Observability

• Metrics: Prometheus at :9090/metrics (scheduler, dispatch, inline runner, undispatchable counters; ADR 0007 has the catalogue).
• Tracing: OpenTelemetry — set LEOFLOW_OBSERVABILITY_OTEL_ENABLED=true and …_OTEL_ENDPOINT.
• Logs: structured slog (JSON by default), one line per HTTP request with a request id.

Observability ships from the first commit (it is not optional).