DexaflowTasksLost
What DexaflowTasksLost means and what to do when it fires.
Symptom. One reaper failed more than thresholds.lostTasksPerHour (10)
attempts in an hour as agent_lost, pod_lost, dispatch_lost or
warm_worker_lost.
Impact. Tasks fail or retry for infrastructure reasons, not their code.
First commands.
sum by (decision_type) (increase(dexaflow_scheduler_decisions_total{decision_type=~".*_lost.*"}[1h]))
rate(dexaflow_agent_token_rejections_total[1h])
kubectl -n <tasks-namespace> get events --sort-by=.lastTimestamp | grep -iE 'evict|oom|preempt' | tail -20
Decide.
agent_lostwith token rejections: a refused agent credential, see agent_lost that is not a network fault.pod_lostwith evictions or preemptions: node pressure or spot reclaim.dispatch_lost: pods never started; see DexaflowTaskQueueWaitHigh.warm_worker_lost_oomkilledrising: raise the task’s memory limit.