DexaflowTaskQueueWaitHigh
What DexaflowTaskQueueWaitHigh means and what to do when it fires.
Symptom. The p99 of dexaflow_task_queue_wait_seconds (from an attempt’s
queued_at to its first running report) of one tenant stayed above
thresholds.queueWaitP99Seconds (300 s) for 15 minutes.
Impact. That tenant’s tasks start late; schedules drift and SLAs on run duration slip.
First commands.
histogram_quantile(0.99, sum by (tenant, le) (rate(dexaflow_task_queue_wait_seconds_bucket[15m])))
dexaflow_task_oldest_waiting_seconds
histogram_quantile(0.99, rate(dexaflow_pod_pending_duration_seconds_bucket[15m]))
dexaflow_dispatch_queue_depth
kubectl -n <tasks-namespace> get pods --field-selector=status.phase=Pending
kubectl -n <tasks-namespace> get events --sort-by=.lastTimestamp | tail -30
Decide.
- Pods sit
Pending: the cluster lacks nodes or quota, or an image pulls slowly. Scale the node pool or fix the pull. dexaflow_dispatch_queue_depthgrows withdexaflow_dispatch_at_capacity_total: the dispatch pool is saturated; see Pools.- Only warm pool tasks wait: see Warm pools.