DexaflowReaperSettlingValveOpen

What DexaflowReaperSettlingValveOpen means and what to do when it fires.

Symptom. The scheduler decision reap_settling_valve_open was recorded: the leader never settled within twice the settling grace, and the reapers ran anyway.

Impact. Reapers act without the reconciler’s sweep behind them, so a task that is alive can be failed as lost. The cause (reconciler or informer broken) also stops outcomes from being settled from pods.

First commands.

kubectl -n <namespace> logs deploy/<release> --tail=500 | grep -E 'settling valve|reconcil|informer'
kubectl auth can-i list pods -n <tasks-namespace> --as system:serviceaccount:<namespace>:<release>

Decide.

  • The informer cannot list or watch pods (RBAC, API server errors): fix access; the next leader tick settles.
  • The reconciler sweep errors on the database: check the scheduler pool (dexaflow_db_pool_*{pool="scheduler"}).

See Scheduler resilience.