Scheduler resilience

How Dexaflow keeps the scheduler honest when something goes wrong: a process dies, an agent goes silent, a dispatch is lost in flight. The control plane ships five reapers — small, single-purpose backstops that turn stuck state back into observable terminal state, so the dashboard never lies about what’s actually running.

This applies to both editions: Lite (single-process) and Pro (multi-replica with leader election). The reapers run only on the leader — reaping writes state, and we want one writer across the fleet. On Kubernetes they run from the leader’s maintenance loop (every 30 s, after the pod reconciler’s sweep; see below). Lite runs its own maintenance loop with the subset of reapers that mean something without pods; see Lite: which reapers run.

Recovery SLAs

Every SLA above is a floor: the reapers run every 30 s, so detection lands up to one cycle after the threshold elapses. Worst case end-to-end: a mid-tick scheduler crash that leaves TIs queued is fully reaped within max(3 min, 5 min) + 30 s — the dispatch-lost reaper runs first, then the orphan-run reaper picks up the now-no-active-TI run on a later cycle.

Large backlogs: the candidate lists are paged

The pod-lost and dispatch-lost reapers read their candidates one page of 100 rows per cycle, oldest first, and keep their place between cycles: each cycle asks for the page after the last row the previous cycle saw, and a page shorter than 100 rows sends the next cycle back to the start. The dispatch-lost threshold and the pod-lost grace are applied in the query, before the page limit, so rows still inside them never take a slot.

Before this, both lists always returned the 100 oldest rows. With 100 or more older attempts whose pods were alive (long tasks, or Pending pods during a capacity shortage), a newer attempt whose pod vanished or whose create never happened was never listed, kept its pool slot and its max_active_tasks charge, and its run never finished (#1597). Now such an attempt is reached within ceil(N / 100) cycles, where N is the number of candidates ahead of it, so the SLAs above hold up to 100 candidates and grow by one cycle (30 s) per further 100. The agent-lost reaper orders by heartbeat staleness and is not paged. The Lite passes over running attempts (the credential ceiling and the never-heartbeated agent-lost check) page the same way.

Who enforces execution_timeout — and why the pod outlives it

A task that declares execution_timeout_seconds and runs in its own pod has two clocks that could kill it, and only one of them can explain itself. (A task served by a warm pool has only the agent’s: warm pods carry no activeDeadlineSeconds at all, and their per-attempt wall-clock bound is the warm worker’s attempt watchdog, derived from auth.max_attempt_credential_lifetime — so everything below about the kubelet racing the agent does not apply to them.) The agent owns the semantic timeout: at the declared boundary it SIGKILLs the task’s whole process group, not just the process it started, and reports execution_timeout: task exceeded Ns limit.

The group matters. Signalling only the direct child left the timeout defeatable by anything that outlived it — a backgrounded daemon, a nohup, a shell that forks — and worse, a descendant that inherited the task’s stdout kept the agent’s own wait open, so the timeout fired and nothing died (#943). The group kill is un-trappable on purpose: a task that handles SIGTERM and declines to exit would otherwise be exempt from its own declared limit.

Three consequences worth knowing:

• A descendant that escapes the group (by calling setsid) is not reached. The agent stops waiting for it after a bounded 10s and reports the timeout anyway, rather than hanging. If your pod declares a terminationGracePeriodSeconds shorter than that, the kubelet can still win the race and you get the degraded reason described above.
• Output written before the kill is kept. The tail still in flight when the group dies is drained within the same bound; only a descendant holding the pipe past it can cost you the last lines, and that is logged.
• A task with no declared timeout that backgrounds a process now costs an extra 10s at exit while the agent waits out the inherited pipe. Previously that case hung indefinitely. The task pod’s activeDeadlineSeconds is only the backstop for an agent that can no longer enforce anything (crashed, wedged, partitioned); when the kubelet gets there first, the pod is gone and the failure reason degrades to what Kubernetes saw from outside, which cannot name a timeout.

The two clocks do not start together. The kubelet counts from the pod’s status.startTime, stamped before the image pull. The agent’s own clock starts much later, inside its execute step — after the image pull, the volume attach and mount, container start, its token bootstrap and exchange, the gRPC dial, Register, GetTaskSpec, the environment build (XCom fan-in and secret resolution, possibly over calls to an external secret backend) and its RUNNING report, whose retry has no budget of its own. On a cold node the image pull dominates all of it. So the pod deadline is deliberately set longer than the declared timeout:

activeDeadlineSeconds = execution_timeout_seconds
                      + 3 min   (startup headroom = the dispatch-lost threshold)
                      + the pod's terminationGracePeriodSeconds
                        (default 30 s, and capped at 60 s here)

The headroom is the dispatch-lost threshold on purpose: that is the window in which the control plane still presumes healthy startup and defers reaping, so the kubelet should not be spending the author’s execution budget inside it. Note what the threshold is not — a bound on how long startup may take. The dispatch-lost reaper checks the pod before failing anything and defers while it is Pending or Running, so a slow image pull is not reaped as a lost dispatch; the threshold is where the control plane stops assuming a dispatch landed and starts looking. The one bound on a slow start is the attempt credential: under the env-var transport it is minted at dispatch with a 10 min TTL, so a pod still Pending 8 min after it was queued is re-placed with a fresh credential as start_expired rather than left to start with a dead one (#1126).

The termination grace is added on top rather than folded in — it covers the shutdown tail (stopping the child, delivering the report), not the startup head. Only up to 60 s of it is added, however much the DAG declares: termination_grace_period_seconds is unvalidated, and adding an hour of it verbatim would put the pod’s deadline an hour past the declared timeout while the kubelet grants that same hour of SIGTERM grace again on top. The pod spec still carries the declared value verbatim; only this arithmetic is capped. The sum is also clamped to 2147483647 (math.MaxInt32), the largest activeDeadlineSeconds the apiserver accepts — otherwise a declared timeout near that bound would be rejected at pod CREATE rather than merely never reached.

Two consequences worth knowing. A pod may outlive its declared timeout by a few minutes when its agent is dead — that is the backstop doing its job, and the reapers above still settle the task instance on their own schedule. And a pathological startup (a cold node pulling a multi-gigabyte image, a throttling registry, or a control-plane outage spanning the RUNNING pre-flight) can still exceed any fixed headroom; the task then fails with the kubelet’s generic reason rather than the timeout diagnosis. If you see that, the image pull is the thing to fix — and it is worth measuring status.startTime → the agent’s task started log line on a cold node.

The pod deadline of a task that declares no execution_timeout is a different mechanism: a floor derived from auth.max_attempt_credential_lifetime (see below), which takes no headroom because no clock inside the pod races it.

Cadence: reconcile, then reap

The reapers do not run from the scheduler’s 1 s tick. They run from the leader’s maintenance loop, which every 30 s performs one ordered cycle:

1. the pod reconciler sweeps the task pods, recovers each finished pod’s durable outcome record (a success whose report was lost during an outage is recorded as a success — ADR 0052), and garbage-collects old finished pods;
2. then the five reapers run.

Each phase runs under its own budget of one interval (30 s): a sweep hung on a slow apiserver cannot starve the reap, and a reap pass over a large namespace cannot starve the sweep it depends on. Overrunning a budget is a load signal, logged at WARN with the budget, not an error; a cycle can take up to two intervals and the ticker coalesces the ticks it overruns.

The order is structural, not a matter of timers lining up. Before this, the reapers ticked at 1 s and the reconciler at 30 s on an independent clock, so a pod-lost verdict could land on a pod the reconciler would have recovered as succeeded at its next sweep. The cost is up to 30 s of extra detection latency on top of thresholds of 60 s–5 min; reaping is a backstop, never the primary path, so that trade is the right one.

The leader-settling gate

A control-plane restart manufactures the very signals the reapers act on: every in-flight heartbeat looks stale (the receiver was down), a task pod that finished during the outage looks lost (its terminal report found no server), a run looks quiet. So after this instance acquires leadership no reaper fires until the leader has settled — all three of:

• the settling grace (180 s, twice the agent-lost threshold) has elapsed since leadership, giving the whole fleet time to re-heartbeat;
• the pod informer cache has synced, so the fleet view is complete;
• a reconciler sweep has completed under this leadership, so every finished pod’s true outcome has been recovered before anything is declared lost.

One gate, at the entry of the reaper pass, covers all five reapers (the warm-worker-lost reaper too: delaying it by the grace only postpones recovering a dead worker’s attempts; pool refill is a separate loop and is not held). Measured from leadership acquisition, so a re-election resets it. While it holds, every cycle records reap_settling_skip.

Liveness valve. A gate that could hold forever would trade “reap wrong” for “never reap”. If the leader has not settled after 2 × grace (360 s) — the reconciler cannot list pods, the informer never syncs — the reapers proceed anyway, with a WARN log and a reap_settling_valve_open decision on every cycle the valve stays open. By then the reconciler has had at least four cycles, so a valve that opens means the sweep really is broken; treat it as an alert.

Why opening it is safe. The usual reason a sweep never completes is an apiserver that cannot be read — unreachable, unauthorized, throttled. The reconciler’s pod LIST and each reaper’s own pod-presence LIST hit that same apiserver, so that failure usually denies every pod-dependent reaper its authorization too: pod-lost and dispatch-lost defer (pod_lost_pod_query_error, dispatch_lost_pod_query_error) and the warm-worker-lost reaper aborts its cycle with zero marks. They fail closed on their own, valve or no valve. The two reads can still diverge, though — a broad namespace-wide LIST can time out where a narrow, server-side-filtered one succeeds, an informer that never syncs needs watch where the reapers need only list, and API Priority and Fairness can starve the heavy request while the cheap one gets through. And because the valve is designed to open, it is never the only guard: the pod-lost reaper fails a task only when the attempt has no pod object at all, a state no grace, cadence or election timing can manufacture. A finished pod is a present pod, and stays the reconciler’s.

The server validates the timing ladder these depend on at boot and refuses to start if a constant was moved out of order: heartbeat (15 s) < agent-lost threshold (90 s) < settling grace (180 s) < attempt token TTL (10 min), and 2 × maintenance interval (60 s) < settling grace, so at least two whole reconcile-then-reap cycles complete inside the grace. On Kubernetes it also checks task termination grace (30 s) + 2 × maintenance interval < infra confirmation valve (2 min) < orphan threshold (5 min) (see below).

Infra marks wait for confirmation (Kubernetes)

The agent-lost, pod-lost and dispatch-lost reapers judge from the control plane’s side: a silent heartbeat or a missing pod. That is a guess about what the task did, and the task may in fact have finished and written a durable SUCCESS record that the reconciler has not read yet. So on Kubernetes an infra mark is provisional: the task instance is failed with last_failure_kind = infra, but infra_confirmed_at stays empty and the planner treats the task as still active. It neither re-places it nor lets a downstream task fail on it.

Each reconcile sweep then looks at the pods of the exact attempt the mark names (the same try and attempt epoch). It confirms the mark when no pod of that attempt is left, or when every task container of the attempt has terminated without a SUCCESS record. It reads the container’s terminated state, not the pod phase, because a pod stopped in place can report Failed while its container still runs. A container still running, or still inside its termination grace, holds the mark back. Once the mark is confirmed, the planner re-places the task, or fails it when the infra budget is spent, exactly as before.

A broken reconciler must not hold a task forever, so the planner also acts on a mark once the infra confirmation valve (2 minutes after the mark) has passed without a confirmation, and meters it as infra_confirm_valve_open. The ladder above keeps that valve longer than a container’s termination grace plus two sweeps, and shorter than the orphan threshold, so a run waiting on a confirmation is never reaped as orphaned.

A durable SUCCESS wins over the guess

If the attempt the mark names did finish, its task container leaves a durable SUCCESS record, and the reconciler reads it on its next sweep. While the mark is still provisional and inside the valve, on a run that is still running, the reconciler settles the task success instead and clears the infra kind. It appends outcome recovered from the durable record over agent_lost (or pod_lost, dispatch_lost, start_expired) to the attempt’s log and counts it in dexaflow_reconcile_infra_override_total{mark}. The task does not run again, and its downstream tasks run as they would have.

The override only ever applies to the attempt the pod’s labels name (the same try and attempt epoch), so a superseded pod cannot settle its replacement. The record itself also names the execution that wrote it: a record whose epoch differs from the pod’s label counts as no record, and the pod settles by its phase. A FAILED record never overrides a mark: the reaper’s own teardown makes the agent write one. A task a user marked failed is a verdict, not a guess: the mark-state action clears the infra kind, so neither the re-place nor the override touches it. A record that arrives after the mark was confirmed, after the valve opened, or after the run finalized is ignored; clear the task to run it again.

On Lite there is no reconciler and no pod to wait for: the mark is confirmed when it is made, and nothing changes. Marks made before the upgrade count as confirmed. Every rail that starts a new execution of the task (retry, clear, re-place, reschedule) clears infra_confirmed_at again.

A pod the cluster stopped is an infra failure

A task pod can end because the cluster stopped it rather than because the task failed: the kubelet shuts the node down gracefully (a spot or preemptible reclaim), evicts the pod under node pressure, the scheduler preempts it, or the kubelet refuses it at admission. ADR 0051 puts these on the infra rail, and the pod reconciler settles them there: the task instance is failed with last_failure_kind = infra and an error_message that starts with pod_disrupted:, and the planner re-places it on the infra budget. The task’s own retries are not spent, so a task with retries: 0 survives a node reclaim.

The reconciler reads two signals from the terminal pod:

• the DisruptionTarget condition with status True (reasons TerminationByKubelet, PreemptionByScheduler, EvictionByEvictionAPI, DeletionByTaintManager);
• the pod reason Evicted, Shutdown, NodeShutdown, Terminated, OutOfcpu, OutOfmemory or UnexpectedAdmissionError, which older kubelets and admission refusals set without the condition.

Two things still win over them. A durable SUCCESS or RESCHEDULE record settles as before (the task finished before the node went away). And an OOMKilled task container stays a task failure even on a disrupted pod, because the kernel enforced the task’s own memory limit. The same holds for an Evicted pod whose kubelet message names a limit of the pod’s own (a container over its ephemeral-storage limit, the pod over its containers’ total ephemeral-storage limit, or an emptyDir volume over its sizeLimit): the task outgrew what it asked for, so it is a task failure that spends a retry, and the error_message quotes the kubelet’s message. Only an eviction for node pressure (“The node was low on resource”) is infra. A FAILED record does not win: the agent writes one when it catches the SIGTERM of a node shutdown, and that record says the task stopped, not why.

The settle is confirmed at once, since the reconciler has just read the terminal pod, so it does not wait for the infra confirmation valve. A pod that vanished with its node instead is still the pod-lost reaper’s pod_lost; both end on the same infra rail. The infra budget bounds the re-places, so a task that keeps landing on reclaimed nodes ends in a terminal infra failure instead of looping.

A start the control plane never answered is an infra failure

The agent retries the calls it makes before the task’s code runs (registering, reading the task spec, upstream XComs, variables and connections, and the running report) through a control-plane restart, a Postgres failover or a Redis restart, for up to 2 minutes (control-plane HA). When the outage outlasts that, the attempt ends before the task’s code ran, so the task is not at fault (ADR 0051, an unexecutable attempt). On a dedicated pod the agent then writes an outcome record marked as infra, with no exit code and a fixed reason, reports nothing, and exits. The pod reconciler settles that record on the infra rail: the task instance is failed with last_failure_kind = infra and an error_message that starts with control_plane_unavailable:, confirmed at once like a disrupted pod, and the planner re-places it on the infra budget. A task with retries: 0 therefore survives an outage that lands on its start.

Only an answer that never came counts: Unavailable, DeadlineExceeded, Aborted, or Internal from an older control plane. A refusal (for example PermissionDenied on the spec read of a superseded attempt) and a failure of an external secret backend keep their task-failure settle. The reconciler honors the infra marker only on a record with an agent-authored reason and no exit code, the shape the agent writes. A task can write its termination message too, so a forged marker could at most buy a re-place on the infra budget, which a task can already get by killing its agent (agent_lost), and the infra budget bounds it.

On Lite the agent writes no record and reports nothing, so the attempt stays queued until the dispatch-lost reaper settles it as dispatch_lost, on the same infra rail. A warm-pool attempt keeps reporting failed: its pod serves other attempts, so no reconciler reads a record for it.

A pod create that fails

A dispatch can fail before any reaper is involved: the pod CREATE itself returns an error. What the scheduler does depends on what the error says, the same on the tick and in the buffered dispatch workers (scheduler.dispatch.buffer_size above 0):

None of these spends a task retry: a dispatch failure is the substrate’s, not the task’s. dispatch_failed is terminal, so a task with retries: 2 that failed to dispatch does not run again on a new try; clear it once the cause is fixed, which restores its retries. Before #1606, an outage longer than about 2m35s (a managed control plane upgrade, an etcd leader change) failed every task due during it, and the failure then fell onto the task’s retries.

The start of a task’s outage is kept on its row (task_instances.dispatch_unavailable_since), so a scheduler restart or a leader change does not restart the 15 minutes. It is reset with the attempt counter: by a retry, an infra re-place, a clear, or a reschedule.

A create whose answer was lost may still have made a pod. Retrying is safe anyway: the re-offer moves the task to a new attempt epoch, so that pod’s agent is told to stop at its RUNNING report, before it starts the task’s code. If that pod reported RUNNING first, the task is running there and the re-offer leaves it alone. Only one of the two pods ever runs the task; the other one’s agent exits with an error without running it.

The scheduler logs every re-offer at WARN (Kubernetes API unavailable; re-offering the task) and the give-up at ERROR; on the tick they are also counted as dexaflow_scheduler_decisions_total{decision_type="dispatch_unavailable"} and {decision_type="dispatch_unavailable_exhausted"}. A rise of the first during a control plane upgrade is expected; any of the second means the API was unavailable to a task for 15 minutes.

Lite: which reapers run

Lite (the subprocess executor behind dexaflow lite) has no pods, no pod informer and no pod reconciler. Its agent is a host process, detached from the server so it survives a server restart. Before #916 Lite ran no reaper at all, so a run whose agent died without reporting stayed running forever. Lite now starts a maintenance loop of its own: every 30 s, on the leader only, it runs the reaper pass under the same leader-settling gate (the 180 s grace since leadership; the informer and sweep conditions do not exist in Lite and are satisfied). There is no reconcile phase because there are no pods to sweep.

Why the process gate. On Kubernetes a reaper that fails a TI also deletes its pod, so the abandoned agent stops. Lite has no equivalent: it never kills an agent. The infra re-place that follows an agent_lost or dispatch_lost keeps the try number, so if the old agent were still alive it could have its RUNNING report accepted and run user code next to the new agent (#911). Lite therefore reaps an attempt only when its agent is provably dead. The server records each spawned agent’s PID, one file per (run, task, try) attempt, under $TMPDIR/dexaflow-agent-pids-<uid> (a directory only that user can write; the server refuses one that is a symlink, owned by someone else, or writable by others); the record lives on disk so a restarted server still sees the agents it spawned before the restart. The reaper probes the PID with signal 0: alive defers (agent_lost_process_alive, dispatch_lost_process_alive), a probe or read error defers (*_process_query_error), and a missing record or a dead PID lets the reap proceed. If the server cannot write the record when it spawns an agent, it stops that agent and fails the dispatch rather than run an agent the reapers cannot see.

The task’s process group counts too. The agent runs the user task as the leader of its own process group, so a task can outlive an agent that is killed outright (kill -9, a crash, the OOM killer): it is reparented and keeps running. Only the agent learns the group id, so the agent writes it into the same directory as soon as the task starts, one .pgid file per attempt next to the agent’s .pid file, together with its own PID and the group leader’s start time. A task whose group the agent cannot record is stopped and its run fails, the same rule as for the agent record. An attempt reads alive while its agent is alive or while any process of its recorded task group exists (signal 0 to the whole group; a member owned by another user still counts as alive). A group id whose leader PID now belongs to a process with a different start time names an unrelated group (the OS never reuses a PID while it is still a group id, so nothing of the recorded group is left), and reads dead.

An agent that dies before its first heartbeat. Agent-lost normally judges only a running TI that has heartbeated at least once, and on Kubernetes the window before the first heartbeat belongs to pod-lost, which has no signal in Lite. So in Lite, and only there, agent-lost also looks at a running TI that never heartbeated and entered running longer ago than the agent-lost threshold (90 s): it is failed as agent_lost (agent_lost_never_heartbeated) only when its agent and its task process group both read dead. Anything alive defers (agent_lost_never_heartbeated_process_alive), a liveness error defers (agent_lost_never_heartbeated_process_query_error), and such an attempt’s orphaned task group is never stopped, only waited for. The Kubernetes path is unchanged.

An orphaned task is stopped before the attempt is re-placed. When an attempt reads alive only because of its task group, and the agent that recorded that group is confirmed dead, the reaper stops the group before it reaps: SIGTERM to the whole group, up to 10 s to exit, then SIGKILL, then up to 5 s more. Only once no process of the group is left does the reap go ahead (agent_lost_orphan_stopped, dispatch_lost_orphan_stopped), so the re-placed attempt never runs beside the old copy. The server signals only a group recorded in its own record directory, and only when the group’s leader is still the very process that was recorded (same PID, same start time); a group it cannot verify that way (its leader exited while other members run on, or the start time could not be read) is never signaled, and the attempt keeps deferring as alive until the group exits. A stop that fails defers too (*_orphan_stop_error). This runs on Linux and macOS; elsewhere a task group cannot be probed, so the reapers keep deferring.

What this does not cover, plainly:

• An agent that is alive but wedged (the process exists, its heartbeat stopped) is not reaped by Lite before the credential ceiling (see below). Stop the process yourself; the next cycle then reaps the TI as agent_lost.
• PID reuse. If the agent died while the server was down and the OS hands its PID to an unrelated process, that attempt reads alive and is deferred until that process exits. This can only delay a reap, never cause a false one.
• A task process that leaves its process group. A descendant that calls setsid (or is handed to a service manager) is outside the recorded group, so it neither keeps the attempt alive nor is stopped with the orphan. This is the same boundary the agent’s own execution timeout has.
• An agent killed in the instant between starting its task and recording the task’s group. That task has no record, so it is invisible to the reapers, exactly as before. The window is the few microseconds between the fork and one small file write.
• A record directory that changes across a restart. The record is found through the server’s $TMPDIR; a server restarted with a different TMPDIR (or after the temp directory was cleaned) sees no record for agents it spawned earlier and treats them as gone. That only matters for an agent that is also silent past the agent-lost threshold or still queued past the dispatch-lost threshold.
• A task that hangs inside a live agent is the agent’s own execution_timeout_seconds to stop, exactly as on Kubernetes. Past auth.max_attempt_credential_lifetime the reaper stops it in any case, the way the pod’s activeDeadlineSeconds floor does on Kubernetes.

Tuning the thresholds

The thresholds and the settling grace are build-time constants (executor.DefaultReaperConfig), not operator knobs: they sit on the ladder above, and the boot-time validator exists precisely because moving one out of order turns a restart into a false reap. The one operator-tunable rung is auth.max_attempt_credential_lifetime, which must stay above the attempt token TTL; boot fails naming the key if it does not. Setting it non-positive is accepted but disables the renewal ceiling, the task-pod activeDeadlineSeconds floor and the warm-pool attempt watchdog together, so boot logs a WARN naming the key.

An attempt that outlives the credential ceiling fails

Past auth.max_attempt_credential_lifetime the control plane stops renewing an attempt’s credential, the last renewed token runs out one attempt token TTL (10 min) later, and the agent’s heartbeats stop. On Kubernetes the pod activeDeadlineSeconds floor or the warm-pool attempt watchdog usually ends the attempt at the ceiling itself, and its heartbeats stop the same way. The heartbeat reaper then sees a silent attempt. If the attempt went silent past the ceiling, that is, its last heartbeat came after it had been running for the ceiling less two heartbeat intervals (30 s), the reaper fails it with credential_ceiling: attempt outlived auth.max_attempt_credential_lifetime instead of agent_lost. The two intervals cover the last renewal, which can land up to one interval before the ceiling. The pod deadline counts from pod start, before running, so an attempt it ends is normally settled as a task failure by the pod reconciler, which reads the failed pod, before the reaper looks at it. A credential_ceiling failure is a task failure: the task’s retry policy decides what happens next, exactly as for a task that exited non-zero. It is not an infra mark, so the planner does not re-place it with a new epoch and a fresh credential, it is never provisional, and a durable SUCCESS record does not override it.

Before this, the silence read as a lost agent, and since agent_lost re-places off the retry budget, a task longer than the ceiling plus the token TTL was re-run from the start each time instead of failing (#1461). An attempt that finishes before the ceiling plus the token TTL is unaffected, and an attempt that goes silent before it reaches the ceiling is still agent_lost, even when the reaper only gets to it after the ceiling: a node that dies at minute 59 of a 60 minute ceiling is re-placed as a lost agent. The reaper judges by the attempt’s last heartbeat measured from its running transition, not by the time of the reap. The settling gate, the destructive gate, the attempt pin and the pod teardown are the same as for agent_lost. A non-positive ceiling disables the check along with the rest of the ceiling.

In Lite the reaper also ends the attempt at the ceiling (#1511). There is no pod deadline in Lite, so before this an attempt past the ceiling was failed only once its task process exited on its own: a task that never exits kept running with a credential that was no longer renewed, and the try log could show task succeeded right before killed: credential_ceiling. Now, on every reaper pass, a running attempt that entered running longer ago than the ceiling and whose agent or task process is still alive is failed with the same credential_ceiling reason (a task failure, so its retry policy applies), and then its task is stopped: SIGTERM to the task’s whole process group, up to 10 s to exit, then SIGKILL and up to 5 s more, the same escalation used for an orphaned task. The age counts from the running transition, which comes after the dispatch the credential ceiling counts from, so the credential has always stopped renewing by then. The attempt is marked first, through the same guarded write as above (running only, pinned to the attempt), so whichever of the mark and the agent’s own terminal report lands first decides the outcome and the other changes nothing. Once marked, the agent’s next heartbeat or report is told to terminate. The killed: credential_ceiling line is written after the task and its agent have exited, so it is the last line of the try log. Like the orphan stop, the server only signals a task group it recorded and can verify; it never signals the agent itself. Unlike Kubernetes, where a declared execution_timeout longer than the ceiling sets the pod deadline, Lite stops the attempt at the ceiling regardless: past it the attempt could not report its result anyway, and Lite has no durable outcome record to recover it from. This runs on Linux and macOS, where a task group can be probed.

The defaults are conservative on purpose: too-tight thresholds risk reaping a legitimately slow dispatch (Kubernetes pod-pull latency under contention) or a busy agent.

The “do no harm” rule

Each reaper requires a positive observable signal before failing anything:

• TI heartbeat reaper — only fires on TIs that did heartbeat at least once and then went silent. A TI that never heartbeated (e.g. a pod that never started, so no agent ever reported) is left alone. “At least once” means in the current attempt: every rail that starts a new execution of a task (retry, clear, infra re-place, reschedule re-dispatch, warm requeue, dispatch-failure backoff) clears the previous attempt’s heartbeat, so a new attempt is not reaped in the interval between reporting running and its first heartbeat. In Lite, a TI that never heartbeated is judged only on the positive signal that its agent and its task process group are both gone (see “Lite: which reapers run”).

• Dispatch-lost reaper — requires a non-zero queued_at older than the threshold AND, on Kubernetes, confirmation that no live pod for the TI exists. If a pod for the TI is Pending/Running, the dispatch actually landed and the node is just slow to pull the image (a cold-node false positive, #461) — the reaper defers. If pod liveness can’t be determined (K8s API error), it also defers. A TI without a queued_at stamp is too poorly observed to reap. The one exception is a pod still Pending once the attempt credential minted at dispatch is within 2 min of running out (env-var transport only): its agent could only start with a credential about to lapse, be refused, stream no log and fail the attempt against the user’s retries. The reaper reads the pod live (the presence cache cannot tell Pending from Running), and only for a Pending pod with no Running one marks the TI start_expired, an infra failure re-placed off the retry budget, and deletes the pod. A read error defers, as above.

• Pod-lost reaper — requires a running TI past its 60 s liveness floor AND a live read confirming no Pending/Running pod exists for exactly that attempt. A cached “pod present” defers without an API call; a cache miss is never trusted and falls through to the live read; a read error defers. And, like every reaper, it waits for the leader-settling gate — so a pod that finished during an outage is recovered by the reconciler, not failed.

• Warm-worker-lost reaper — requires a live LIST of the warm pods (not the cache) showing the bound worker gone; a LIST error aborts the pass with zero marks. It never deletes a pod: a warm worker outlives its attempts.

• Orphan-run reaper: requires state = 'running' AND no live TI on the run: every TI must be settled (success/failed/skipped/ upstream_failed) or never started (none). A run with any TI in scheduled/queued/running is left alone (the dispatch-lost reaper unblocks this case by failing the stuck queued TIs first, so a later cycle sees no active TIs). So is a run whose TI is parked waiting for the scheduler to bring it back: up_for_retry during its retry_delay, up_for_reschedule between the pokes of a reschedule-mode sensor, or the reserved deferred state. Those states stamp no fresh activity timestamp, so without this rule a retry_delay or poke_interval of 5 minutes or more would get a healthy run failed as orphaned. A TI in none does not keep the run alive: once its upstreams settle, the next scheduler tick decides it. Releasing a TI back to none for another attempt (a retry, a reschedule poke, an infra re-place, an operator clear) clears its per-attempt timestamps, and it only becomes scheduled on the next tick; the release stamps released_at (migration 036), which counts as run activity, so the run never looks orphaned in that tick.

  The list is only a snapshot, so the reap re-checks the whole predicate atomically: it share-locks the run’s TIs without waiting, then fails the run only if it is still running, still has no live TI and its last activity is still older than the threshold. If a TI moved in between (say a retriable failure went to up_for_retry, or a TI was scheduled), fresh activity landed, or another transaction is writing one of the run’s TIs at that moment, the reap is a no-op: nothing is written, no pod is torn down, and the reaper records orphan_reap_noop.

Tearing down the reaped task’s pod

Failing a TI in the metadatabase is not enough on its own: a reaped task’s pod can still be running user code, which breaks at-most-once execution if that work commits or a retry runs it again (#474). So, after the durable DB transition, each reaper tears the pod down:

• The heartbeat, dispatch-lost and pod-lost reapers delete exactly the reaped attempt’s pod, pinned by its (run-id, task-id, try-number) labels and its leoflow.io/attempt-epoch label. A retry dispatches a new pod with a new try-number, and every other new execution of the same try (an infra re-place, a reschedule poke, a repeated dispatch) carries a new attempt epoch, so a newer live attempt can never be the one deleted (#901). A pod created before 0.5.1 has no epoch label and counts as epoch 0. Each delete names the pod the reaper listed and pins its UID, so a pod created between the list and the delete is never touched. The reaper’s mark is pinned to the same (try, epoch), so a row re-placed between the list and the mark is left alone, and the pod-presence check that defers a reap only counts the attempt’s own pod.
• The task-id label is lossy: it is lowercased and _ and . become -, so task ids such as load_a and load-a, or Load and load, share one value. Every task pod therefore also carries its exact task id in the leoflow.io/raw-task-id annotation, and the teardown, the pod-presence check (live and cached) and the reconciler’s infra confirmation only act on a pod whose annotation names the attempt’s own task. Tearing down one of two such tasks no longer stops the other’s live pod, and a task with no pod no longer reads its sibling’s pod as its own (#1598). A pod created by a release before 0.5.4 has no such annotation and is matched on its labels alone, as before, until it finishes.
• The reconciler settles a finished pod’s outcome only against the attempt named by that pod’s labels, try and epoch. A superseded pod’s success record can no longer settle the replacement that shares its try (#1130). The labels are written by the control plane, and the task cannot change them.
• The orphan-run reaper deletes every pod of the abandoned run (the run-id is unique per run, so no other run’s pod can match).
• A pod that already reached a terminal phase (Succeeded/Failed) is skipped, not deleted (#928). It has no container left to stop, so deleting it would buy nothing and cost the durable outcome record on its termination message — the only evidence the reconciler can settle the attempt from (ADR 0052). Collecting those pods is the reconciler’s job: it settles each one, then garbage-collects it once it ages past the 10-minute grace. The skip is enforced once, at the teardown itself, so it holds for every reaper — including the heartbeat reaper, which fires on heartbeat staleness alone at 90 s and reads no pod state, and the orphan-run reaper, which applies it per pod inside the run. No reaper’s decision changes: a TI the reaper marked stays marked. Look for reap teardown: task pod is already in a terminal phase at INFO to see which pods were left behind; a failed delete is the *_pod_delete_error decision below, which is a different thing.
• A started pod is stopped in place, not deleted. The teardown lowers its activeDeadlineSeconds to 1 second (the deadline counts from the pod’s start, so it has always elapsed). The kubelet stops the containers with the pod’s normal termination grace, the pod goes Failed with reason DeadlineExceeded, and the pod object stays, with the task container’s outcome record. The reaped attempt stops running user code exactly as before (#474); if it had in fact finished, the reconciler can still read its SUCCESS record. A pod that never started (no start time) has no container and no record, and a Pending pod can still start the task, so it is deleted. The executor Role needs patch on pods for this; if the patch is refused (a hand-maintained Role, an admission webhook rejecting pod updates), the teardown deletes the pod as before and meters reap_teardown_delete_fallback.
• A pod in phase Unknown is torn down. It may still be running a container, which is the case teardown exists for, and the reconciler treats Unknown as non-terminal — it neither settles nor collects it — so nothing else would.
• Belt and suspenders: the control plane also answers a stale agent ReportState/Heartbeat — one whose attempt no longer matches the live row — with should_terminate, so a reaped-but-still-alive pod that we couldn’t delete (e.g. during a K8s API outage) cancels its own work. The “stale” test is exactly the source-state, try_number and attempt-epoch guard the state write already uses (#467, #911): the report applies for the live, matching attempt, so a live execution is never told to stop.

These teardown steps are best-effort and off the critical path: a delete failure is logged and metered but never undoes the DB reap, and the pod’s own activeDeadlineSeconds plus the reconciler’s GC remain backstops. In Lite (subprocess executor) there are no pods and no teardown: the reapers only fail an attempt whose agent process is already dead (see Lite: which reapers run), so there is nothing left to stop, and a run failed by the orphan-run reaper stops any agent still attached to it through the should_terminate signal.

A terminal pod the teardown skips is collected by the reconciler, so a reconciler that is not sweeping leaves those pod objects behind. That is not a new dependency: every normally-finished task pod has always been the reconciler’s to collect, and the reap-time delete only ever covered the subset belonging to a reaped TI or run. A terminated pod holds no node CPU or memory — it costs an API object and a slot against any count/pods quota — and Kubernetes’ own terminated-pod GC (--terminated-pod-gc-threshold, 12500 by default) is the cluster-level floor under it. A reconciler stuck for longer than that shows up first as reap_settling_valve_open, which is the label to alert on.

The load-bearing invariant

Recovery is bounded by the slowest reaper that applies, not the fastest — usually fine, sometimes worth tuning (ADR 0031). The invariant that governs every reap decision is never fail or tear down the live current attempt — only one that is genuinely stale or lost, and never destroy the evidence of one that already finished. It is preserved end-to-end: the DB transitions are guarded on source state (WHERE state IN (...)), pod deletes are pinned to the exact (run, task, try) reaped and skip a pod already in a terminal phase, the dispatch-lost reaper defers whenever a pod is live or its liveness is unknown, and the should_terminate signal fires only when the reporting attempt has provably moved on. When in doubt, a reaper defers rather than reap.

Observability

Each reap action is metered as a scheduler decision. Watch these labels in your Prometheus dashboard:

A sustained non-zero rate on any of these is worth investigating — reapers are backstops, not the primary path; if they fire often, something upstream is broken.

Run start cap

Run finalize

Loop and dispatch latency

These collectors are written on every tick or dispatch (before 0.5.4 they were registered but stayed empty, #1186):

What’s NOT a scheduler concern

• Postgres unreachable — the scheduler’s Heartbeat() goes unhealthy; the /monitor/health endpoint surfaces it; runs queue up and resume when the DB returns.
• Agent’s task container OOM-killed — surfaces as a non-zero exit code through the agent (if it survived) or as agent_lost (if the agent went with it).
• K8s API outage — pods stay where they are; new dispatches fail at the executor layer and are re-offered for up to 15 minutes without spending their dispatch attempts, then fail as dispatch_failed (see A pod create that fails); the dispatch-lost reaper does NOT fail TIs during the outage — it cannot read pod liveness, so it defers (dispatch_lost_pod_query_error) rather than risk a false positive. Reap-time pod teardown is likewise skipped and retried once the API returns; a reaped-but-alive pod stops itself via the should_terminate signal on its next heartbeat.

See also: ADR 0009 (leader election), ADR 0031 (scheduler architecture).