Control-plane HA and disruption posture

A Pro control plane with one replica is not “stable until something breaks”. It is evicted as routine cluster housekeeping — and every eviction is a full restart. This page explains the restart window and what is at risk inside it, why running two replicas is the recommended production posture, the storage precondition that makes it safe, and how the chart guards the combinations that are not.

The restart window

When the single control-plane pod is evicted, the replacement has to:

1. Pull the image on whatever node it lands on (nothing cached — the old node is usually the one being drained).
2. Boot: connect to Postgres and Redis, start the API, gRPC and metrics listeners, pass readiness.
3. Take leadership: acquire the scheduler’s advisory lock (ADR 0009) and start the loop. Until the new leader has settled — the settling grace (180 s by default) has elapsed, the pod informer has synced, and the pod reconciler has completed a sweep under this leadership — no reaper fires, so a task that merely lost its control plane for the duration of the restart is not failed before the reconciler has recovered its durable outcome. One gate covers all five reapers; a liveness valve opens it after 2 × grace if the sweep never completes (scheduler resilience).

Measured on a production drill (EKS Auto Mode), that window is 48–80 seconds, dominated by the image pull. During it:

• Nothing dispatches. Queued task instances wait; scheduled runs slip.
• In-flight tasks lose their control plane. Agents keep running the task and retry their heartbeats and outcome reports until the server returns. The leader-settling gate on the reapers is what keeps that from becoming a false agent_lost / pod_lost failure; the server also validates at boot that the timing ladder this depends on holds (heartbeat < agent-lost threshold < settling grace < token TTL, and two maintenance cycles inside the grace), refusing to start if a knob was moved out of order. These bound the damage; they do not remove the window.
• Tasks starting or finishing in the window wait instead of failing. An agent that cannot read its task spec, variables, connections or upstream XComs at start, or deliver its return value and XComs at the end, retries with jittered backoff, so a restart of a single control-plane replica, a Postgres failover or a Redis restart does not spend the user’s retries. The bound is 2 minutes of wall time per phase, counted from the first try and including the time each try takes: all the reads before the task runs share one 2-minute budget, and all the output pushes share another. A try that hangs is cut at the end of the budget rather than at its own 30-second deadline, and the agent never sleeps past it, so a start against a control plane that answers nothing gives up 2 minutes after it began, however many upstream XComs it reads. The control plane answers a store outage (connection refused or reset, a failover’s admin shutdown, a full connection pool, a statement timeout, Redis loading or read-only) with gRPC Unavailable rather than Internal. The agent also retries an Internal answer to a state report for up to 2 minutes of backoff, which covers an older control plane during an upgrade. An outage longer than that still ends the attempt, but when it ends before the task’s code ran (the agent could not register, read its task spec or upstream XComs, or report running), the task does not pay for it: on Kubernetes the agent leaves an outcome record that the pod reconciler settles on the infra rail as control_plane_unavailable, and on Lite the unreported attempt is dispatch_lost. Either way the task is re-placed on the infra budget without spending one of its retries (scheduler resilience). An outage that outlasts the push of a finished task’s outputs still fails the attempt, since its code already ran.
• The API and UI are down. Every replica is the same binary, so one replica means one API.

Why eviction is routine, not exceptional

On a managed autoscaling platform the control-plane pod is just another pod to bin-pack:

• Karpenter (EKS Auto Mode, or self-managed) consolidates under-utilized nodes as a matter of course: it evicts the pods, deletes the node, and the pods reschedule elsewhere. A single-replica control plane is consolidated like any stateless workload — the drill above observed it as ordinary bin-packing, not a failure.
• GKE Autopilot does the equivalent — the platform owns the nodes and compacts them.
• Node auto-upgrades (EKS, GKE, AKS) drain every node in turn on a schedule you may not control.
• Involuntary disruptions — node failure, spot/preemptible reclamation, a kernel panic, a kubelet crash — hit every platform and give no warning that a budget could honor.

The first three are voluntary disruptions: Kubernetes asks before evicting and honors a PodDisruptionBudget. The last is not. The posture below handles both.

HA is the recommended production posture

The control plane already supports more than one replica: the scheduler leader-elects — one active scheduler, the others standing by on the same Postgres advisory lock — and the API serves active-active from every replica (ADR 0009). With replicaCount: 2 (non-split — see split mode below):

• an eviction of the leader becomes a failover measured in seconds — the standby is already pulled, booted and connected; it only has to win the lock (followers poll every few seconds);
• the API and UI stay up on the surviving replica;
• a rolling upgrade is a real rolling upgrade instead of a stop-the-world Recreate.

One switch: the HA profile

The chart ships a complete overlay, helm/dexaflow/examples/values-ha.yaml:

helm upgrade --install dexaflow oci://ghcr.io/dexadata/charts/dexaflow --version <x.y.z> \
  -n leoflow -f values-ha.yaml

It sets, and documents why:

Edit the CHANGEME datastore URLs, secrets and bucket, and bind the control-plane ServiceAccount to the cloud identity that may write the bucket: IRSA on EKS (serviceAccount.annotations), EKS Pod Identity — AWS’s current recommendation, which needs no annotation and is configured as a Pod Identity association against the ServiceAccount instead — or Workload Identity on GKE. In split mode serviceAccount.annotations is rendered onto both ServiceAccounts and both need the identity: the scheduler writes task logs to the bucket and any api replica reads them back to serve the UI, so annotating one role leaves half the control plane without credentials.

Two replicas on one node are one replica

Nothing in Kubernetes keeps two replicas of a Deployment apart by default, and autoscaler consolidation actively pushes them together — bin-packing both onto one node is exactly what it is for. Then a node failure takes the whole control plane, and the auto PDB makes it worse: minAvailable: 1 with both pods on the node being drained blocks the drain, which is the single-replica trap one level up. The profile therefore sets a topologySpreadConstraints entry on kubernetes.io/hostname with maxSkew: 1:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway

whenUnsatisfiable: ScheduleAnyway, deliberately not DoNotSchedule: on a single-node cluster DoNotSchedule leaves the second replica Pending forever, and the PDB then blocks every drain of the one node. A constraint that omits labelSelector gets the Deployment’s own selector labels from the chart, so the values file does not need to know the release name. On multi-zone clusters add a second constraint on topology.kubernetes.io/zone (also ScheduleAnyway).

Split mode is not dispatch-HA

The api/scheduler role split (ADR 0049) is a security topology, not an availability one. Its api Deployment is HA — it runs split.api.replicaCount active-active replicas and gets the auto PDB — but its scheduler Deployment is pinned to one replica by the chart, with no standby and, correctly, no PDB (a budget over one pod would block every drain). Every eviction of the split scheduler is therefore still the full restart window above: image pull, boot, leadership. Non-split replicaCount: 2 is the only topology that gives dispatch failover today; choose the split when you need the restricted API identity more than scheduler failover, and know which one you picked.

The restricted API identity goes one step further than an unbound ServiceAccount: the api pods do not mount a ServiceAccount token at all (automountServiceAccountToken: false), because the api role never calls the Kubernetes API. The scheduler pods, and the single Deployment when split is off, keep the token, since they create the task pods. Cloud identity for the object log sink (IRSA, EKS Pod Identity, GKE Workload Identity) does not use this mount and keeps working. A sidecar that reads the default token mount needs split.api.automountServiceAccountToken: true.

Warm pools are rebuilt on the far side of a failover

Warm worker pools survive a failover as pods, but not as a pool. The assignment stream every warm worker holds open (AwaitAssignment) is gated to the scheduler leader, and the registry of which workers are warm, which pool they serve and which are free is in-memory and leader-local — none of it is persisted or shared between replicas. So a new leader starts with an empty registry and the pool is rebuilt from the workers’ side:

• Idle warm workers exit and are replaced. A control plane on its way out ends every assignment stream at SIGTERM with Unavailable, and a worker sitting idle treats that as a clean recycle: it exits 0, with no ERROR line and no failed pod. A hard-killed leader is the same code but not the same timing. A killed process has its sockets torn down by the kernel, so the worker’s blocked receive fails promptly with Unavailable too. A lost node closes nothing, so the agent finds out by pinging: its gRPC channel sends a keepalive ping after 20 seconds without traffic, including on an idle assignment stream, and closes the connection when a ping goes unanswered for 10 seconds, so the worker sees Unavailable within about 30 seconds and recycles. The control plane permits those pings (one every 10 seconds at most, with or without an active stream) and sends none of its own: a vanished agent is found by the heartbeat reapers, and an assignment sent to a vanished warm worker is reclaimed when its ack lease expires.
• No agent call waits on a silent control plane. Every unary call the agent makes carries a deadline: 10 seconds for a heartbeat, so one unanswered heartbeat never stalls the heartbeat loop into an agent_lost reap of a healthy task, and 30 seconds for the rest (task spec, state reports, XCom, secrets). A state report that times out is retried like any other transient failure. These are fixed values, not settings (#946).
• The warm-pool reconciler rebuilds the buffer. The new leader’s warm-pool reconciler is leader-gated as well, and it counts live warm pods from the apiserver and busy ones from the durable warm_worker_id binding, never from the registry, so it rebuilds each active DAG version’s minIdleWorkers buffer on its own cycle without double-creating.
• A worker handed a follower reconnects toward the leader. Every replica serves the agent gRPC port, so the Service can route a worker to a follower, which refuses the stream with FailedPrecondition (“not the scheduler leader”). The worker re-dials with jittered exponential backoff until it reaches the leader — bounded at 10 consecutive rejections, a fixed constant today (the bound is a field only tests set), after which it exits non-zero and lets the reconciler replace the pod rather than spinning forever against a misconfigured deployment.
• A leader that steps down without restarting sends its workers on. A replica can lose leadership and keep running as a follower (a lost advisory lock, failed leadership checks). Its open assignment streams re-check leadership about once a second: each idle worker’s stream ends with the same FailedPrecondition, and the worker re-dials toward the new leader as above. The 10-rejection bound counts only consecutive rejections: a stream a leader accepted (it announces the registration, or hands out an assignment) starts the count over, so a long-lived worker survives any number of leader changes. A worker running an attempt keeps its stream until the attempt is done and its slot is free, then is sent on the same way. The warm-pool reconcile and staging-volume GC cycles that were running at the step-down stop within about a second as well, so they do not create or delete pods next to the new leader’s own cycles.
• A busy worker finishes its attempt, then exits non-zero — by design. The server puts no busy predicate on the shutdown: the raw SIGTERM context is the shutdown signal the assignment handler selects on, so every open stream ends there, busy or idle. What is idle-specific is the worker’s handling, and deliberately so — it accepts Unavailable as a clean end only in the branch that is waiting for work, because an attempt dying mid-flight and a real outage must both still surface as failures, and a test locks that. So a busy worker runs its attempt to completion and reports its terminal state; then its slot-free send fails on the dead stream, the error is loop-fatal, it propagates out of the worker loop, and the process logs one ERROR and exits 1. Warm pods are RestartPolicy: Never, so expect one Failed warm pod and one ERROR line per busy worker per control-plane restart — and that Failed pod is exactly the signal the warm-worker-lost reaper keys on. A terminal warm pod is not live, so any attempt still bound to it — via the durable warm_worker_id written when the worker acked the assignment — is re-placed on the infrastructure-retry budget without charging the user’s try_number (scheduler resilience).

The cost is a cold window just after a failover, on top of those exits. Warm placement is assign-if-free-else-dedicated, so until workers have re-registered against the new leader, attempts fall through to the dedicated pod-per-task path and pay the start-up they would otherwise have amortized. Attempts themselves are not stranded — the binding and the reaper cover the ones a dead worker held — but a failover is not free of failures either: budget one Failed warm pod per busy worker, and read a post-failover ERROR from a warm worker as the expected shape rather than an incident.

The storage precondition

Task pods stream their logs over gRPC to the control plane, which persists them under config.logsDir. Every replica must be able to write (the leader does) and read (any API replica may serve the log) the same store. That rules out the default volume:

What the object sink makes durable, and when. The live tail the UI shows while a task runs is Redis pub/sub and works from any replica. The durable object is rewritten incrementally while the attempt runs: object stores have no append, so the control plane accumulates the attempt in memory and re-uploads the whole object (an overwrite Put of the same key) whenever the unflushed tail reaches 1 MiB, or every 5 s while anything is unflushed, with a final write on close. Once the stored object is large the cadence is damped — a flush waits until the unflushed tail is at least one eighth of what is already stored — so a big log costs about nine times its final size in upload rather than a quadratic bill; small logs (the common case) trail the live tail by at most a few seconds. What a kill loses is therefore the unflushed tail of each running attempt — for a typical task, the lines of the last ~5 s — not the attempt’s whole log; and an orderly shutdown loses nothing on the control-plane side, because open log streams are closed and flushed at SIGTERM (see drain grace). Two honest caveats. First, when the control plane closes a task’s log stream mid-task, the agent re-opens it against whichever replica answers next, retrying with the same capped backoff as its final report, and the new stream continues the stored log rather than starting it over (the control plane reads the stored object back before it accepts the stream). The lines the task prints while no control plane is reachable wait in the agent, up to 4 MiB, and are shipped once the stream is back. What is still lost: the lines already in flight when the stream broke, lines past that 4 MiB, and everything after the re-open is refused for good (a store that keeps failing the read of the stored object, or a control plane older than this behavior, which the agent detects and does not write to). The task itself is never affected. Second, a bucket with object versioning keeps every rewrite as a version — set a lifecycle rule on noncurrent versions or expect the versioned size to be several times the log size. On memory: the whole attempt is still held in RAM until it ends (capped at 128 MiB per attempt), so size resources.limits.memory for concurrent attempts × their log volume, not for the chart’s 512 Mi default (a wide fan-out of chatty tasks can OOM a small control plane, and OOM is a restart — the flushed prefix survives it). If even a few seconds of tail is unacceptable for you, prefer the RWX PVC, which flushes at 1 MiB without re-uploading.

The chart refuses to render the unsafe combination. replicaCount > 1 (or an HPA with maxReplicas > 1, or the api/scheduler split — any shape where more than one pod mounts the PVC) with a single-writer access mode fails helm install / helm upgrade / helm template with:

more than one control-plane pod would mount the logs PVC (replicaCount > 1,
autoscaling.maxReplicas > 1, or split.enabled), but
logs.persistence.accessMode=ReadWriteOnce attaches the volume to a single
node/pod: the extra pods hang in ContainerCreating with a Multi-Attach error
while the install reports success. HA needs a log store every replica can write.
Recommended: set logs.persistence.enabled=false and ship task logs to object
storage (logs.sink.provider=s3|gcs with logs.sink.bucket). Alternative:
logs.persistence.accessMode=ReadWriteMany on an RWX StorageClass (EFS, Filestore,
Azure Files, NFS, CephFS, Longhorn-rwx). Or keep a single replica. See
helm/dexaflow/examples/values-ha.yaml.

Safe-by-default means exactly this: HA can never silently deploy onto a volume only one pod can hold.

Moving an existing single-replica install to HA

1. Pick the log store first. Switch to the object sink (recommended) or re-create the logs PVC on an RWX class. The chart cannot convert an existing ReadWriteOnce claim in place — access modes are immutable on a bound PVC.
2. Know what happens to old logs. Logs written before the switch stay where they were. With the object sink the control plane serves logs from the bucket only, so keep the old PVC around (or copy it into the bucket under the same {prefix}/{tenant}/{dag}/{run}/{task}/{try}.e{epoch}.log layout, one object per execution of the try; logs written before 0.5.1 are at {try}.log) if you need the history in the UI.
3. Apply the profile. helm upgrade -f values-ha.yaml. With the PVC gone the update strategy auto-selects RollingUpdate, the second replica comes up alongside the first on another node (the spread constraint), and the PDB appears — the install NOTES say so.
4. Confirm leadership moves. Delete the leader pod once and watch the standby take the lock within seconds — no image pull, and the dispatch gap is the re-election poll, not a restart.

Autoscaling: set both bounds, or neither

autoscaling.minReplicas and autoscaling.maxReplicas are independent values with independent defaults — 2 and 6 — so overriding one and not the other inverts them. The obvious shrink, --set autoscaling.maxReplicas=1, leaves the minimum at 2 and produces a HorizontalPodAutoscaler with a maximum below its minimum, which the apiserver rejects.

The chart now refuses that pair at render time rather than letting the install fail, and it refuses a minimum below 1 as well: Kubernetes rejects one unless the HPAScaleToZero feature gate is enabled, which this chart does not assume.

autoscaling:
  enabled: true
  minReplicas: 1   # set both together
  maxReplicas: 3

With maxReplicas above 1 you also need the storage precondition above satisfied — more than one control-plane pod cannot share a ReadWriteOnce log volume, and the chart refuses that combination outright, so the snippet above only installs as written at maxReplicas: 1. Above that, pair it with logs.persistence.enabled: false and a logs.sink, or an RWX access mode.

Both bounds must be whole numbers. A float in a values file used to coerce past the check and render a fractional replica count the apiserver rejects, so non-integers are refused too.

Postgres connection budget

Every control-plane pod opens three connection pools against the same database:

So one pod can hold up to maxOpenConns + 3 connections (plus schedulerMaxConns on a pod that runs the scheduler, when it is set), and the cluster-wide ceiling is that number times the most pods that can be up at once: the HPA’s maxReplicas (or replicaCount, or split.api.replicaCount + 1 in split mode), plus the surge pods a rolling update starts before it stops old ones. The chart sets no maxSurge, so Kubernetes’ default of 25% of the replicas, rounded up, applies to each Deployment. The migration Job adds one short-lived connection per upgrade.

With the shipped defaults and the HPA on at maxReplicas 6, a rollout at full scale runs 6 pods plus ceil(25% of 6) = 2 surge pods, so the worst case is 8 * (20 + 3) = 184 connections. Stock Postgres allows max_connections = 100, of which 3 are reserved for superusers, and managed offerings often size it from instance memory, so a small instance can allow fewer. When the pools reach the limit, new connections fail with FATAL: sorry, too many clients already: requests return 5xx and /readyz can fail across every replica at once.

Keep pods * (maxOpenConns + 3), counting the surge pods, below the server’s max_connections minus whatever else connects to it (backups, migrations, dashboards). In order of preference:

1. Lower database.maxOpenConns. A replica rarely needs 20 connections in flight; 8 at 6 replicas plus 2 surge pods is 8 * (8 + 3) = 88.
2. Cap autoscaling.maxReplicas at what the database can serve.
3. Raise max_connections on the server, if the instance has the memory for it (each connection is a backend process).

PgBouncer. A pooler in front of Postgres multiplexes many client connections over few server connections, but only in transaction pool mode, and two things in the control plane need a real session:

• The scheduler’s leadership is a session-scoped pg_try_advisory_lock (ADR 0009). In transaction mode the lock is taken on whichever server connection served that statement and is not tied to the pod, so two pods can both believe they lead. Processes that run the scheduler must connect to Postgres directly, or through PgBouncer in session mode.
• The migration Job takes an advisory lock the same way.

The api role in split mode holds no session state and can go through transaction mode. pgx then needs default_query_exec_mode=simple_protocol in the DSN (for example postgres://user:pass@pgbouncer:6432/dexaflow?default_query_exec_mode=simple_protocol), because named prepared statements do not survive a change of server connection. The chart takes one database.url for every role, so it cannot send the api Deployment through PgBouncer and the scheduler around it; until it can, size the pools as above.

The PodDisruptionBudget — and the single-replica trap

A PodDisruptionBudget tells the eviction API how many pods must stay up during a voluntary disruption. With two replicas and minAvailable: 1, a drain or a consolidation evicts one replica, waits for the other to be ready, then the next — the control plane never disappears. That is why the chart renders the PDB automatically when the guaranteed replica floor is above one (replicaCount, or the HPA minReplicas, or split.api.replicaCount in split mode).

The same budget over a single replica is a trap. minAvailable: 1 over one pod means no voluntary eviction of that pod is ever allowed:

• kubectl drain hangs on it indefinitely.
• Cluster and node auto-upgrades stall on that node — until the platform’s patience runs out and it overrides the budget anyway.
• Autoscaler consolidation of that node is blocked (Karpenter) — which sounds like protection but leaves an under-utilized node pinned by one pod.
• And none of it helps against an involuntary disruption: the pod still dies with the node.

It protects the pod at the cost of cluster operations. So the chart never turns it on for a single replica by default; podDisruptionBudget.enabled: true still forces it (an informed choice — the install NOTES say what it costs), and false forces it off even in HA.

Two more details the chart gets right for you:

• unhealthyPodEvictionPolicy: AlwaysAllow (the chart default; "" omits the field for apiservers older than 1.27). Kubernetes’ own default, IfHealthyBudget, refuses to evict even unhealthy pods once the budget is unmet — so with both replicas unready (bad database credentials after a rotation, an image-pull failure, a wedged rollout) a node drain hangs on a control plane that is already down. AlwaysAllow lets the broken pods go.
• Auto mode counts every shape. The floor is replicaCount, or the HPA minReplicas when autoscaling is on, or split.api.replicaCount in split mode. That means an existing split.enabled install (api default 2) or autoscaling.enabled install (minReplicas default 2) gains a PDB on upgrade to this chart version; the upgrade NOTES call it out, and podDisruptionBudget.enabled: false opts out if your maintenance tooling assumed none.

How platforms honor a PDB

Two replicas plus the auto PDB behave well on every row of that table. A single replica with a forced PDB behaves badly on every row.

karpenter.sh/do-not-disrupt — EKS/Karpenter-only opt-in

Karpenter honors a pod annotation, karpenter.sh/do-not-disrupt: "true", that exempts the pod’s node from Karpenter’s voluntary disruption: no consolidation, no drift replacement, no expiration while the pod runs. The chart wires podAnnotations into the control-plane pod template, so it is one uncommented line in the HA profile:

podAnnotations:
  karpenter.sh/do-not-disrupt: "true"

It is deliberately not set by default:

• it is meaningless outside Karpenter (GKE, AKS and a plain cluster-autoscaler ignore it);
• it trades maintenance friction for eviction protection — that node is never consolidated or rolled by Karpenter while the control plane sits on it, so drift (a new AMI) and consolidation savings stop at that node;
• it does nothing against involuntary disruptions.

Reach for it only when you have measured that consolidation on your cluster still evicts the control plane faster than failover recovers it — and prefer the second replica first.

Drain grace

A voluntary eviction sends SIGTERM and waits terminationGracePeriodSeconds (Kubernetes default 30s) before SIGKILL. Be precise about what the grace buys, because it is not leadership:

• Leadership handoff does not depend on it. The scheduler releases its advisory lock within about one tick of SIGTERM, and the lock frees anyway the moment the Postgres connection drops — SIGKILL included. The standby wins the lock on its next poll either way.

• What it does buy is drain headroom. On SIGTERM the server gives in-flight HTTP requests up to 10 s to finish, then drains the dispatch pool so dispatches already in progress settle instead of leaving task instances stuck queued. Reapers are gated off during the step-down so nothing destructive fires from a dying leader.

• A second replica can stop taking new streams before the first one stops. Endpoint removal is asynchronous, so for a propagation window after termination begins, task pods still open new agent log streams against the replica that is on its way out. deployment.preStopSleepSeconds spends that window in a preStop sleep before the process is signalled, so those streams open against the surviving replica instead. It is off by default and set to 5 in examples/values-ha.yaml, because moving a stream needs somewhere to move it to: a single-replica install (the chart default, and the split topology’s scheduler, which is what serves agent gRPC) has no second endpoint, so the sleep buys nothing for agent log streams there and adds its own seconds to every upgrade. That is the scheduler-side singleton case, not the whole story: the hook renders on both split Deployments deliberately, and on the api side with split.api.replicaCount > 1 the same sleep spends the same propagation window for HTTP and UI requests, which have a second endpoint to land on. What it never buys, at any replica count, is an already-established stream: those connections are pinned by conntrack and get Unavailable at SIGTERM regardless — only not-yet-opened streams move. The sleep runs inside terminationGracePeriodSeconds, ahead of everything below, so count it in the budget (Kubernetes rejects the pod spec outright if the sleep alone exceeds the grace, and the chart refuses to render that). 5 s is the low end; raise it when a service-mesh sidecar sits in the path. The hook needs Kubernetes 1.30, where the native sleep action is beta and on by default; it is alpha and off in 1.29, and an apiserver without it rejects the empty preStop: {} it is left with instead of ignoring it. So the chart renders the hook only on 1.30+ and omits it below — a 1.27–1.29 install keeps working, with the pre-hook behavior, and the install notes warn that the value you set had no effect rather than leaving the omit silent.

• With tasks running, the stop is bounded. Open agent log streams are closed and flushed the moment SIGTERM arrives (the agent keeps running its task; log shipping is best-effort), idle warm-worker assignment streams end the same way, and the gRPC graceful stop that follows is bounded at 5 s before falling back to a forced stop — which still lets the remaining handlers finish their deferred flushes, waited for up to another 5 s.

  Two different clocks, and the grace has to cover both. Process shutdown is everything after SIGTERM: normally milliseconds, well under a second, warm pools on or off; bounded at HTTP (≤10 s) + dispatch drain (≤15 s, configurable)

  • gRPC stop (≤10 s) ≈ 35 s plus the telemetry flush. Pod termination is what an operator watches, and it also includes the preStop sleep, which runs before SIGTERM and is pure wall-clock: with the HA profile’s 5 a completely healthy pod takes just over 5 s to go away, and that is expected, not a slow shutdown.

  So size the grace by the rule, not by a number: terminationGracePeriodSeconds ≥ deployment.preStopSleepSeconds + 10 s (HTTP) + the dispatch drain + 10 s (gRPC), with headroom. The default 30 s covers a normal shutdown but not that worst case, so an installation running the object log sink at scale should raise it — the HA profile’s 60 holds the full bound plus its 5 s sleep. A SIGKILL (exitCode: 137 on the terminated container) means a stop that overran the grace — look for the agent grpc graceful stop exceeded its bound warning first, then for a slow object store. That warning is meant to be rare: every long-lived agent stream ends itself at SIGTERM, so seeing it on every shutdown is a bug, not a tuning problem. Nothing in the leadership handoff needs any of this to finish.

The HA profile sets 60 for comfortable HTTP + dispatch drain headroom under load. The chart deliberately ships no default: a default would add up to 30 s of downtime to every single-replica Recreate upgrade, for nothing.

Three values mean the same thing here — unset, null, and an explicit 0: the chart omits the field and Kubernetes’ own 30 s applies, which is also the grace the preStopSleepSeconds guard reasons about. A literal 0 is deliberately not rendered, because in a pod spec it means SIGKILL with nothing drained at all — in-flight HTTP requests cut, the dispatch pool never settling (task instances left stuck queued), open agent log streams never flushed, and any preStop sleep unsatisfiable. If you really want no grace, set it on the pod spec yourself rather than through this value.

Involuntary disruptions: why HA is the posture that matters

No PDB, annotation or grace period prevents:

• node failure — hardware, kernel panic, kubelet or container-runtime crash;
• spot / preemptible reclamation — a two-minute notice at best, no eviction API involved;
• zone or network partition that isolates the node;
• platform-side forced maintenance once a bounded PDB window expires.

Each of those is the 48–80 second restart window with no warning. The only thing that shrinks it is a replica that is already running somewhere else — and the scheduler’s own resilience — the leader-settling gate that holds every reaper until the reconciler has swept under the new leader, the destructive gate that holds every reaper during a step-down, the reconciler that recovers a pod’s durable outcome (now run in the same cycle as, and before, the reapers), and the boot-time check of the timing ladder they depend on, described in scheduler resilience — is what keeps the tasks that were in flight during the window from being falsely failed. HA shortens the window; the resilience mechanisms make the remaining window survivable. Run both.

When the database disappears without a reset

A managed failover that moves the database address, a network partition or a lost database node can leave the leader’s lock session open on a socket that no longer answers. Two settings bound how long that lasts:

• On the scheduler. Every leadership check, and every attempt to take leadership, runs under a deadline of one check interval (5 s). A check that cannot confirm the lock in time counts as failed, and after 3 failed checks in a row the leader steps down (reason check_timeout) and campaigns again, so a leader cut off from its database stops scheduling within about 15 to 30 seconds. Before, the check waited for the kernel to give up on the socket (about 15 minutes with the default tcp_retries2), while a follower on the new primary could already lead.
• On the database server. The leader session sets tcp_keepalives_idle = 30, tcp_keepalives_interval = 10, tcp_keepalives_count = 3 and tcp_user_timeout = 60000 for itself, so Postgres ends a session whose client vanished within about a minute and the advisory lock is free for the next leader. With the operating system defaults that takes about two hours, and no replica can lead in the meantime. The settings are sent with SET (not as startup parameters, which PgBouncer refuses), apply to the leader session only, and are skipped with a warning where the server refuses one.

Nothing needs configuring: both apply to every install.

Related

• Helm chart — the entry point and the chart README with the full values table.
• Scheduler resilience — what happens to in-flight tasks around a restart.
• Warm worker pools — the pool is leader-local and is rebuilt on failover; what that costs the attempts that follow one.
• Upgrades — rolling a control-plane release, edition by edition.
• ADR 0009 — leader election; ADR 0049 — api/scheduler split; ADR 0056 — the task-log object sink.