ADR 0066: Weighted pool slots and an operator resource unit
Status: Accepted Date: 2026-10-06 (proposed and accepted the same day by the project owner) Relates: ADR 0053 (admission and placement; this ADR changes what the Stage 3 pool gate counts), ADR 0023 (dexaflow.yaml config binding; adds one per-task and one DAG-default knob), ADR 0063 (operator executor policy, proposed; its resource ceiling composes with the unit below), ADR 0058 (warm worker pools; section 3 says which tasks may use them), ADR 0062 (ships in v0.5.2 under the exception recorded in its section 6.2), ADR 0011 (strict TDD). Guide: Pools
Numbering. 0063 (#1365) and 0064 (#1368) are open PRs. If another ADR merges first under 0066, this file is renumbered before merge.
Context
The pool gate of ADR 0053 Stage 3 admits a task only while its pool has a free
slot, and every task takes exactly one slot (poolHasSlot and
activePoolCounts in internal/scheduler/; the API reports pool_slots: 1 for
every task instance). A pool therefore caps how many tasks run, not how
much compute they hold: a pool of 4 admits four tasks of 250m / 512Mi or four
tasks of 8 CPU / 32Gi alike.
That is the wrong axis for anyone who shares a cluster between teams or tenants and wants to size a team’s share in compute:
- A team’s budget is CPU and memory. A slot that means “one task of any size”
makes the budget meaningless as soon as one DAG declares large
resources. - Airflow already has the knob for this. A task’s
pool_slots(default 1) is how many slots of its pool it takes. Dexaflow ignores it: the parser does not capture it and the gate counts 1. - Nothing ties a slot to resources, so an operator who wants “one slot = N CPU
and M memory” has to police every DAG’s
resourcesby hand.
Decision
1. pool_slots weights the pool gate
TaskSpec gains pool_slots (JSON pool_slots, integer, 0 or absent means 1),
with Airflow’s meaning: the number of slots the task takes in its pool while it
is queued or running.
- The gate admits a task only when
occupied + promoted_this_tick + pool_slots <= slots. Occupancy is the sum ofpool_slotsof the pool’s queued and running task instances, not their count; the within-tick folding across runs (ADR 0053) sums the same way. - A task that does not fit stays
scheduled, exactly like a task that finds the pool full today. Nothing fails. max_active_tasks(Stage 1) keeps counting tasks: it is a per-DAG fan-out cap, not a compute budget.- No behavior change for existing DAGs: none sets
pool_slotstoday, so every task weighs 1, as now. Lite has no pools and ignores the field. - The API and the UI report the real
pool_slots, and pool occupancy (PoolSlotUsage) sums it. To sum it in SQL,task_instancesgains apool_slotscolumn (constant default 1, so Postgres adds it without rewriting the table and every existing row weighs 1), written when the task instances are materialized.
2. Authoring
dag.py:@task(pool_slots=N)and an operator’spool_slots=Nare captured by the parser, the same wayretriesis.dexaflow.yaml:tasks.<task_id>.size: Nanddefaults.size: N. The YAML name issizebecause, once a unit is configured (section 3), the number is the size of the task, and that is how an author thinks about it. It compiles topool_slots. The key is read fromleoflow.yamlexactly as fromdexaflow.yaml, like every other key of ADR 0023. Precedence follows ADR 0023, most specific wins: the YAML task override > the task’s ownpool_slotsindag.py> the YAMLdefaults.size> 1.- Registration refuses
pool_slots < 1or above a ceiling of 1024. - The field is omitted from the spec JSON when it is 1 or unset, so the canonical hash of every DAG registered today is unchanged and an upgrade creates no new DAG versions.
3. Optional operator resource unit
Server config executor.unit.cpu and executor.unit.memory (Kubernetes
quantities; both or neither). Unset is the default and changes nothing. Like
every server key, each binds from DEXAFLOW_* and from the legacy LEOFLOW_*
name (DEXAFLOW_EXECUTOR_UNIT_CPU, LEOFLOW_EXECUTOR_UNIT_CPU, …). When
set:
- Dispatch. A task that declares no
resourcesgetsrequests = limits = pool_slots x unit(Guaranteed QoS). This replaces the L0executor.defaults.resourcesfor that task; the default stays in force while no unit is configured. - Registration. A task that declares its own
resourcesmust fit in its size: every requested or limited cpu and memory must be at mostpool_slots x unit. Otherwise registration fails with the size the task would need, for exampletask "train" asks for 2 CPU, which is 8 units of 250m; set size: 8. The engine never clamps or rewrites the author’s resources (the rule ADR 0063 also follows). - Dispatch, again. A DAG registered before the unit was configured is
checked at dispatch with the same rule, and a task that does not fit fails
with the same message instead of running larger than it is charged. This is
a permanent refusal, not the transient
Rejectedthat ADR 0031 retries: the dispatcher gains aRefusedoutcome that fails the attempt once, with the message as its note, and does not spend dispatch retries on it. ADR 0063 needs the same outcome; whichever lands first adds it and the other reuses it. - Rollout.
executor.unit.enforceisrefuse(the default) orwarn. Underwarn, registration accepts a task that does not fit and dispatch runs it with its ownresources; both log the task, its size and the size it would need, and count it indexaflow_unit_misfit_total. An operator turns the unit on inwarn, fixes the DAGs the log names, then switches torefuse, so a production install never starts failing tasks without notice. - Size bound without a pool budget. A pool with no positive budget, and,
when
confine_undefined_poolsis off, a pool the tenant never defined, is unlimited (ADR 0053 fails open). With a unit configured that would let one task ask for1024 x unitwith nothing to stop it. So, while a unit is set, registration also refuses a task whosepool_slotsis aboveexecutor.unit.max_size(default 64), whatever its pool, and dispatch checks it again for DAGs registered earlier. - Warm workers. A warm pod (ADR 0058) is created with
requests = limits = 1 x unitwhile a unit is set, and only a task of size 1 that declares noresourcesis placed on one; any other task takes the dedicated pod path, the same degrade-not-strand exclusion as staging. With no unit, warm placement is unchanged. - Ephemeral storage and DRA claims are outside the unit; ADR 0063’s ceiling covers them.
- With ADR 0063: the policy’s
resources.maxstays the per-task ceiling in absolute quantities, and the unit makes the pool a ceiling on the sum.
4. A large task is not starved
Weighted admission has a failure mode that unweighted admission did not: a task of 4 slots waiting in a pool of 6 never fits while tasks of 1 slot keep taking the space as it frees. The scheduler keeps an in-memory reservation per pool:
- When a scheduled task does not fit and has been waiting longer than
scheduler.pool_starvation_threshold(default 60s, 0 disables), the pool is reserved for that task. Only a task that passes every other gate in that tick can reserve: its dispatch backoff has elapsed (ADR 0031) and its DAG is undermax_active_tasks. A task held by anything but the pool never freezes it. While reserved, the pool admits no other task; the reserving task is admitted as soon as it fits, which releases the reservation. - One reservation per pool at a time: the oldest waiting task wins, by the time the scheduler first saw it blocked.
- A reservation is dropped on the next tick when its task is no longer
scheduled(run cancelled, task skipped), when it stops passing the other gates, or when the pool’s budget changes so that the task is larger than the whole pool or the pool becomes unlimited. - A task larger than its pool’s total slots is never reserved for (it would freeze the pool); it waits, and the scheduler logs a warning naming the pool and the sizes once per task. Registration-time limits (section 5) are the way to refuse it up front.
- The state lives only in the leader. A failover forgets the waiting times, so the worst case is one extra threshold of waiting; no new table or query.
5. Per-tenant ceiling on a task’s size
The service tenant limits (migration 040) gain max_task_pool_slots, a new
tenants column added by its own migration (the next free number, constant
default 0, so Postgres adds it without rewriting the table) and an optional
field of PUT /api/v2/service/tenants/{tenant}: 0 is
unlimited; otherwise registration refuses a DAG with a task whose pool_slots
is above it, naming the task and both numbers. A platform that sizes each
tenant’s default pool sets it to that pool’s slots, so a task that could never
fit is refused when it is pushed instead of waiting forever.
Consequences
- An operator can size a team’s or tenant’s share in compute: unit = 250m / 512Mi and a default pool of 8 slots means 2 CPU and 4Gi at once, whatever the mix of task sizes.
- Pools stay task-count pools for every install that sets no unit and no
pool_slots. - The pool gate’s cost does not change: weights come from the specs the tick
already loads, the reservation map is O(pools), and no query is added to the
tick. Section 1 adds one column to
task_instancesand section 5 one totenants; neither is read by the tick. - Ships in v0.5.2 by the owner’s decision, recorded as an exception in ADR
0062 section 6.2. Each PR lands on
mainwith thev0.5.2milestone and is cherry-picked intorelease-0.5. - Delivered in independent PRs, each with its tests first: (1)
pool_slotsin the spec, parser, YAML and the weighted gate; (2) the unit at dispatch and registration; (3) the starvation reservation; (4)max_task_pool_slots.