Admission hooks

Status: available in both editions, off by default (ADR 0100, issue #2177).

An admission hook is asked about each task attempt after Dexaflow’s own checks (the placement allowlist, the image allowlist, the resource unit) and before the attempt’s agent token is minted or its pod is created. It answers one of:

Use it to place each tenant’s tasks on its own nodes, to label pods for cost reporting, to keep work off the cluster during a freeze window, or to refuse work a budget does not cover. Because the decision comes before the token and the pod, a refused attempt never runs and never holds a credential, on Lite as on Cluster.

Turn it on

Point execution.admission.webhook.url at an endpoint. With the Helm chart:

execution:
  admission:
    webhook:
      url: https://admission.policy.svc:8443/admit
      # Optional: a ConfigMap with key ca.crt that signs the endpoint's
      # certificate, and a Secret with key token sent as a bearer.
      caConfigMap: policy-ca
      tokenSecret: policy-token
    timeout: 500ms
    failOpen: false

The chart mounts the CA bundle and the token only on the pods that dispatch tasks (the scheduler in a split install), never on the API pods. The control plane connects to the endpoint directly: HTTPS_PROXY and the other proxy variables are not used.

The request

The control plane POSTs one JSON document per attempt:

{
  "api_version": "dexaflow.io/admission/v1",
  "kind": "AdmissionReview",
  "request": {
    "uid": "6f1c2a8e-0d4b-4f5e-9a3c-2b7d8e1f0a42",
    "tenant_id": "acme",
    "dag_id": "nightly_etl",
    "run_id": "0193f2c4-...",
    "task_id": "extract",
    "try_number": 1,
    "operator": "python",
    "image": "ghcr.io/acme/etl@sha256:...",
    "source_mode": false,
    "resources": {
      "requests": {"cpu": "500m", "memory": "1Gi"},
      "limits": {"memory": "1Gi"}
    },
    "execution": {
      "service_account": "dexaflow-task",
      "node_selector": {"zone": "a"},
      "labels": {"team": "data"}
    },
    "service_account_chosen": false
  }
}

resources are the effective ones, after the resource unit or the platform defaults. execution.service_account is the account the pod would run as; service_account_chosen is false when that is the operator’s default rather than one the task named. The request never carries the task’s environment, its source, its secrets or its token, but it does carry tenant and DAG names, so the endpoint must be https unless it is on the same host.

The answer

Answer with HTTP 200 and the same envelope, echoing the request’s uid:

{
  "api_version": "dexaflow.io/admission/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "6f1c2a8e-0d4b-4f5e-9a3c-2b7d8e1f0a42",
    "decision": "admit",
    "changes": {
      "node_selector": {"tenant": "acme"},
      "tolerations": [{"key": "tenant", "operator": "Equal", "value": "acme", "effect": "NoSchedule"}],
      "labels": {"cost-center": "4711"}
    }
  }
}

• decision is admit, hold or refuse. hold and refuse take a reason, which the task instance shows (control characters removed, cut to 512 bytes), and no changes.
• changes may set node_selector and labels/annotations (merged over the task’s own, your keys win), tolerations (appended), priority_class_name and service_account. Nothing else: the image, command, environment, namespace and resources stay as Dexaflow chose them, so the attempt is charged the pool slots it was admitted with.

A change is refused, naming the hook, when it would set a label or annotation key under leoflow.io/ or dexaflow.io/, a key, value or toleration Kubernetes would reject, a priority_class_name outside execution_allowlist.priority_classes, or a service_account outside execution_allowlist.service_accounts. A webhook may change the service account only when that list is set, so whoever runs the endpoint cannot move a task onto any account of the namespace. Changes are not checked against execution_allowlist.fields: that list governs what DAG authors set, and the task’s own fields are still checked against it before the hook is asked.

An attempt whose placement the hook changed runs on a dedicated pod, like a task that sets its own placement, never on a warm worker.

The answer must be at most 64 KiB and contain no field besides the ones above (field names are matched as Go’s JSON decoder matches them, ignoring case). An admit that carries a reason, or a hold or refuse that carries changes, is a malformed answer and is treated like a broken endpoint, below. Redirects are not followed.

When the endpoint is broken

A connection error, a status other than 200, an answer that does not decode or does not echo the uid, or no answer within the timeout all mean the hook is broken. By default the attempt is held, as with hold, so a task never runs unchecked while the endpoint is down; it fails as dispatch_failed once it has waited 15 minutes, naming the admission hook. That failure says only that the request failed or timed out, never the endpoint’s address; the server log has the details.

After 5 failures in a row the control plane stops asking for 30 seconds, then tries one attempt; while it is not asking, attempts are held without waiting out the timeout. A hold or refuse is the endpoint working and never counts as a failure.

With fail_open: true a broken endpoint admits attempts unchanged instead. The server logs that at boot and on every such attempt. A hold or refuse still applies.

A hook in your own binary

A program that builds the server with pkg/server can register a hook in process instead of, or before, the webhook. It implements pkg/admission’s Hook, receives the same request and returns the same decisions, with no network hop:

freeze := admission.HookFunc(func(_ context.Context, req *admission.Request) (admission.Decision, error) {
	if req.TenantID == "acme" {
		return admission.Refuse("acme is frozen until 06:00 UTC"), nil
	}
	return admission.Admit(), nil
})
err := server.Run(ctx, server.WithAdmission(freeze))

Hooks registered this way run in the order they were registered, before the webhook when one is configured. The first refusal wins and each later hook sees the earlier changes. execution.admission.timeout and fail_open, the checks on what a change may set, and the breaker apply to them as to the webhook. A hook must return once its context is done.

Monitor it

Alert on error, breaker_open and fail_open: each means attempts are waiting on, or bypassing, a broken endpoint.