Alerts and runbooks

The alerts the Helm chart ships, what each one means, and what to do when it fires.

The chart renders a PrometheusRule with two groups when metrics.prometheusRule.enabled is true. It needs the Prometheus Operator CRDs (kube-prometheus-stack) and a scrape of /metrics (metrics.serviceMonitor.enabled).

  • page (severity: critical, one warning case): the control plane cannot schedule, settle or serve right now. Route it to whoever is on call.
  • ticket (severity: warning): something degrades and needs a look during working hours.

Every rule is scoped to the release namespace and carries a runbook_url annotation that points at its page below. Set metrics.prometheusRule.runbookBaseURL when the people on call read a mirror of these docs. Thresholds are values under metrics.prometheusRule.thresholds; the defaults are in the table.

metrics:
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
    additionalLabels:
      release: kube-prometheus-stack   # match your Prometheus ruleSelector
    thresholds:
      queueWaitP99Seconds: 300

Page

AlertFires whenDefault
DexaflowSchemaNotReadydexaflow_schema_ready == 0 for 5 minutes (critical only for a replaced database)
DexaflowNoSchedulerLeadera scheduler replica is up and no leader tick ran for 5 minutes, for 5 minutes
DexaflowReaperSettlingValveOpenany reap_settling_valve_open decision in 10 minutes
DexaflowInfraConfirmValveOpenany infra_confirm_valve_open decision in 10 minutes
DexaflowTaskQueueWaitHighp99 of dexaflow_task_queue_wait_seconds of a tenant over the threshold for 15 minutesqueueWaitP99Seconds: 300
DexaflowAPIErrorRatioHighshare of 5xx answers over the threshold for 10 minutesapiErrorRatio: 0.01
DexaflowDatabasePoolCanceledAcquirescanceled connection acquires on a pool every 5 minutes for 10 minutes
DexaflowRedisCommandFailuresRedis command failures (client cancellations excluded) every 5 minutes for 10 minutes
DexaflowRedisEvictingKeysRedis evicted_keys grew in the last 10 minutes
DexaflowResilienceBackstopDisableda resilience backstop is disabled, for 5 minutes
DexaflowAgentLostReapingHeldany agent_lost_mass_silence_hold or agent_lost_mass_silence_valve_open decision in 10 minutes

Ticket

AlertFires whenDefault
DexaflowRedisEvictionPolicyUnsafeRedis runs a maxmemory-policy other than noeviction, for 15 minutes
DexaflowSchedulerStepDownsleader step-downs in one hour over the threshold, shutdowns excludedstepDownsPerHour: 3
DexaflowTasksLostattempts failed as agent_lost, pod_lost, dispatch_lost or warm_worker_lost in one hour over the thresholdlostTasksPerHour: 10
DexaflowTLSReloadFaileda rotated certificate failed to load in the last 15 minutes
DexaflowAlertDispatchFailingan on-failure alert failed to send in the last 30 minutes
DexaflowUsageEventsLagusage events lag over the threshold for 15 minutesusageEventsLagSeconds: 900
DexaflowOldestQueuedTaskthe oldest queued attempt of a tenant older than the threshold, for 5 minutesoldestQueuedSeconds: 600
DexaflowMixedVersionsprocesses report more than one version for longer than the thresholdmixedVersionsFor: 30m

What is not covered yet

The retention janitor exports no error counter, so a failing retention cycle has no alert; its log line (retention) is the signal for now.

The rules are tested: scripts/prometheus-rules-test.sh renders them and runs promtool test rules against helm/dexaflow/tests/promtool/, with a case that fires and one that stays quiet for every alert. Change a rule and its test together.

See Metrics for every family the rules read.


DexaflowSchemaNotReady

What DexaflowSchemaNotReady means and what to do when it fires.

DexaflowNoSchedulerLeader

What DexaflowNoSchedulerLeader means and what to do when it fires.

DexaflowReaperSettlingValveOpen

What DexaflowReaperSettlingValveOpen means and what to do when it fires.

DexaflowInfraConfirmValveOpen

What DexaflowInfraConfirmValveOpen means and what to do when it fires.

DexaflowTaskQueueWaitHigh

What DexaflowTaskQueueWaitHigh means and what to do when it fires.

DexaflowAPIErrorRatioHigh

What DexaflowAPIErrorRatioHigh means and what to do when it fires.

DexaflowDatabasePoolCanceledAcquires

What DexaflowDatabasePoolCanceledAcquires means and what to do when it fires.

DexaflowRedisCommandFailures

What DexaflowRedisCommandFailures means and what to do when it fires.

DexaflowRedisEvictingKeys

What DexaflowRedisEvictingKeys means and what to do when it fires.

DexaflowResilienceBackstopDisabled

What DexaflowResilienceBackstopDisabled means and what to do when it fires.

DexaflowAgentLostReapingHeld

What DexaflowAgentLostReapingHeld means and what to do when it fires.

DexaflowSchedulerStepDowns

What DexaflowSchedulerStepDowns means and what to do when it fires.

DexaflowTasksLost

What DexaflowTasksLost means and what to do when it fires.

DexaflowTLSReloadFailed

What DexaflowTLSReloadFailed means and what to do when it fires.

DexaflowAlertDispatchFailing

What DexaflowAlertDispatchFailing means and what to do when it fires.

DexaflowUsageEventsLag

What DexaflowUsageEventsLag means and what to do when it fires.

DexaflowOldestQueuedTask

What DexaflowOldestQueuedTask means and what to do when it fires.

DexaflowMixedVersions

What DexaflowMixedVersions means and what to do when it fires.

DexaflowRedisEvictionPolicyUnsafe

What DexaflowRedisEvictionPolicyUnsafe means and what to do when it fires.