DexaflowSchemaNotReady
What DexaflowSchemaNotReady means and what to do when it fires.
The chart renders a PrometheusRule with two groups when
metrics.prometheusRule.enabled is true. It needs the Prometheus Operator
CRDs (kube-prometheus-stack) and a scrape of /metrics
(metrics.serviceMonitor.enabled).
severity: critical, one warning case): the control plane cannot
schedule, settle or serve right now. Route it to whoever is on call.severity: warning): something degrades and needs a look during
working hours.Every rule is scoped to the release namespace and carries a runbook_url
annotation that points at its page below. Set
metrics.prometheusRule.runbookBaseURL when the people on call read a mirror
of these docs. Thresholds are values under metrics.prometheusRule.thresholds;
the defaults are in the table.
metrics:
serviceMonitor:
enabled: true
prometheusRule:
enabled: true
additionalLabels:
release: kube-prometheus-stack # match your Prometheus ruleSelector
thresholds:
queueWaitP99Seconds: 300
| Alert | Fires when | Default |
|---|---|---|
| DexaflowSchemaNotReady | dexaflow_schema_ready == 0 for 5 minutes (critical only for a replaced database) | |
| DexaflowNoSchedulerLeader | a scheduler replica is up and no leader tick ran for 5 minutes, for 5 minutes | |
| DexaflowReaperSettlingValveOpen | any reap_settling_valve_open decision in 10 minutes | |
| DexaflowInfraConfirmValveOpen | any infra_confirm_valve_open decision in 10 minutes | |
| DexaflowTaskQueueWaitHigh | p99 of dexaflow_task_queue_wait_seconds of a tenant over the threshold for 15 minutes | queueWaitP99Seconds: 300 |
| DexaflowAPIErrorRatioHigh | share of 5xx answers over the threshold for 10 minutes | apiErrorRatio: 0.01 |
| DexaflowDatabasePoolCanceledAcquires | canceled connection acquires on a pool every 5 minutes for 10 minutes | |
| DexaflowRedisCommandFailures | Redis command failures (client cancellations excluded) every 5 minutes for 10 minutes | |
| DexaflowRedisEvictingKeys | Redis evicted_keys grew in the last 10 minutes | |
| DexaflowResilienceBackstopDisabled | a resilience backstop is disabled, for 5 minutes | |
| DexaflowAgentLostReapingHeld | any agent_lost_mass_silence_hold or agent_lost_mass_silence_valve_open decision in 10 minutes |
| Alert | Fires when | Default |
|---|---|---|
| DexaflowRedisEvictionPolicyUnsafe | Redis runs a maxmemory-policy other than noeviction, for 15 minutes | |
| DexaflowSchedulerStepDowns | leader step-downs in one hour over the threshold, shutdowns excluded | stepDownsPerHour: 3 |
| DexaflowTasksLost | attempts failed as agent_lost, pod_lost, dispatch_lost or warm_worker_lost in one hour over the threshold | lostTasksPerHour: 10 |
| DexaflowTLSReloadFailed | a rotated certificate failed to load in the last 15 minutes | |
| DexaflowAlertDispatchFailing | an on-failure alert failed to send in the last 30 minutes | |
| DexaflowUsageEventsLag | usage events lag over the threshold for 15 minutes | usageEventsLagSeconds: 900 |
| DexaflowOldestQueuedTask | the oldest queued attempt of a tenant older than the threshold, for 5 minutes | oldestQueuedSeconds: 600 |
| DexaflowMixedVersions | processes report more than one version for longer than the threshold | mixedVersionsFor: 30m |
The retention janitor exports no error counter, so a failing retention cycle
has no alert; its log line (retention) is the signal for now.
The rules are tested: scripts/prometheus-rules-test.sh renders them and runs
promtool test rules against helm/dexaflow/tests/promtool/, with a case that
fires and one that stays quiet for every alert. Change a rule and its test
together.
See Metrics for every family the rules read.
What DexaflowSchemaNotReady means and what to do when it fires.
What DexaflowNoSchedulerLeader means and what to do when it fires.
What DexaflowReaperSettlingValveOpen means and what to do when it fires.
What DexaflowInfraConfirmValveOpen means and what to do when it fires.
What DexaflowTaskQueueWaitHigh means and what to do when it fires.
What DexaflowAPIErrorRatioHigh means and what to do when it fires.
What DexaflowDatabasePoolCanceledAcquires means and what to do when it fires.
What DexaflowRedisCommandFailures means and what to do when it fires.
What DexaflowRedisEvictingKeys means and what to do when it fires.
What DexaflowResilienceBackstopDisabled means and what to do when it fires.
What DexaflowAgentLostReapingHeld means and what to do when it fires.
What DexaflowSchedulerStepDowns means and what to do when it fires.
What DexaflowTasksLost means and what to do when it fires.
What DexaflowTLSReloadFailed means and what to do when it fires.
What DexaflowAlertDispatchFailing means and what to do when it fires.
What DexaflowUsageEventsLag means and what to do when it fires.
What DexaflowOldestQueuedTask means and what to do when it fires.
What DexaflowMixedVersions means and what to do when it fires.
What DexaflowRedisEvictionPolicyUnsafe means and what to do when it fires.