Core Concepts
Smart Alerting
How Kubeadapt's alerts decide to fire, how cost-spike detection works, and the four alert types you can configure.
Smart Alerting sends a message to the right channel when one of four things happens: spend spikes, a budget runs hot, resources sit idle, or an expensive new workload appears.
How alerts decide to fire
Three of the four alert types compare a value with a threshold you set: a budget percentage, an idle percentage or a monthly dollar amount. Cost spike is different. It learns what normal spend looks like for the scope and fires when spend leaves that range, so there is no threshold to set.
The four user-configurable alert types
| Alert type | Fires when… | Common scope |
|---|---|---|
cost_spike | Spend stays above the range expected from the scope's own history, by more than a minimum dollar amount. | Cluster, team |
budget_threshold | Period-to-date spend crosses 50%, 80%, 90%, or 100% of a budget you set. | Org, department |
unused_resources | A scope's idle resources cross a configured percentage. Delivered as a weekly digest. | Cluster |
new_expensive_workload | A workload appears that's projected to cost more than a configured monthly threshold. | Cluster, team |
You configure each type's conditions on a Rule. One Rule can enable any combination of the four. See Rules for the rule shape.
How cost-spike actually works
Cost spike is the only alert type that sets its own threshold, so this section explains how it decides.
Cost spike compares each day's spend with a band that Kubeadapt fits to that scope's own history, including day-of-week and hour-of-day patterns. It opens an incident only when spend stays outside the band and the excess clears a minimum dollar amount. You don't set a threshold. The incident shows the expected band next to what actually happened.
A few consequences follow from that design:
- Persistence matters. A single noisy hour does not open an incident. Spend has to stay outside the band.
- Small scopes stay quiet. The minimum dollar amount keeps a large percentage jump on a cheap scope from alerting.
- New scopes need history. Until a scope has enough history to fit a band, cost spike does not fire for it.
- There are no knobs on the rule. The band and the dollar floor come from the scope's spending pattern.
The other three alert types
Budget threshold is the simplest. You set a budget for a period (monthly, quarterly, or fiscal year), pick which threshold percentages to be notified at (50%, 80%, 90%, 100% — any non-empty subset), and pick a cost mode (Fully Loaded vs Workload Only). The rule fires once when each selected threshold is first crossed; the counter resets at the start of the next period.
Unused resources runs on a weekly schedule. It rolls up the scope's idle resources (low CPU/memory utilization combined with low network activity), filters by the minimum idle percentage you set, and delivers the top N most expensive idle items in a single digest. You configure the percentage and the cap; the cadence is fixed at weekly. Use this when you want a "did we leave money on the table this week?" recap instead of a real-time alert.
New expensive workload watches for workloads that appear and exceed a monthly dollar projection. You configure the dollar threshold, a minimum age (so a workload has to run long enough for the cost projection to stabilize), and a list of owner Kinds to exclude. Defaults exclude short-lived built-ins (Job, CronJob) and common ML operators (Argo Workflows, Spark, Ray, Kubeflow). The auto-exclude option for custom-resource-owned workloads is on by default, which is the right setting for most teams running operators.
Scope: where a rule applies
Every rule has a scope. The valid kinds:
organization: every cluster in your org.cluster: a single cluster.namespace: a namespace within a single cluster.team: workloads attributed to a Team (via labels or Assignment Workbench).department: workloads attributed to a Department.
Team and department scopes are dynamic. Kubeadapt resolves membership at evaluation time and compares a stable set of workloads, so attribution changes (a team grows, a workload moves) don't create cost-spike alerts on their own.
Where alerts go
A Rule decides what to fire on. A Policy decides where the alert goes. A Channel is the destination (Slack, email, webhook, in-app). The three layers are separate, so you can send critical alerts to an on-call Slack channel or webhook and informational alerts to a mailing list without duplicating rule definitions.
The full routing model is covered in Policies and Channels.
Incident lifecycle
When a rule fires, an incident is created. Incidents move through five states: pending (just created), firing (notification dispatched), acknowledged (a human marked it seen), snoozed (silenced for a window), resolved (the condition cleared). The state column is visible on every incident; transitions are logged on the timeline.
A rule that's currently muted, disabled, or degraded won't fire new incidents — see Rules for the difference between those states.