Incidents

Correlated candidates

Correlation groups detector hits by time, service, and the dependency graph. Isolated spikes stay out of this list.

Run a scenario from Overview to populate this view.

Why this is not an alert list

Fourteen detector hits in the checkout cascade collapse to one incident because they share a window, a service, and a graph neighborhood. If you still see fourteen rows here, correlation failed — that is a bug, not a dashboard preference.

Reading the score

Correlation score is a weighted sum: time proximity, same service, dependency edges, cross-signal bonus. Higher is tighter. It is not a probability and it is not severity. Severity lives on the evidence rows inside the candidate.

Empty board

Healthy baseline is supposed to produce no incidents. Degraded partial may produce a degraded run with warnings and still zero candidates. Only Checkout cascade is guaranteed to light this list in a clean clone.

How the score is built

Weights live in configs/default.yaml under correlation. Same-family evidence is intentionally weak and cannot cluster two spikes alone.

temporal

0.4

Anomalies must share a window inside max_time_distance_minutes (10)

same_service

1.0

Hits on one service are the strongest join

dependency

0.8

Direct graph edges, not transitive rumor

cross_signal

0.5

CPU plus latency plus errors on the same owner

same_family

0.2

Intentionally weak — two latency spikes cannot cluster alone

relationship_threshold

1.0

Lonely spikes stay anomalies, not incidents

Do not

  1. 01

    Do not restart every waiter

    Downstream 504s are blast radius. The origin is the service the graph and the detectors agree on.

  2. 02

    Do not treat support as probability

    0.76 root-cause support means the top candidate outscored the rest in this incident. It does not mean 76% chance in the wild.

  3. 03

    Do not silently edit facts

    Facts are detector rows. If the graph was wrong, reject the report with a note. That note is how the corpus learns.

  4. 04

    Do not ship a detector you cannot regress

    Change a z-threshold, run eval/benchmarks, compare against eval/golden. If root-cause correctness moves, the gate should fail.