Method
The pipeline is a sequence of contracts. LogEvent and MetricPoint in. AnomalyCandidate out of detectors. CorrelatedIncidentCandidate out of the graph. RootCauseHypothesis out of scoring. FinalIncidentReport last, with claim citations mapped back to evidence ids.
Optional OpenAI is a rewriter sitting on those contracts. The default path composes analyst copy from the evidence JSON itself. There is no Ollama sidecar, no downloaded weights, no mock “Prompt size=” theater. If you cannot point at a detector row, you do not get to claim a root cause.
Order of operations
01
Log ingestion accepts .csv, .json, and .jsonl. Metric ingestion accepts the same plus a Prometheus query_range adapter. Each row is validated against LogEvent or MetricPoint. Quality metrics — parse failures, missing timestamps, unknown services — land in the run summary instead of disappearing.
02
Naive timestamps are assumed UTC. Offset-aware values are converted. Events then fold into configurable buckets (default five minutes) so a latency spike and an error-log burst in the same window can actually meet.
03
Detectors are independent. A quiet CPU series does not suppress an error-rate spike. Support windows, z-thresholds, MAD multipliers, and minimum relative change all live in configs/default.yaml.
04
Temporal distance, same-service bonus, dependency edges, and cross-signal bonuses are weighted. A lonely traffic blip without companions does not become an incident.
05
checkout-service saturating while api-gateway pages is a classic trap. The scorer applies a downstream bonus and an upstream penalty so the origin is preferred when the graph agrees.
06
Grounding policy can warn or fail. Overlap against evidence ids is measured. Citations from runbooks and historical incidents attach as snippets, not as vibes.
07
Default compose is a deterministic narrative over the evidence JSON: summary, RCA, executive, handoff, remediations. The console shows facts on the left and the rewrite on the right. Toggle heuristic vs Groq on the same incident — detectors do not run twice.
08
Transitions require a reviewer name and a note. Webhook destinations must match the allowlist. Delivery attempts are appended as JSONL under the run's exports directory.
Detectors
Thresholds are not hidden in a notebook. They live in configs/default.yaml under anomaly_detection, with min_support, lookback_windows, z_threshold, mad_multiplier, and min_relative_change per family.
error_rate_spike
HTTP 5xx / error_rate
z ≥ 2.5 · MAD × 2.5 · Δ ≥ 15%
latency_spike
p95 / request_latency_ms
z ≥ 2.5 · MAD × 2.5 · Δ ≥ 20%
cpu_anomaly
cpu_usage / cpu_percent
z ≥ 2.5 · lookback 20 buckets
memory_anomaly
memory_usage_mb
z ≥ 2.5 · relative change 15%
traffic_drop
rps / throughput
z ≥ 2.2 · MAD × 2.2
service_unavailability
upstream_failure / unavailable
z ≥ 2.0 · higher severity weight
error_log_burst
ERROR log count
burst vs baseline · min support 3
critical_log_burst
CRITICAL log count
severity weight 1.2 in correlation
Correlation & RCA
Anomalies within max_time_distance_minutes can join. Same-service weight is 1.0, dependency edges 0.8, cross-signal 0.5. A relationship_threshold of 1.0 keeps lonely spikes from becoming incidents. The graph path is configs/service_dependencies.yaml.
Root-cause support is the top candidate score divided by the sum of all candidate scores. Downstream pain gets a bonus so it is visible, and an upstream penalty so it is not automatically the cause. Unresolved ambiguities stay on the hypothesis.
temporal
0.4
Anomalies must share a window inside max_time_distance_minutes (10)
same_service
1.0
Hits on one service are the strongest join
dependency
0.8
Direct graph edges, not transitive rumor
cross_signal
0.5
CPU plus latency plus errors on the same owner
same_family
0.2
Intentionally weak — two latency spikes cannot cluster alone
relationship_threshold
1.0
Lonely spikes stay anomalies, not incidents
Bundled scenarios
checkout-cascade
Checkout cascade
Saturated checkout-service: CPU, memory, latency, error rate, traffic drop, unavailability, and an error-log burst in overlapping five-minute windows. Expected: one correlated incident, origin checkout-service.
healthy-baseline
Healthy baseline
Quiet traffic, no detector support. Expected: a completed job with an empty ledger. If this table is not empty, the gates are too hungry.
degraded-partial
Degraded partial
Missing-signal resilience. The run summary warns. Detectors that still have support fire. The board still renders.
Report anatomy
Incident summary
What happened and when, using only services present in evidence. Inference is withheld until RCA.
Root cause
Ranked origin, support, rationale, ambiguities. Downstream pain is visible and not automatically the cause.
Executive
The paragraph a VP can read. No detector jargon. Uncertainties stay labeled.
Engineering handoff
The section that belongs in the ticket: windows, facts, next checks, who to page.
Facts / inferences / uncertainties
Three lists. Only facts may be treated as ground truth. Inferences are labeled. Uncertainties stay on the page.
Remediations
Advice, not claims about the past. Grounding marks them not_applicable so they cannot fake overlap.
Grounding labels
supported
Overlap with an evidence id or hypothesis rationale above minimum_support_overlap (0.34).
unsupported
Fluent sentence, no bundle overlap. Stays in the report with a reason.
contradictory
Reserved for claims that fight detector output. Policy fail can drop the report.
not_applicable
Remediations and advice. They are not facts about the past.
After the page
Contain
Roll back or shed load on the ranked origin. Do not restart every downstream waiter because they are noisy.
Verify
Re-run the same log and metric files. Confirm the detector windows still exist. If they vanished, you contained the symptom, not the cause.
Handoff
Paste the engineering section into the ticket. Facts stay attached to evidence ids so the next shift does not re-litigate the origin.
Review
Mark reviewed, then approved when the owner agrees. Reject with a note if the graph was wrong — that note is how the corpus learns.
Anti-patterns
01
Downstream 504s are blast radius. The origin is the service the graph and the detectors agree on.
02
0.76 root-cause support means the top candidate outscored the rest in this incident. It does not mean 76% chance in the wild.
03
Facts are detector rows. If the graph was wrong, reject the report with a note. That note is how the corpus learns.
04
Change a z-threshold, run eval/benchmarks, compare against eval/golden. If root-cause correctness moves, the gate should fail.
Detectors and correlation do not wait on a model. If generation is unavailable, the incident still exists.
Facts carry evidence ids. Inferences are labeled. The UI never presents a paragraph as gospel.
Missing logs, missing metrics, and provider failure become warnings in the run summary, not a blank page.
Every run writes JSON the same way a compiler writes IR. Reviewers can disagree with the machine, in public.