Incident F1
Did correlation produce the expected candidate set?
Grounded analysis · heuristic compose · optional Groq or OpenAI
Regression
Benchmarks live in eval/benchmarks. Golden summaries live in eval/golden. The console is honest about what the harness measures. If a detector change moves root-cause correctness, the gate should fail — not the landing page copy.
heuristic-only
F1 0.94
Root 76.7%
Detectors and RCA without generated prose.
grounded-compose
F1 0.94
Root pipeline
Same analysis, narrative composed from the evidence bundle.
openai-optional
F1 opt-in
Root opt-in
Remote rewrite. Never the default path.
mock-llm-retrieval
F1 harness
Root harness
Eval-only path for citation and completeness gates.
Did correlation produce the expected candidate set?
Did RCA name the labeled origin service?
Blast radius vs the benchmark graph.
Which detector families fired when they should.
Share of report claims with evidence overlap.
Did compose invent a service not in the bundle?
Runbook / corpus snippets actually attached.
Required sections present even under degradation.
From the repo: incident-agent run-eval --benchmark-path eval/benchmarks/scenarios.json --artifact-root artifacts/eval then incident-agent compare-eval against eval/golden/baseline_summary.json. Real-LLM modes are opt-in and require INCIDENT_AGENT_OPENAI_API_KEY. Default CI does not call a remote model.
pytest + coverage
85% coverage gate on the Python core. Detectors and RCA have adversarial fixtures.
ruff + mypy
Lint and strict types on every contract the pipeline emits.
eval regression
compare-eval against golden summaries. A detector change that moves F1 fails CI.
CodeQL + Dependabot
Static analysis and dependency review on the same cadence as the product UI.
Benchmark labels include incident_expected, expected_root_cause, allowed_root_causes, expected_impacted_services, and expected anomaly types. A detector change that moves root-cause correctness should fail compare-eval — not a screenshot on this page.