Core
src/incident_agent — ingestion, detectors, correlation, RCA, grounding, artifact storage. This is the compiler. The UI is a viewer.
Architecture
Schemas in the middle.
The Python package is the system of record. FastAPI exposes jobs, incidents, anomalies, and reports. Next.js 16 is the product surface — Three.js for the horizon, TypeScript for every contract the console touches. Nothing important lives only in a React state tree.
API
FastAPI on :8000 with CORS for the studio, an in-memory job store, and artifacts on disk. Restarts clear jobs; they do not clear JSON.
Studio
web/ — App Router, Geist + Instrument Serif, navy dark with amber action and ice signal. Pages proxy /backend to the API so the browser never hardcodes ports in fetch logic.
Runtime
Four layers. None of them is a local model.
Python 3.12
incident_agent is the system of record. Typed schemas, detectors, correlation, RCA, grounding, artifacts.
FastAPI :8000
Jobs, incidents, anomalies, reports, review, webhook. In-memory job store. JSON on disk survives restarts.
Next.js 16 :3000
Studio and console. /backend rewrites to the API so the browser never hardcodes ports in fetch logic.
No containers
No Docker in the happy path. No local model runtime. Optional OpenAI is a remote rewriter only.
Contracts
Every stage emits a typed object. The next stage refuses anything else.
LogEvent
ingestion
timestamp, service, severity, message
MetricPoint
ingestion
timestamp, service, metric_name, value
AnomalyCandidate
detectors
type, window, observed, baseline, severity
CorrelatedIncidentCandidate
correlation
incident_id, services, evidence, score
RootCauseHypothesis
RCA
origin service, support, rationale, ambiguities
FinalIncidentReport
compose
summary, RCA, exec, handoff, facts, citations, review
Package map
Where the work actually lives
ingestion/
Typed log and metric parsers, quality counts
anomaly_detection/
Independent detectors per signal family
normalization/
UTC conversion and bucket alignment
correlation/
Graph + temporal grouping
rca/
Evidence ranking and origin scoring
grounding/
Claim overlap and policy
knowledge/
Runbook and corpus retrieval
export/
JSON / Markdown / HTML / webhook
eval/
Benchmark runner, golden compare, mode matrix
api/
FastAPI jobs, review, webhook, samples
HTTP
The console only speaks these routes
GET
/health
Liveness for the console pill
GET
/config
Effective YAML after load, without secrets
GET
/workspace/samples
Bundled scenarios for one-click runs
POST
/analysis-jobs
Execute the file pipeline and store the job
GET
/analysis-jobs
List in-memory jobs, newest first
GET
/analysis-jobs/{id}
Incidents, anomalies, and reports together
GET
/analysis-jobs/{id}/reports
Reports for one job, filterable by review
POST
/analysis-jobs/{id}/reports/{inc}/review
draft → reviewed → approved | rejected
POST
/analysis-jobs/{id}/reports/{inc}/export-webhook
POST an approved report to an allowlisted URL
GET
/incidents
Candidates across jobs or one job_id
GET
/anomalies
Detector ledger for the board
POST
/analyze-pipeline
Synchronous full run without the job store
On disk
A run is a directory, not a chat log
normalized/timeline.json
UTC buckets and aligned events
anomalies/anomalies.json
Every detector hit with observed vs baseline
incidents/incidents.json
Correlated candidates and evidence lists
rca/rca_hypotheses.json
Ranked origin, support, ambiguities
grounding/grounding_summary.json
Claim overlap and policy result
reports/final_reports.json
The document the console renders
run_summary.json
Stages, warnings, degraded flags, token usage
exports/webhook_deliveries.jsonl
Audit of outbound approved reports
CLI
The console is a viewer. The compiler also has a keyboard.
run-demo
Deterministic portfolio run under artifacts/demo/portfolio-demo
run-pipeline
Full compiler: ingest through compose and persist artifacts
detect-anomalies
Detectors only — useful when you are tuning YAML gates
correlate-incidents
Group detector hits with the dependency graph
run-rca
Rank origin without generating prose
list-reports / show-report
Inspect review state and a single document
export-report
JSON, Markdown, or HTML from an artifact directory
run-eval / compare-eval
Benchmark modes against eval/golden/baseline_summary.json
Samples
What GET /workspace/samples returns
checkout-cascade
Saturated checkout-service: CPU, memory, latency, error rate, traffic drop, unavailability, and an error-log burst in overlapping five-minute windows. Expected: one correlated incident, origin checkout-service.
healthy-baseline
Quiet traffic, no detector support. Expected: a completed job with an empty ledger. If this table is not empty, the gates are too hungry.
degraded-partial
Missing-signal resilience. The run summary warns. Detectors that still have support fire. The board still renders.
Quality
CI is part of the architecture
pytest + coverage
85% coverage gate on the Python core. Detectors and RCA have adversarial fixtures.
ruff + mypy
Lint and strict types on every contract the pipeline emits.
eval regression
compare-eval against golden summaries. A detector change that moves F1 fails CI.
CodeQL + Dependabot
Static analysis and dependency review on the same cadence as the product UI.
Pipeline
01
Ingest
CSV, JSON, and JSONL become typed log events and metric points. Invalid rows are counted, not silently dropped.
Log ingestion accepts .csv, .json, and .jsonl. Metric ingestion accepts the same plus a Prometheus query_range adapter. Each row is validated against LogEvent or MetricPoint. Quality metrics — parse failures, missing timestamps, unknown services — land in the run summary instead of disappearing.
02
Normalize
Timestamps lock to UTC. Signals align into five-minute buckets so detectors share one clock.
Naive timestamps are assumed UTC. Offset-aware values are converted. Events then fold into configurable buckets (default five minutes) so a latency spike and an error-log burst in the same window can actually meet.
03
Detect
Latency, error rate, CPU, memory, traffic, and availability each have their own z-score and MAD gates.
Detectors are independent. A quiet CPU series does not suppress an error-rate spike. Support windows, z-thresholds, MAD multipliers, and minimum relative change all live in configs/default.yaml.
04
Correlate
Anomalies group by time, service, and the dependency graph. Isolated noise stays isolated.
Temporal distance, same-service bonus, dependency edges, and cross-signal bonuses are weighted. A lonely traffic blip without companions does not become an incident.
05
Rank evidence
RCA scores origin vs blast radius. Downstream pain is not automatically the cause.
checkout-service saturating while api-gateway pages is a classic trap. The scorer applies a downstream bonus and an upstream penalty so the origin is preferred when the graph agrees.
06
Ground
Every claim is checked against detector output and retrieved runbooks. Unsupported prose is marked.
Grounding policy can warn or fail. Overlap against evidence ids is measured. Citations from runbooks and historical incidents attach as snippets, not as vibes.
07
Compose
The report is assembled from ranked evidence. Optional Groq or OpenAI can rewrite; the facts never leave the bundle.
Default compose is a deterministic narrative over the evidence JSON: summary, RCA, executive, handoff, remediations. The console shows facts on the left and the rewrite on the right. Toggle heuristic vs Groq on the same incident — detectors do not run twice.
08
Review
Draft, reviewed, approved, rejected. Approved reports can leave through a webhook with an audit log.
Transitions require a reviewer name and a note. Webhook destinations must match the allowlist. Delivery attempts are appended as JSONL under the run's exports directory.