Building an Incident Assistant On-Call Engineers Trust

The value of an incident assistant is decided in the first ten minutes of an incident, when a woken engineer is looking at one alert and does not yet know which of forty services is involved. Everything after that point is diagnosis, and diagnosis is not what to automate first.

The reason is asymmetry. Gathering context is tedious, mechanical and safe to get slightly wrong. Diagnosis is judgement, and a confident wrong hypothesis at 3am does not merely fail to help — it costs twenty minutes of someone investigating the wrong subsystem while the incident continues.

Automate the gather, not the conclusion

Write down what a competent responder does in their first ten minutes. It is roughly the same list every time: what deployed recently, which alerts fired and in what order, what the error rate and latency look like against yesterday, whether a dependency is degraded, whether this alert has fired before and what closed it last time.

Every item on that list is a query against a system you already have. None of them requires a model to answer — a model is only needed to decide which ones are relevant to this alert and to arrange the results into something readable in one screen.

Build that first and ship it before any hypothesis generation. A bot that posts, within thirty seconds of the page, a compact block of deploys, alert sequence, dependency status and the three most similar past incidents is useful on day one and cannot be wrong in a way that costs anyone time. If the block is unhelpful the responder ignores it and loses nothing.

Read-only tools, no exceptions in version one

The temptation to let the assistant act — restart a pod, roll back a deploy, scale a group — should be resisted for longer than feels necessary. An incident is precisely the situation where your monitoring is degraded, your assumptions are wrong, and an automated action can convert a partial outage into a total one.

Give it query tools only: logs, metrics, traces, deploy history, alert history, the on-call schedule, the service catalogue. Every tool should be scoped to read a single system and should be impossible to misuse into a write. The isolation argument in agent sandboxing applies with more force here than in a development loop, because the blast radius is production during an outage.

If you eventually add actions, add them as a proposal the responder confirms with a single click, with the exact command shown. That keeps a human in the decision, which is the only defensible position for an irreversible operation — the trade-offs are covered in human-in-the-loop design.

Every claim carries a link

The output contract that makes this trustworthy is simple: no sentence without a source. Each line is a claim plus a link to the dashboard, log query or deploy record that produced it.

This does two things. It lets the responder verify anything in two seconds instead of taking it on faith, and it makes the failure mode visible — a claim without a link is either an inference or an invention, and both should be labelled as such rather than presented alongside facts.

Structure the output so inference is visually separated from observation. An "observed" block of linked facts, then a clearly marked "possible" block of at most three hypotheses, each with the specific check that would confirm or rule it out. A hypothesis paired with a falsifying test is useful even when it is wrong; a hypothesis on its own is a distraction. The general problem is worth understanding — why LLMs hallucinate explains why confident unsupported claims are the default rather than an aberration.

The latency budget is the hard constraint

An assistant that posts its context block after the humans have already assembled it is worse than nothing, because it adds a message to a channel people are trying to read.

Budget thirty seconds from page to post, and design backwards from there. That means the deterministic queries fire in parallel the moment the alert arrives, before any model call. It means one model call to select and arrange, not an agent loop with six sequential tool calls. It means a partial post at the deadline with whatever came back, marked as partial, rather than waiting on a slow metrics backend.

Model choice matters here in a way it usually does not — this is a latency-bound workload, and the considerations in picking a model for low latency outrank raw capability. A faster model with linked evidence beats a stronger model that arrives after the responder has moved on.

Correlation is a join, not a prompt

The most common design mistake is dumping raw logs, metrics and deploy records into a context window and asking the model to correlate them. That is expensive, slow, and unreliable, because the model has to do exact identity matching across long inputs, which is the thing it does worst.

Do the joins in code. Deploys and alerts join on time window and service. Traces and logs join on trace ID. Alerts and past incidents join on alert name plus service. Compute those relationships deterministically and hand the model a small, already-correlated table.

Log volume in particular needs reducing before it reaches a model at all — cluster by template, count occurrences, and send the top few patterns with counts and one example each rather than ten thousand lines. The techniques in using an LLM for log analysis are the difference between a useful summary and an expensive one.

Similar past incidents are the highest-value retrieval

The single most useful thing an assistant can surface is: this alert fired four times before, here is what closed it. That is a retrieval problem over your own postmortems and incident channels, and it does not need a frontier model.

Index every resolved incident with its alert names, affected services, symptom text and resolution. Retrieve on the current alert and show the top three with their resolutions. In organisations with any operational history, a meaningful share of pages are repeats, and this alone justifies the project.

It also creates a virtuous loop: the better the postmortems, the better the retrieval, which gives people a reason to write them properly.

Postmortem drafting is the safer half of the product

After the incident, the assistant has the entire timeline it collected, the chat transcript, the deploys and the resolution. Drafting a timeline from that is low risk, genuinely tedious for a human, and does not run against a clock.

Generate the factual timeline with links, the list of people involved, and the detection and mitigation durations. Leave the contributing factors and the actions to humans — those are the parts that require organisational judgement, and a model-written cause analysis reads as generic in a way that quietly devalues the whole document.

Instrument the assistant itself the way you would any other agent, so you can answer which tool calls it made and why after the fact. Agent observability and tracing covers the shape of that; during an incident review, "the bot said X" is not an acceptable end of a trail.

Evaluate against incidents that already happened

You have a test set: your last thirty incidents, with known resolutions. Replay the initial alert for each and score the output on whether the affected service was correctly identified, whether the causal deploy appeared in the context block, and whether any stated hypothesis was misleading.

Track the misleading rate specifically, and treat it as the metric that gates rollout. One misleading hypothesis per ten incidents is probably tolerable if the context block is good; one per three is not, and the fix is to remove hypothesis generation rather than to tune the prompt.

Then roll out to one team, in a channel they can mute, with an explicit instruction that nothing the bot says is authoritative. Trust in an incident tool is built over months and lost in a single outage where someone chased a fabricated cause. Ship the boring half first and let the rest earn its way in.