Module 4. Error analysis¶
In this module
You will learn: how to find the specific ways your system fails by reading real traces with a domain expert, how to group what you find into failure modes, and how labelling works in practice. This is Step 3 of the lifecycle.
Reading time: about 10 minutes.
Step 3: Look at the data and label it¶
This is the highest-return step in the lifecycle, and the one most often skipped. Before you write a single metric or rubric, read real traces.
Teams that grab an off-the-shelf framework and use its default metrics (helpfulness, coherence, hallucination) get dashboards that look reasonable but are too generic to show what is actually breaking. A helpfulness score of 4.1 out of 5 tells you nothing about why users are leaving.
The process has four parts:
- Sample. Take about 100 diverse traces (30 to 50 is a workable minimum for a first pass). Include ones with negative feedback, and some ordinary ones.
- Annotate with free text ("open coding"). The domain expert reads each trace and writes a short note about the first thing that went wrong, in their own words: "used the old refund policy", "cheerful tone to an angry customer", "didn't escalate". Not a score; a note. ("Coding" here means tagging text with labels, not programming.)
- Group the notes ("axial coding"). Cluster the notes into failure modes: named, specific ways your system goes wrong. "Used the old policy", "quoted a price from the wrong region" and "cited an outdated document" might become one failure mode: stale source. Do not aim for a fixed number; keep going until new traces stop revealing new kinds of failure.
- Count them. Now you know what actually breaks and how often. This is your priority list.
Run the session like this
Format: 90 minutes. The expert reads and writes notes; one engineer scribes and shows each trace in a readable view (not raw JSON).
Pace: aim for 2 to 3 minutes per trace, so 100 traces take two to three sessions.
Rules: note only the first thing that went wrong; write "ok" if nothing did; no scoring; no fixing during the session.
Afterwards: the engineer clusters the notes into failure modes and the expert approves the grouping in a 30-minute review.
Done when the last 20 traces added no new failure mode.
The result is a set of domain-specific failure modes ("Cited a non-existent policy", "Failed to escalate to a human") that are far more useful than generic scores. In the next module you will build one check for each.
For agents and pipelines: count first failures per step¶
An agent or multi-step pipeline runs several steps per request. A failure mode tells you what went wrong. It does not always tell you which step to fix. So, for each failed trace, also record the first step where it went wrong. Then count the failed traces per step.
Why the first step? Errors cascade. If the agent retrieves the old 60-day policy, the reply will quote 60 days too. The reply step did its best with bad input, so the fix belongs in retrieval. Hamel Husain and Shreya Shankar advise noting the first failure in a trace for this reason: upstream errors cause downstream ones. Andrew Ng teaches a closely related method, error analysis by component, in DeepLearning.AI's Agentic AI course (2025): take only the failed examples, read their traces, and count in a spreadsheet how often each step produced output clearly worse than a human expert would have.
In practice, add one column, failed_step, to the sheet from your session. Here is an illustrative count for the refund agent, from 40 failed traces:
| Step | First failures | Share |
|---|---|---|
| Understand the request | 4 | 10% |
| Retrieve the policy | 17 | 42% |
| Look up the order | 6 | 15% |
| Write the reply | 9 | 23% |
| Escalate to a human | 4 | 10% |
How to read it: the step with the most first failures is where to look first. Here that is retrieval, and the notes would show why (for example, the old 60-day document ranks above the current one). Ng adds a second test: also ask whether you have ideas to improve that step. A step that fails often but that you cannot change yet may come second.
Optional: a transition failure matrix for agents with states
When the agent can take different paths (look up the order first, or retrieve the policy first, or escalate straight away), one count per step can hide where the path breaks. Hamel Husain and Shreya Shankar suggest a transition failure matrix: "rows represent the last successful state and columns represent where the first failure occurred". Their FAQ credits one example to Bryan Bischof's talk "Failure is A Funnel" (Data Council, 2025).
The same 40 illustrative failures, as a matrix:
| Last successful step ↓ / First failed step → | Understand | Retrieve policy | Look up order | Write reply | Escalate |
|---|---|---|---|---|---|
| (none) | 4 | 0 | 0 | 0 | 0 |
| Understand the request | 0 | 12 | 4 | 0 | 3 |
| Retrieve the policy | 0 | 0 | 2 | 6 | 0 |
| Look up the order | 0 | 5 | 0 | 3 | 1 |
Each column total matches the per-step count above. The matrix adds the path: the hotspot is understand, then retrieve (12). Read those traces first, and the notes then show why. Before you compare cells, check how many traces took each path, because a busy path collects more failures. Lay out the rows and columns to fit your own agent's steps.
Counting every failed step instead. Ng's spreadsheet marks every step that did clearly worse than an expert would have with the same input, not only the first. A step that did its best with bad input is not marked. His percentages can then add up to more than 100%, because one trace can fail at several steps. Counting only the first failure gives one entry per trace and is quicker. Either works; be consistent within one round of analysis.
If you use this count during development, to decide what to change next and to check that a change helped, read Appendix C. Evals during development.
How labelling works in practice¶
Later (when you build evals and validate judges, Modules 5 and 7) you will label examples pass or fail per failure mode, with a short reason:
| trace_id | failure_mode | label | reason |
|---|---|---|---|
| t-8f21 | uses_current_policy | fail | Used the 60-day policy; the current one is 30 days |
| t-8f21 | tone_appropriate | pass | Polite and clear |
| t-9a07 | escalates_when_required | fail | Customer mentioned a legal complaint; agent kept chatting |
An expert's verdict on an example is called a label, and the set of labels is treated as ground truth: the correct answers everything else is measured against.
The labels come from the one lead domain expert, the benevolent dictator from Module 3. In practice even one expert's time is hard to get, so several annotators are rarely practical.
Before labelling, write a short labelling guide: for each failure mode, a definition and two clear pass examples and two clear fail examples. If the expert hesitates on a case, add it to the guide. The guide keeps the expert consistent with themselves over time, and lets anyone see why a trace was labelled the way it was.
Optional: measuring how reliable your labels are
Your judges are only as good as the labels you test them against (and so is any system that learns from those labels). When a set of evals is critical, there are techniques to make labels more reliable.
Inter-observer reliability (also called inter-rater or inter-annotator reliability) compares labels from two or more experts on the same traces, for example a second person labelling the same 30 or so traces. If they agree on fewer than roughly 80 to 90% of labels (a rule of thumb), tighten the definition before building a judge. If equally expert people disagree, the task may be genuinely hard or ambiguous, or the definition is unclear. Either way, the disagreements are useful input for refining the labelling guide.
Raw agreement can flatter you. If 95% of traces pass, two labellers who mark everything "pass" agree 95% of the time without catching a single failure. So check that they agree on the failures specifically, or use a measure that corrects for chance agreement.
Intra-observer reliability (also called intra-rater or test-retest reliability) checks one expert against themselves. They re-label a random sample of traces they labelled earlier, for example a week before, without seeing their earlier labels. You expect the same verdicts, so any inconsistency shows how reliable that annotator is. Fatigue, stress or time pressure can make the same person label differently on different days.
Other related techniques, in one line each:
- Chance-corrected agreement statistics: Cohen's kappa for two labellers; Fleiss' kappa or Krippendorff's alpha for more than two.
- Adjudication: a third expert, or a discussion, resolves each disagreement, and the resolution goes into the labelling guide.
For most eval work, especially when speed matters or you are starting out, the single lead expert (Hamel Husain's benevolent dictator) with a labelling guide is the way to go. Start simple and add these techniques later, only when the evals are critical, the resources are available and your eval practice is mature.
You can start in a shared spreadsheet with columns trace_id | link to trace | input | output | failure_mode | label | reason | labeller | date, where label is pass or fail. An annotation tool is much better than raw JSON in a spreadsheet; Module 11 describes the workflow and the minimum features to look for.
Who labels matters
Do not hand labelling to whoever is available. A non-expert will mark confidently wrong answers as correct. The domain expert's time is the scarcest input to the whole process, so spend it on reading real traces, not on writing long rubrics.
Done when: you have a ranked list of failure modes with counts, approved by the lead expert and the product owner.
Key takeaways¶
- Read real traces before writing any metric or rubric. Generic scores hide what is actually breaking.
- The expert notes the first thing that went wrong on each trace; you group the notes into failure modes and count them.
- Keep going until new traces stop revealing new failure modes.
- For agents and pipelines, also record the first step that failed in each trace and count per step. Look first at the step with the most first failures.
- Labels come from one lead domain expert (the "benevolent dictator"), guided by a written labelling guide. Checks with several labellers are worth adding later, and only when the evals are critical and you have the resources.
Check your understanding¶
Why does the expert note only the first thing that went wrong?
Later problems are often consequences of the first one, and noting everything slows the session down. The first failure is usually the one to fix, and it keeps the notes easy to group.
If you do add a second labeller and you agree on 95% of traces, is the labelling guide good enough?
Not necessarily. If 95% of traces pass, you could agree 95% of the time by both marking everything "pass". Check your agreement on the failures, or use a chance-corrected measure such as Cohen's kappa.
A trace retrieved the old 60-day policy, and the reply then quoted 60 days. Which step do you record?
Retrieval, because it is the first step that went wrong. The reply step worked with the input it was given. Fixing the reply would hide the problem; fixing retrieval removes it.
When do you stop reading traces?
When new traces stop revealing new kinds of failure, for example when the last 20 traces added no new failure mode.