Glossary

Every term is also explained in the module where it first appears. Terms are in alphabetical order.

Term	Meaning in this course
Accuracy	The share of all verdicts a judge gets right. Misleading when one outcome is rare: see why not accuracy and Pitfall 5.
Alias	A model name the provider can repoint to a newer snapshot, such as a dateless or -latest name. See pinned snapshots and floating aliases.
Benevolent dictator	The single lead domain expert whose pass or fail judgement is final; Hamel Husain's term. Preferred over a panel of labellers because it gives consistent labels and one expert's time is easier to get. See Module 3 and Module 4.
Calibration	Checking, and if needed correcting, whether a model's scores match reality: whether outputs scored 0.9 are right about 90% of the time. Matters only when you act on a numeric score. See Module 8.
Capability eval, regression eval	Two kinds of system test, both run from the regression set. Capability eval: inputs the system still fails, used to measure progress; expect a pass rate well below 100% and do not gate on it. Regression eval: inputs that used to pass and must keep passing; expect close to 100%. Neither is the golden dataset, which tests the judge. See Module 5 and Pitfall 5.
Class imbalance	When one outcome (usually failure) is much rarer than the other. It makes accuracy misleading and random samples short of failures. See Module 7.
Classifier	Any model that assigns an input to one of a fixed set of labels, for example PASS or FAIL. LLM judges, decision models and pre-trained models are all used as classifiers.
Confidence interval	A range of plausible values for a true rate, given how few examples you measured. Optional, but useful when a test set is small or two versions differ by only a few points. See Module 7.
Confusion matrix	A table counting the four ways a judge can be right or wrong: true and false positives, true and false negatives. See Module 6.
Cross-validation	A way to tune and measure on the same small labelled set without fooling yourself. k-fold cross-validation splits the data into k equal parts, tunes (or trains) on k-1 of them and tests on the one left out, repeats this so each part is the test set once, and reports the average score. See the threshold recipe and Module 7.
Decision model	A small, fast model that returns a typed verdict with a probability, such as "true, 0.87", instead of free text. See Module 6.
Deprecation, retirement	Deprecation: the provider announces that a model version will go away and stops recommending it. Retirement (or shutdown, end of life): calls to it stop working. The provider sets the date; your team chooses the replacement.
Dev set, test set, few-shot pool	The three stratified splits of the labelled pool. Few-shot pool: examples you may paste into the judge prompt. Dev set: where you iterate on the prompt and, for scores, tune the threshold and fit calibration. Test set: used once to report TPR and TNR. See Module 5.
Drift	A sustained change, after you evaluated the system, in the conditions it runs under. Kinds: data, concept, criteria, upstream, model and judge. See Appendix B.
Error analysis	Reading real traces with a domain expert to find, name and count the ways your system fails. See Module 4.
Eval (evaluation)	Any systematic way of measuring how good a model's or agent's output is. Broader than a test: an eval can use code checks, models or humans as graders.
F1	A single number combining precision and recall (their harmonic mean) that is high only when both are high. The usual choice when both kinds of mistake matter about equally. See Module 6.
Failure mode	A named, specific way your system goes wrong, such as "cites an outdated policy". You build one check or judge per failure mode.
Faithfulness	In RAG, whether an answer says only what its retrieved sources support, with no added claims. An answer can be faithful and still wrong if the sources are wrong. See Module 1.
Few-shot, zero-shot	Zero-shot: a model used as-is on a task, with no worked examples in the prompt. Few-shot: a few worked examples are included in the prompt. Neither removes the need for labelled data to validate the model. See zero-shot. For how few-shot examples are chosen and kept apart from the dev and test sets, with an example prompt, see the few-shot pool.
First-failure count, transition failure matrix	First-failure count: during error analysis, note only the first thing that went wrong in each trace, because upstream errors cause downstream ones, then count them per failure mode or, for agents and pipelines, per step. Transition failure matrix: for multi-step agents, a table whose rows are the last step that succeeded and whose columns are the step where the first failure occurred, so the steps where failures cluster stand out (Hamel Husain and Shreya Shankar). See Module 4.
Golden dataset	The expert-labelled pool: rows of input, output, a pass or fail label per failure mode, and a reason. The labelled rows test your judges; the inputs also feed the regression set. Not a "golden file" from snapshot testing. See Module 5.
Grader	Whatever decides pass or fail on one output. Three kinds: code, model and human. See Module 2.
Ground truth, label	The expert's verdict on an example, treated as the correct answer.
Guardrail	A classifier at the input or output boundary of an LLM system that blocks, denies or redacts, for example blocking a jailbreak attempt. Same machinery as an eval, different action. See Module 2.
Judge, LLM-as-judge	A grader that is itself a model, most often an LLM prompted with a description of what to check. Its verdicts must be tested against expert labels.
Logit	A raw, unbounded number a classifier produces internally before it is converted to a score between 0 and 1.
Non-deterministic	Can give different outputs for the same input. Hosted LLMs often are, even at the lowest temperature. Deterministic does not mean correct.
Offline eval, online eval	Offline: run against a fixed dataset, for example in CI. Online: run against a sample of live production traffic. You need both.
Overfitting	Tuning something until it looks perfect on examples it has seen, so that it fails on new ones. Prevented by a held-out test set.
pass@k, pass^k	Two ways to summarise k runs of the same input when the system varies between runs. pass@k: at least one run passes (can it do this at all?). pass^k: every run passes (does it do this every time?). See Module 9.
Precision, recall	Precision: when a judge flags a problem, how often it is right. Recall: of the real problems, how many it catches. See Module 6.
RAG	Retrieval-augmented generation: search your documents, paste the best matches into the prompt, then ask the model to answer from them.
Red teaming	Deliberately attacking your own system (prompt injection, jailbreaks, tool misuse) to find weaknesses. See Module 9.
Regression set	Inputs you re-run on every change, and on a schedule, to check the system did not get worse: the golden dataset's inputs plus hand-written edge cases. The new outputs are scored by code checks and validated judges, not by the old labels. See Module 5.
Rubric	A criterion broken into several small yes/no questions, each judged on its own. See Pitfall 1.
Snapshot (pinned version)	A fixed model version ID, such as gpt-4o-2024-08-06.
Temperature	A setting that controls how random an LLM's output is. Lower is less random, but hosted LLMs can still vary at the lowest setting, and some models do not let you set it.
Threshold	The cut-off that turns a numeric score into a verdict. A business decision, not a default. See Module 8.
TPR (true positive rate)	With "positive" meaning a good output: the share of genuinely good outputs the judge passes. See Module 6.
TNR (true negative rate)	The share of genuinely bad outputs the judge catches. See Module 6.
Trace, span, session	A trace is the full record of one request through your system, as a tree of spans (one per step: an LLM call, a retrieval, a tool call). A session groups the traces of one conversation.
Training, weights	A model's behaviour is produced by training: automatically adjusting millions or billions of numbers (the weights) until its outputs fit a large set of examples. You cannot read its rules, so you test its behaviour.