References and further reading

Hamel Husain and Shreya Shankar

Many of the ideas in this course follow their writing.

• Shreya Shankar and Hamel Husain, Evals for AI Engineers: Systematically Measuring and Improving AI Applications (O'Reilly, 2026; source of the quote on the welcome page).
• Hamel Husain, AI Evals: Everything You Need to Know. A good first read after this course.
• Hamel Husain, Your AI Product Needs Evals and A Field Guide to Rapidly Improving AI Products.
• Hamel Husain, Using LLM-as-a-Judge For Evaluation: A Complete Guide.
• Hamel Husain, The Revenge of the Data Scientist (source of several pitfalls in Module 10).
• Hamel Husain and Antaripa Saha, How To Use Decision Models for Evals.
• Hamel Husain, Selecting The Right AI Evals Tool.
• Hamel Husain and Shreya Shankar, How are evaluations used differently in CI/CD vs. monitoring production?.
• Hamel Husain and Shreya Shankar, How do I evaluate agentic workflows? and How do I evaluate complex multi-step workflows? (source of the transition failure matrix: rows are the last successful state, columns are where the first failure occurred; one example is credited to Bryan Bischof's talk "Failure is A Funnel", Data Council 2025).
• Hamel Husain and Shreya Shankar, Why is error analysis so important in LLM evals, and how is it performed? (noting the first failure in each trace, then counting failures per category) and How often should I re-run error analysis on my production system? (re-run after significant changes such as prompt updates and model switches).

Andrew Ng and DeepLearning.AI

• Andrew Ng, Agentic AI course, Module 4 (DeepLearning.AI, 2025): Error analysis and prioritizing next steps, More error analysis examples and Component-level evaluations. Error analysis on failed examples only: read the traces and count, often in a spreadsheet, how often each component did materially worse than a human expert would, then focus on the component that fails most often.
• Andrew Ng, The Batch, Improve Agentic Performance with Evals and Error Analysis, Part 1 (15 October 2025) and Part 2 (22 October 2025).

Playbook

• Jasmeet Kaur, Production AI Evaluation & Assurance Playbook (source of the tiered evaluation layers and the starting TPR and TNR targets).

Papers

• Shreya Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (source of the term criteria drift).
• Yang Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.
• Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
• Lin Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge.
• Saito et al., Verbosity Bias in Preference Labeling by Large Language Models.
• Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations.
• Verga et al., Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models.
• Xiong et al., Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.
• Tian et al., Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.
• D. Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015; source of "Changing Anything Changes Everything").
• João Gama et al., A Survey on Concept Drift Adaptation (ACM Computing Surveys, 2014).
• Mark Chen et al., Evaluating Large Language Models Trained on Code (2021; the Codex and HumanEval paper, which made pass@k popular and gives its unbiased estimator. It credits Kulal et al., 2019, with introducing pass@k).
• Shunyu Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024; source of pass^k).
• Evan Miller (Anthropic), Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024; why evals should report confidence intervals).
• Lingjiao Chen, Matei Zaharia and James Zou, How is ChatGPT's behavior changing over time? (figures quoted from the current version of the paper).

Drift and production monitoring

• Chip Huyen, Data Distribution Shifts and Monitoring.
• Evidently AI, What is data drift in ML.
• Arvind Narayanan and Sayash Kapoor, Is GPT-4 getting worse over time? (the behaviour versus capability distinction).
• LangSmith, Online evaluations.
• AWS, SageMaker shadow tests (the shadow-testing pattern).

Model deprecation and retirement policies

Checked 24 October 2026; these pages change.

• OpenAI, Deprecations.
• Anthropic, Model deprecations and Models overview.
• Anthropic, A postmortem of three recent issues (September 2025; infrastructure bugs that degraded outputs with no change to the model ID).
• AWS, Amazon Bedrock model lifecycle (models launched before 7 September 2026) and Model lifecycle (newer models).
• Microsoft, Foundry Models lifecycle and support policy.
• Google, Gemini API deprecations.
• OpenAI Developer Community, GPT-4o default model will be updated to latest version on October 2nd, 2024.
• Jigsaw, Perspective API (shutdown notice: 31 December 2026).

"Newer is not automatically better"

• OpenAI, OpenAI o3 and o4-mini System Card (April 2025; PersonQA accuracy and hallucination rates). Further reading: TechCrunch, OpenAI's new reasoning AI models hallucinate more.
• OpenAI, Expanding on what we missed with sycophancy (May 2025; the GPT-4o rollback). Further reading: TechCrunch, OpenAI explains why ChatGPT became too sycophantic.
• OpenAI, Reasoning models guide (reasoning tokens are billed as output tokens).

Other guidance and documentation

• Anthropic, Demystifying evals for AI agents (see its section on non-determinism for pass@k and pass^k).
• Eugene Yan, Task-specific LLM evals that do and don't work.
• OpenTelemetry GenAI semantic conventions.
• scikit-learn user guide, Cross-validation: evaluating estimator performance (section 3.1; a plain introduction to k-fold cross-validation).
• Tool documentation: Promptfoo, DeepEval, Langfuse annotation queues.