16% of the exam — about 10 of the 63 questions — across 6 sub-objectives.
Domain 4 has one organizing idea. Evals come before the code, not after it as a QA step. Writing them first forces you to say what success means in measurable terms, exposes design assumptions while changing them is still cheap, and gives you a gate that can tell whether a model swap or a prompt change actually helped.
The practical test is blunt: if you cannot write an eval for a behavior, you have not defined the behavior well enough to build it.
Questions here usually hand you a measurement and four fixes. Three of them address the half of the system the measurement has already cleared, so read the diagnostic before choosing.
Define evaluation metrics (accuracy, latency, cost, safety, security)
A metric earns its place by being tied to a business requirement and expressed as a threshold somebody agreed to. Accuracy above ninety percent on a defined dataset is a metric. Better answers is not.
The exam asks you to turn a stated business need into something measurable, and the credited answers name both the number and what it is measured against. Watch for scenarios where the obvious metric misses the point: a summarisation system scored on similarity to a reference summary can score well while omitting the one clause that mattered.
Cost and latency belong in the metric set from the start rather than arriving as a surprise in production. So does safety, expressed as a rate rather than an absence, since a system that has not yet produced a harmful output has not demonstrated it cannot.
From the free practice form
A support-triage assistant scores 92% classification accuracy on a 500-ticket curated eval set, and the team declares it production-ready. Two weeks after launch, agents complain that responses take 11 seconds at p95 during peak hours, and one response echoed a customer's full credit card number from the ticket body. Leadership asks why evaluation didn't catch either problem. What was the core evaluation design flaw?
Design evaluation datasets and test frameworks using mixed methodologies
The dataset is the work. Assemble the inputs the system will genuinely meet, including edge cases and counterexamples, because a suite made of comfortable examples reports a number that means nothing.
Three grading methods exist and mixing them is the point. Code-based checks are deterministic, run in milliseconds, and cost nothing, so use them for anything with a single correct answer such as valid JSON or a required field. Model-based judges handle qualities needing interpretation, like tone or reasoning quality, and cost about as much as the call being judged. Human review is for high-stakes or novel behavior where neither of the others can be trusted, and it does not scale, which is why you reserve it.
The wrong answer in this sub-objective is usually a judge model applied to something a code check would settle exactly, or human review proposed for a volume nobody could staff.
From the free practice form
To build an eval set for a document Q&A system, a team samples 300 real user queries from production logs, filtering to conversations that received a thumbs-up so that reference answers are trustworthy. The system scores 96% on this set, but a manual audit of live traffic estimates only 78% of production answers are acceptable. What is the most likely explanation for the gap?
Second question, same objective
A contract-analysis system handles 2,000 queries per day. The eval framework runs deterministic checks (JSON schema, required citation fields) on every response and an LLM-judge for answer quality on a 10% sample. The team can afford expert human review of only 50 responses per week. To make the mixed-methodology framework most trustworthy, where should the scarce human review be focused?
Conduct A/B testing and iterative improvements
An eval suite is what makes a change decidable. Without one, a prompt change is an opinion.
The failure worth knowing is a change that raises the mean while quietly regressing a subset. Aggregate scores hide this, and the exam tests whether you look underneath the average. Segment the results, and hold out a set the change was not tuned against.
Change one thing at a time. A scenario that swaps the model and rewrites the prompt together and then reports an improvement cannot attribute it, and the credited answer usually says so.
From the free practice form
An engineer iterates on a new prompt (v2) for an extraction pipeline, repeatedly testing against a 200-example eval set and tweaking wording until failures on it are fixed. In the final comparison on that same 200-example set, v2 scores 89% versus v1's 84%, and the team prepares to ship. What is the most serious flaw in this comparison?
Second question, same objective
A team ships assistant variant B to its 4,000 opted-in power users while the remaining 36,000 users stay on variant A. After two weeks, B shows a 6-point higher thumbs-up rate (58% vs 52%, n large enough for significance), and the team cites this as proof B is better. Why is this conclusion unsafe?
Diagnose system issues (prompt failure, hallucinations, model mismatch)
Diagnosis questions supply a measurement so you can tell which half is broken, and the distractors are legitimate improvements aimed at the other half.
Retrieval recall reported at ninety-four percent rules out retrieval. The fault is in generation, and options offering a better chunker, more retrieved documents, or a different index are all improvements to something already working. The same shape runs in reverse: a model returning fluent answers about documents it was never given is a retrieval problem, not a prompting one.
Fabrication has a specific meaning worth holding. A model asserting something absent from its grounding is different from a model reasoning incorrectly over material it was given, and the fixes differ. The first wants grounding and citation; the second wants a reasoning technique or a more capable model.
From the free practice form
A RAG-based benefits assistant tells a user their annual physiotherapy limit is $5,000 when the policy says $500. Trace logs show the retriever returned the correct policy chunk — containing the $500 figure — in position 1 of the context, and the model even cited that chunk in its answer. The team's first instinct is to rebuild the vector index with better embeddings. Why is that the wrong fix, and what is the actual failure?
Second question, same objective
After a pipeline is migrated to a newer model version, a JSON extraction service's parse-error rate jumps from 2% to 14% overnight. Inspecting failures shows the outputs are factually correct but now often wrapped in markdown code fences or prefixed with 'Here is the extracted data:', which the strict parser rejects. What is the most accurate diagnosis and first-line fix?
Optimize token usage, latency, and cost-performance trade-offs
Cost is tokens, so start by splitting prompt tokens from completion tokens and trending them separately. Prompt tokens rising against flat traffic points at grounding, meaning more chunks retrieved or larger ones. Completion tokens rising points at generation length.
The levers are known and each costs something. Caching a stable prefix cuts cost and time to first token, and requires ordering the prompt so the front is invariant. A smaller model cuts unit price and some accuracy. Fewer retrieved chunks cut input size and some recall. Shorter reasoning cuts latency and some accuracy on hard items.
The move the exam rewards is fixing the quantity before the unit price. Sending five agents the entire conversation and then choosing a cheaper model optimizes the wrong term.
From the free practice form
An agentic document-processing pipeline costs $0.42 per session. Token accounting shows each session averages 20 model calls; every call re-sends an identical 12,000-token prefix (system prompt plus tool definitions), and 6 of the 20 calls are trivial yes/no routing classifications currently sent to the same frontier model that handles complex synthesis steps. Quality on the synthesis steps must not regress. Which TWO changes best reduce cost while protecting quality?
Second question, same objective
A customer-facing assistant has 8.2s p95 end-to-end latency. Stage timings: retrieval 300ms, request queuing 150ms, time-to-first-token 900ms, and the remainder is generation of answers averaging 1,200 output tokens, most of which is boilerplate restating the user's question and policy disclaimers before a short actual answer. The team proposes migrating to a faster model tier, projecting a 25% throughput gain at some accuracy risk. What should they do instead?
Monitor system performance using logging and observability tools
Production monitoring answers a question offline evals cannot: is quality drifting now, on real traffic.
The signal designs forget is downstream acceptance. Whether the next system took the output, or a human overrode it, is the cheapest quality measure available and it needs no judge. Rising override rates say quality is falling before any metric you defined does.
Latency and error rate are necessary and not sufficient, because an LLM system fails by returning something plausible rather than by crashing. A scenario where everything looks healthy and users are unhappy is testing exactly that gap.
From the free practice form
A production assistant handling 40,000 requests/day currently logs only the final response text and an HTTP status code. Users report that answers 'sometimes get worse in the afternoon,' but engineers cannot reproduce the issue or tell whether it stems from retrieval, prompt assembly, model behavior, or load. Which observability change most directly enables diagnosing this class of intermittent quality issue?