An evaluation dashboard drops from 82% to 76%. The product team starts a rollback. Then someone notices that the application commit never changed—the grader model alias did.
That is the central failure mode of an LLM-as-a-judge pipeline: its output looks like a measurement, but the measuring instrument is another probabilistic system. A score change may come from a different candidate response, a different decision by the same judge, a new rubric, a silent backend change, or an actual product regression. A single aggregate cannot identify which one happened.
The useful goal is not perfect determinism. It is attribution: enough evidence to say “system,” “judge,” “ordinary variation,” or “we cannot safely tell yet.”
Run the experiment that matches the question
An end-to-end eval contains two sampling stages:
texttest case -> product model -> saved candidate -> judge model -> decision
If you regenerate the candidate and re-grade it, both stages can move. If you want to measure judge repeatability, freeze the candidate and call only the judge again.
Use four distinct experiments:
| Question | Keep fixed | Vary | Evidence you get |
|---|---|---|---|
| Does the judge repeat itself? | Test case, candidate output, rubric, judge configuration | Judge invocation | Score spread and pass/fail flips |
| Is the product experience stable? | Dataset and versioned contracts | Candidate generation and grading | End-to-end outcome distribution |
| Is candidate B better than A? | Cases and calibrated judge contract | Product version | Paired difference with uncertainty |
| Did the judge drift? | Human-labelled anchor outputs | Current judge over time | Change in judge-to-human disagreement |
This separation requires retaining candidate outputs or an approved replayable representation. A chart with only daily averages cannot reconstruct judge-only variance after the fact. Each grade should be traceable to a case ID, dataset version, product version, candidate output ID, generation configuration, judge model, rubric version, parser version, and timestamp.
Sensitive prompts and outputs may require encryption, redaction, short retention, or a controlled replay service. The answer is a data-governance design—not silently dropping the fields needed for attribution.
Remove every judgment you can replace with a check
Do not ask a model whether JSON parsed, a required tool was called, a field equals the reference, or a citation belongs to an allowlist. Those are executable contracts. Give them deterministic graders and fail with a specific reason.
Reserve the LLM judge for criteria that genuinely require semantic judgment: whether a summary is faithful, an answer resolves the user’s need, a refusal follows policy without becoming useless, or a tone fits its audience.
OpenAI’s current evaluation best-practices guide recommends task-specific evals, continuous evaluation, human calibration, and more constrained decisions such as pairwise comparison or pass/fail. Its graders guide likewise separates string checks, similarity measures, and model graders. The product APIs are provider-specific, but the design principle is portable: use the least ambiguous measurement that can answer the question.
A release decision becomes easier to debug when it retains separate outputs:
- deterministic contract failures;
- semantic criteria, each with its own decision;
- judge instability or calibration failure;
- cases that require an expert.
Collapsing everything into 7.4/10 hides whether you have a factual defect, a style preference, or an unreliable instrument.
Build a calibration set for the judge itself
A judge should not enter CI until it has passed an eval of its own. The dataset needs more than random production examples:
- obvious passes and obvious failures;
- borderline cases where policy needs a deliberate decision;
- real incidents and expert corrections;
- pairs with similar quality but different order, length, formatting, identity, or confidence;
- fluent answers with a material error and plain answers that are correct;
- adversarially verbose or authoritative-looking responses;
- the languages, tasks, risk levels, and response lengths you actually ship.
Ask domain experts to label independently before reconciling. Their disagreements matter. They may reveal a vague rubric, two legitimate policies, or a case that should never be auto-gated.
Research supports both the value and the limits of model judges. The original MT-Bench and Chatbot Arena study reported strong judge–human agreement in its setup, while documenting position, verbosity, self-enhancement, and reasoning limitations. The agreement number belongs to those models, prompts, and datasets; it is not a transferable accuracy claim.
Large Language Models Are Not Fair Evaluators demonstrated that changing candidate order could change rankings and tested balanced position, multiple evidence, and human-in-the-loop calibration. A later systematic position-bias study covered 12 judges, 22 tasks, and more than 100,000 evaluation instances, finding that the effect varied by judge and task. Model capability alone does not certify your rubric.
Measure decisions, not just average score
Calibration is not “humans average 0.72, the judge averages 0.79, subtract 0.07.” A constant offset does not fix wrong ordering, asymmetric errors, or failures concentrated in one segment.
Evaluate the decisions your workflow will make:
- On clear passes, how often does the judge block?
- On material failures, how often does it pass?
- Where do expert and judge decisions diverge?
- Does error change by task, language, output length, or risk tier?
- How wide is run-to-run variation near the gate?
- Which cases are unstable for both humans and the judge?
False accepts and false rejects rarely cost the same. A missed unsupported medical claim and an unnecessary review of a marketing headline need different policies. Define those costs before choosing a threshold.
For a release gate, three outcomes are often more honest than a fine-grained score:
- pass, when calibrated evidence clears the decision boundary;
- fail, when a deterministic contract or calibrated semantic criterion is clearly violated;
- review, when the result sits in an uncertainty region or affects a high-risk case.
Continuous scores can remain useful for analysis. They should not create precision that the calibration data cannot support.
Probe the judge for known shortcuts
Repeat the frozen sample
Run the same saved candidate through the same judge contract multiple times. Store every decision. Report flip rate and per-criterion spread, not only the mean.
If obvious cases flip, the judge is not ready for a binary gate. If variation concentrates on genuinely ambiguous cases, create a review region instead of prompt-tuning until the ambiguity disappears from the dashboard.
Swap A and B
For pairwise grading, evaluate both A/B and B/A. A conservative policy declares a winner only when both orders agree; conflicting results become a tie or review. At scale, randomized order can reduce directional bias, but you should still report order consistency separately.
Change surface form without changing task quality
Create controlled variants that alter length, headings, assistant names, tone, or formatting while preserving the answer’s substance. A score that follows those surfaces may be measuring a judge preference, not the product criterion.
This test also catches grader hacking. A system can learn to produce longer, more ceremonious answers that score higher while experts see no improvement. Keep new production failures and counterexamples in the judge’s own eval set.
Supply references when truth is available
For extraction, grounded Q&A, summary faithfulness, math, or tool use, give the grader the source, reference, or executable result. A reference may be imperfect, but it focuses the decision on evidence instead of the candidate’s confidence.
A judge explanation is useful for diagnosis, not proof. The same model can provide a convincing rationale for an incorrect verdict.
Keep an anchor stream beside the production stream
Once the judge is calibrated, preserve a stable core of human-labelled outputs. Re-score those anchors with the current judge at a steady cadence.
textproduction stream: current product outputs -> current judge -> product signal anchor stream: fixed human labels -> current judge -> judge signal
The two streams make score movement interpretable:
| Production signal | Anchor signal | First hypothesis to investigate |
|---|---|---|
| Down | Stable | Product behavior or production traffic changed |
| Down | Judge became stricter | Judge model, rubric, examples, parser, or backend changed |
| Stable | Drifted | The dashboard may be masking a product change |
| Both noisy | No attribution yet | Measurement system is not healthy enough to gate |
The 2026 preprint Who Drifted: the System or the Judge? formalizes this pattern with a fixed human-labelled anchor set and separate monitors for the judge–human gap and the main stream. Its detection rates and cost results are experiment-specific, not deployment guarantees. The transferable insight is identification: product changes cannot alter the fixed anchor outputs, while judge changes can alter how those anchors are scored.
Do not rewrite the anchor core whenever a release changes. Add new cases under a new dataset version, preserve continuity, and record why any human label or rubric changed.
Make the CI gate route evidence
A reliable gate asks questions in order:
textif deterministic_contract_failed: BLOCK elif judge_anchor_outside_calibrated_range: INVESTIGATE_JUDGE elif frozen_output_repeatability_failed: INVESTIGATE_MEASUREMENT elif candidate_difference_clearly_crosses_policy_boundary: PASS_OR_BLOCK else: HUMAN_REVIEW
The real implementation should use the uncertainty method and boundaries validated for your task. There is no universal repeat count, sample size, or 0.80 cutoff. Choose them from the cost of errors, observed variance, segmentation needs, and available human-review capacity.
Record the entire judge contract as a versioned artifact:
json{ "dataset_version": "support-summary-2026-08", "judge_model": "pinned-model-id", "rubric_version": "faithfulness-v4", "examples_version": "anchors-v3", "parser_version": "grade-schema-v2", "sampling": {"temperature": 0, "seed": 123}, "decision_policy": "pass-fail-review-v2" }
The seed is diagnostic metadata, not a guarantee. OpenAI’s historical reproducible-outputs example says matching seed, request parameters, and backend fingerprint made supported preview-model calls mostly consistent, while explicitly noting that determinism was not guaranteed. That old Chat Completions example should not be generalized to every current model, endpoint, or provider.
Pin model versions where the provider supports it. Log exposed backend fingerprints and request IDs. Version the rubric, few-shot examples, output schema, parser, and aggregation code. Any of them can change the measuring instrument.
Triage a score incident without guessing
When a metric moves:
- Confirm the dataset, product, judge, rubric, parser, and aggregation versions.
- Compare deterministic contract results before semantic scores.
- Re-grade saved candidates to measure judge-only behavior.
- Inspect the human anchor stream.
- Re-run controlled order and surface-form probes if the affected criterion suggests bias.
- Segment the change by task, language, length, risk, and traffic source.
- Send disputed high-impact samples to blinded expert review.
- Change the release decision only after one hypothesis has evidence the alternatives do not explain.
“Temperature is zero” is not an incident conclusion. Nor is “the average recovered on the second run.” One hides system-level sources of variation; the other may simply select a favorable sample.
An LLM judge earns authority gradually. First make objective conditions executable. Then validate its decisions against experts, measure repeatability on frozen outputs, probe shortcuts, and keep a human anchor stream alive. When the gate can say “we do not know yet,” it becomes safer—and more useful—than a deterministic-looking number with no provenance.
