LLM-as-a-Judge Drift: How to Tell It From a Real Regression

Your LLM-as-a-judge scores dropped, and you need to know whether the product got worse or the judge changed. The fastest answer comes from two re-runs that do not involve the product at all:

1. Re-score the saved outputs from your last good run with today's judge setup. Those outputs did not change, so if their scores moved, the measurement moved.
2. Re-score a fixed set of human-labeled outputs (an anchor set) and compare the judge with the human labels. If agreement with humans shifted, the judge drifted. If anchors hold steady while production scores fall, look at the product.

Until one of those checks points somewhere, treat the score change as unattributed and keep it out of the release decision. The rest of this guide covers what can move a judge score, what model version pinning actually guarantees on the Claude and OpenAI APIs, a side-by-side procedure for switching judge models, and how to calibrate a judge against human labels with agreement numbers you can recompute.

Five things that can move an LLM-as-a-judge score

An end-to-end eval samples twice: once when the product generates a response, and again when the judge grades it. A single average hides which stage moved.

Every row after the first can move scores while the product stays exactly the same. That is why the retained record for each grade matters: case ID, dataset version, product version, a stored copy of the candidate output, generation settings, judge model ID, the request parameters actually sent, rubric and parser versions, timestamp, and any request ID the provider returns. Without the stored output, you cannot re-score anything after the fact. If prompts or outputs are sensitive, encrypt or redact them with short retention, but keep a replayable copy for the anchor set and the last known good run.

How to tell judge drift (evaluator drift) from a product regression

Work through these in order. Each step is cheap, and each one rules out a cause before you touch the product.

1. Diff the metadata. Compare dataset, product, judge model ID, request parameters, rubric, few-shot examples, parser and aggregation versions between the two runs. A changed judge ID or rubric version doesn't prove drift, but it tells you which comparison to run next.
2. Re-score frozen outputs. Take the stored outputs from the last good run and grade them with the current judge setup. If the scores on unchanged outputs move by about as much as production did, the judge side changed. If they reproduce the old scores, the difference came from new outputs.
3. Re-score the anchor set. Anchors are outputs with human labels that never get regenerated. Only the judge can change how they score. Measure judge-versus-human agreement on them the same way every time (see the kappa example below).
4. Read production and anchors together.

The third row deserves attention. A practitioner write-up on judge upgrades and phantom regressions argues that "phantom wins" after a judge change are worse than phantom regressions, because nobody questions good news. Apply the same suspicion to scores that rise right after the judge setup changed.

One condition makes this whole method work: anchor outputs must stay frozen. If you regenerate them with the current product, a product change can move them too, and they no longer separate the two causes.

The anchor idea has been formalized. The June 2026 arXiv preprint Who Drifted: the System or the Judge? re-scores a fixed human-labeled anchor set at a steady interleave and returns a verdict of none, system or judge. On two real judge changes in the author's setup, a silent version bump was flagged as judge drift in 60 of 60 runs with no misattribution to the product, and a stricter scoring prompt was attributed correctly in 110 of 120 runs. A rolling z-test alarm false-alarmed on 75% of streams with no drift at all. Those numbers come from a single-author, not-yet-reviewed paper and its specific experiments. The design principle carries over; the rates don't.

What version pinning guarantees, and why temperature 0 is not the fix

Pinning the judge model removes one cause of drift. It doesn't freeze the judge's behavior, and the classic "set temperature to 0" advice no longer applies to several current frontier judges.

Claude API. Anthropic's model IDs and versions page says each Claude model ID identifies a pinned version whose underlying model stays constant for the life of that ID. From the 4.6 generation on, dateless IDs such as claude-opus-5-5 are pinned snapshots, not pointers to the newest model; older aliases like claude-sonnet-4-5 point to the latest dated snapshot. The same page draws a line between weights and serving infrastructure. The request router, safety classifiers and sampling logic can change, and "occasionally" produce minor behavior differences on an unchanged ID. Anthropic names an infrastructure update as the most likely cause when a previously stable ID starts behaving differently. So a pinned Claude judge still needs an anchor set.

OpenAI API. OpenAI's model pages say snapshots "let you lock in a specific version of the model so that performance and behavior remain consistent." Moving aliases are a different matter: the chat-latest model page says its underlying snapshot "will be regularly updated," and OpenAI points production API users to other models instead. Don't use a moving alias as a judge ID. OpenAI Text Models in 2026 explains how the gpt-5.6 alias routes to Sol and when to re-evaluate after an alias change; the same alias-versus-dated-ID question comes up on xAI in Grok Text Models in 2026.

Sampling parameters. Setting temperature, top_p or top_k to any non-default value on Claude Opus 4.7 and later, including Opus 5.5, returns a 400 error, according to the Opus 5.5 migration guide. The guide adds that temperature 0 never guaranteed identical outputs on earlier models either. OpenAI's GPT-6 model guidance says to remove temperature, top_p and top_logprobs whenever reasoning effort is not none, and GPT-6 Astra doesn't support none. If your judge config still sends temperature: 0, moving to one of these models will either fail or require dropping the parameter, and that change itself can shift scores. Claude Opus 5.5: API Price, Model ID, and Migration Checks lists the other request changes to make before switching a judge to that model.

Seeds. OpenAI's older reproducible-outputs cookbook described a Chat Completions beta where matching seed, request parameters and system_fingerprint gave "mostly" identical outputs, and it stated that determinism was not guaranteed. Treat seeds and fingerprints from that era as diagnostic metadata, not as a reproducibility guarantee for current models or for the Responses API (parameter support differs between the two APIs).

What to do instead:

• Log the exact model ID and the request parameters you actually sent, including reasoning effort where the model supports it.
• Measure repeatability rather than assuming it: grade the same frozen outputs several times and report the pass/fail flip rate per criterion.
• Keep the anchor set running on a schedule, because pinning can't catch serving-side changes.

Switching judge models: run old and new side by side

Moving the judge to a new model is the most common way to break a score history. Treat it as a migration with its own acceptance test, not as a config edit.

1. Freeze the inputs. Pick one stored dataset: recent production outputs, your eval set and the human-labeled anchors. Do not regenerate anything during the comparison.
2. Adapt the request, then lock it. Remove parameters the new model rejects, set reasoning effort explicitly, and record the new configuration as its own version.
3. Run both judges on the same outputs. Keep the old pinned judge running in parallel with the new one.
4. Compare three ways. Old judge vs new judge on the full set, old judge vs humans on the anchors, and new judge vs humans on the anchors. Break every comparison down by category, language and risk tier.
5. Rebase where they agree. For categories where the new judge agrees with the old one and with humans, start a new baseline and mark the switch date on every chart.
6. Freeze where they disagree. For categories with low agreement, stop comparing scores across the switch and re-calibrate the rubric or prompt before those scores feed a gate again.
7. Never draw a trend line across the switch. A 4.2 from the old judge and a 4.4 from the new one are measurements from two different instruments.

Steps 3 to 7 follow the re-pinning procedure in the practitioner write-up linked above, and match Datadog's advice in its LLM-as-a-judge explainer to "pin judge model versions and re-validate against a held-out, human-labeled set on every judge upgrade." If you're choosing between Claude tiers for the new judge, Claude Sonnet vs Opus vs Haiku vs Fable: Which Model to Use compares them by price, speed and effort.

How to calibrate an LLM judge against human labels

Calibration means checking whether the judge makes the same decisions your experts would, on the cases you actually ship. It's not subtracting an average offset between human and judge scores; a constant shift doesn't fix a judge that misses one kind of failure.

Take out what code can check. Whether JSON parses, a required tool was called, a field matches the reference or a citation is on an allowlist are executable checks. Give them deterministic graders with specific failure messages and keep the LLM judge for criteria that need semantic judgment, such as faithfulness, whether the answer resolves the request, policy-compliant refusals or tone.

Make the judge's output easy to score. OpenAI's evaluation best practices recommend pairwise comparison or pass/fail "for more reliability," controlling for response length, and validating agreement against your human labels before optimizing for cost or latency. Anthropic's guide to building evaluations asks for detailed rubrics, a specific output such as correct/incorrect or a 1–5 scale, and reasoning that is produced first and then discarded, with reliability tested before you scale.

Build a calibration set that stresses the judge. Random production samples are not enough. Include:

• clear passes and clear failures;
• borderline cases where your policy needs a deliberate call;
• past incidents and expert corrections;
• fluent answers with a material error, and plain answers that are correct;
• pairs of similar quality that differ only in order, length, formatting or confidence;
• the languages, tasks, risk levels and response lengths you actually serve.

Label independently first. Have at least two experts label the same items without seeing each other's answers, then reconcile. Their disagreement is information: it may show a vague rubric, two defensible policies, or a case type that should never be auto-gated. Humans disagree too, so human labels are a reference you've agreed on, not ground truth.

Know what "20 examples" means. LangSmith's Align Evaluator suggests starting with at least 20 labeled examples, balanced between 0 and 1, and reports an alignment score that is the percentage of examples where the judge matches the human. That's a good start for prompt iteration. For a release gate you need more labeled failures, per segment, than 20 items can hold, and you need a chance-corrected metric alongside raw agreement.

Worked example: 93% agreement, kappa 0.59

Raw agreement looks reassuring on imbalanced data because most items pass. The numbers below are an illustration, not a measurement.

Suppose 200 anchors, of which humans labeled 20 FAIL and 180 PASS. The judge's labels:

• Raw agreement: (12 + 174) / 200 = 93%.
• Chance agreement from each rater's label mix: (20/200 × 18/200) + (180/200 × 182/200) = 0.009 + 0.819 = 0.828.
• Cohen's kappa = (observed − chance) / (1 − chance) = (0.93 − 0.828) / (1 − 0.828) ≈ 0.59.
• Fail recall: 12 of 20 real failures caught, so 60%. Eight of every twenty real failures would pass the gate.
• Fail precision: 12 of 18 judge FAILs were real, about 67%.

A 93% headline can coexist with 40% of real failures getting through. Now the extreme case: 100 items, 5 human FAIL, and a judge that says PASS to everything. Raw agreement is 95%, kappa is exactly 0, and fail recall is 0.

This dependency-free Python reproduces both results:

def agreement_report(human, judge, positive="FAIL"):
    n = len(human)
    pairs = list(zip(human, judge))
    labels = set(human) | set(judge)
    p_o = sum(h == j for h, j in pairs) / n
    p_e = sum((human.count(l) / n) * (judge.count(l) / n) for l in labels)
    kappa = (p_o - p_e) / (1 - p_e) if p_e < 1 else float("nan")
    hits = sum(h == positive and j == positive for h, j in pairs)
    recall = hits / human.count(positive) if positive in human else float("nan")
    precision = hits / judge.count(positive) if positive in judge else float("nan")
    return {"raw_agreement": round(p_o, 3), "kappa": round(kappa, 3),
            "fail_recall": round(recall, 3), "fail_precision": round(precision, 3)}

# 200 anchors: 20 human FAIL, 180 human PASS
human = ["FAIL"] * 20 + ["PASS"] * 180
judge = ["FAIL"] * 12 + ["PASS"] * 8 + ["FAIL"] * 6 + ["PASS"] * 174
print(agreement_report(human, judge))
# {'raw_agreement': 0.93, 'kappa': 0.593, 'fail_recall': 0.6, 'fail_precision': 0.667}

# 100 items, 5 human FAIL, judge passes everything
print(agreement_report(["FAIL"] * 5 + ["PASS"] * 95, ["PASS"] * 100))
# {'raw_agreement': 0.95, 'kappa': 0.0, 'fail_recall': 0.0, 'fail_precision': nan}

If you use scikit-learn, cohen_kappa_score gives the same kappa and accepts weights="linear" or "quadratic" for ordinal scales like 1–5, where a 4-vs-5 disagreement should count less than 1-vs-5.

What to report every time you calibrate or re-check:

• raw agreement and a chance-corrected metric (Cohen's kappa for two raters, weighted kappa for ordered labels);
• label counts per class, and the full confusion matrix;
• recall and precision for the class you care about, usually FAIL;
• the same numbers per segment, because a good average can hide a bad language or task;
• the judge model ID, prompt version, request parameters and date.

Two traps are worth knowing. Kappa can come out surprisingly low under heavy class imbalance even when raw agreement is high, a known effect sometimes called the prevalence paradox; the Arize guide to measuring human–LLM judge alignment covers it and suggests other metrics for other annotation setups. And if you add extra failures to your anchor set to get enough of them, precision on that set won't match production precision. Keep a natural-distribution set for precision and a failure-heavy set for recall.

What Cohen's kappa is acceptable or good enough for an LLM judge?

There is no universal cutoff. You'll find claims that 0.80 is "the standard," but Arize's guide explicitly advises against universal thresholds such as "0.6 is good" or "0.8 is acceptable," and the OpenAI, Anthropic and LangSmith docs cited above give no number at all. Set your own bar from four inputs:

• Human–human agreement on the same items. Measure it pairwise, with the same metric, on the same examples, and compare each human–judge pair against it. Don't add the judge to the human pool and report one combined score; that answers a different question. Human agreement is context, not a ceiling: a judge can match a reconciled reference more often than individual annotators match each other.
• Error costs. A missed unsupported medical claim and an unnecessary review of a marketing headline need different policies. When a missed failure is expensive, fail recall matters more than overall kappa.
• Counts per segment. A kappa computed from four failures in one language says almost nothing. Report the counts next to the metric and don't gate on segments that have too few labeled failures.
• What the score decides. A hard block on release needs a higher bar than a trend chart or a triage queue that humans review anyway.

For context on what strong judges have achieved elsewhere: the MT-Bench and Chatbot Arena study reported GPT-4-era judges reaching over 80% agreement with human preferences, about the same level humans reached with each other on that data. That's raw pairwise-preference agreement for those models and datasets. It says nothing about your rubric or your judge.

Probe known judge biases before you trust the scores

Calibration numbers can look fine while the judge still rewards the wrong things. A few targeted probes catch the common shortcuts.

• Swap the order. In a 2023 study, changing the order of candidates let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator. A later study of 15 judges and over 150,000 evaluation instances found position bias is not random, varies by judge and task, and grows when the two answers are close in quality. For pairwise grading, run A/B and B/A and only declare a winner when both orders agree; treat a split as a tie or send it to review.
• Change presentation without changing substance. Make variants that differ only in length, headings, tone or formatting. If scores follow those changes, the judge is measuring its own taste. OpenAI's guidance names verbosity bias and recommends controlling for length.
• Give it the reference when one exists. For extraction, grounded Q&A, summaries, math or tool use, include the source or expected result so the judge checks facts instead of confidence.
• Don't treat the explanation as proof. A judge can write a convincing rationale for a wrong verdict. Use explanations for debugging, not as the reason to accept a grade.

Add every new production failure and every probe that caught the judge to the calibration set, so the next judge change is tested against them.

The release gate: pass, block, investigate, or human review

Once the checks above exist, a gate can ask its questions in a fixed order:

if deterministic_check_failed:
    BLOCK                     # return the specific failure
elif anchor_agreement_outside_calibrated_range:
    INVESTIGATE_JUDGE         # model, rubric, examples, parser, serving stack
elif frozen_output_repeatability_failed:
    INVESTIGATE_MEASUREMENT   # judge too unstable on unchanged outputs
elif difference_clearly_crosses_policy_boundary:
    PASS_OR_BLOCK             # per the calibrated policy
else:
    HUMAN_REVIEW              # review band, high-risk case, or unclear result

"Calibrated range" and "clearly crosses" are thresholds you set from the calibration data, error costs and observed repeatability, not defaults.

Record the judge setup as a versioned artifact, stored next to every result. An example with illustrative values:

{
  "judge_model": "claude-opus-5-5",
  "request_params_sent": {"effort": "medium"},
  "rubric_version": "faithfulness-v5",
  "examples_version": "anchors-v3",
  "parser_version": "grade-schema-v2",
  "aggregation_version": "pass-fail-review-v3",
  "dataset_version": "support-summary-2026-09",
  "calibrated_on": "2026-09-25"
}

The request parameters field records what was actually sent. On Opus 5.5 that means no temperature, and effort is the request parameter that controls thinking depth; on other judges it's whatever that model accepts. Store per-call request IDs with the grades rather than in this file.

When to stop trusting the metric. Take a criterion out of automatic gating, and send it to human review, when any of these hold:

• anchor agreement moved and you can't tie it to a known change;
• clear passes or clear failures flip when you re-grade the same frozen output;
• the judge was switched and that category hasn't been re-calibrated;
• human experts can't agree with each other on the criterion.

A gate that can say "not known yet" is more useful than a precise-looking score with no history behind it.

FAQ

Does pinning the judge model stop drift?

It removes model-weight changes, not all drift. On the Claude API, a model ID always maps to the same weights, but Anthropic says serving infrastructure changes can occasionally change behavior on an unchanged ID. On OpenAI, snapshots lock the version, while moving aliases such as chat-latest are updated regularly. Pin, and keep re-scoring an anchor set.

Should I set temperature to 0 for my judge?

Only if the model accepts it, and don't count on it for determinism. Claude Opus 4.7 and later reject non-default temperature, and GPT-6 models need it removed whenever reasoning effort isn't none. Record the parameters you send and measure the flip rate on frozen outputs instead.

Is 90% agreement with human labels good enough?

Not by itself. With rare failures, a judge can reach 90% or more while missing many of them; in the worked example above, 93% agreement caught only 60% of failures. Look at kappa, fail recall and the confusion matrix, and compare against human–human agreement on the same items.

How many human-labeled examples do I need?

No fixed number fits every case. Twenty balanced examples are a reasonable start for tuning a judge prompt. A release gate needs enough labeled failures in each segment you gate on for recall and kappa to mean something. Twenty balanced examples hold only about ten failures in total, too few to split by language or task.

Can I compare scores from before and after a judge upgrade?

Only for categories where the old and new judges agree with each other and with humans on the same frozen outputs. Rebase those with a marked discontinuity; freeze the rest until they're re-calibrated.