Calibrating LLM Judges to Business Value

Rigorous on experiments. Casual on offline evals.

Most organizations that deploy LLMs have two evaluation cultures living side by side.

On the online experiment side, there's real discipline. Teams define estimands (e.g., an average treatment effect). They compute sample sizes. They report p-values and confidence intervals. If the result comes back p = 0.08, they don't ship — they collect more data or move on.

On the offline eval side, the same teams will judge 1,000 responses with an LLM, hand-label 20 examples to check agreement, use the same 20 examples they used to tune the judge prompt, report “80% agreement,” and call it ready to ship. No holdout. No confidence intervals. No calibration to what they actually care about.

This isn't because anyone is careless. It's because the tooling and methodology for rigorous offline evaluation hasn't caught up with the tooling for online experimentation. A/B testing has decades of statistical infrastructure behind it. LLM-as-a-judge evaluation is a few years old and still running on vibes.

The cost of that gap is real. Teams ship based on judge scores that look precise but measure the wrong thing. They're being rigorous about the wrong quantity — carefully quantifying uncertainty about the judge's opinion when the real question is whether the judge's opinion tracks the business outcome at all.

Even when teams do quantify uncertainty in AI evals, they're often precise about the wrong thing. A common pattern: report “80% agreement with human labels” on the same examples used to refine the judge rubric. Did you improve the judge, or did you overfit 20 examples? You've also used up the very labels you'd need to estimate calibration error.

The real question is not “how precisely did we estimate the mean judge score?” It is “how precisely did we estimate the business outcome we actually care about?” Those are different questions with very different answers.

Your judge score is not your business outcome. It's a signal that's correlated with your business outcome, measured on a different scale, with its own systematic biases. The gap between those two things is where expensive mistakes hide. And most eval pipelines have no way to measure that gap, let alone close it.

That's the problem Causal Judge Evaluation (CJE) solves.

The gap you can't prompt-engineer away

The instinct when judge scores seem off is to fix the judge: rewrite the rubric, add few-shot examples, try a stronger model. And sometimes that helps. But it doesn't address the structural issue.

Judge scores are surrogates — cheap, fast proxies for an outcome you can't afford to measure on every sample. The outcome you actually care about might be user satisfaction, clinical accuracy, task completion, revenue per session, or expert preference. Call that your business outcome. Your LLM judge is not measuring that. It's measuring something that correlates with it.

That correlation is not as stable as you think. Three ways it breaks:

Preference inversion. A judge that rewards verbosity will rank a long, mediocre answer above a short, excellent one. A judge that rewards confident tone will score sycophantic responses highly — even when users find them unhelpful. The judge rewards eloquence. The business needs helpfulness. Those can diverge silently, with the judge score never moving while the outcome you care about deteriorates. This is Goodhart's Law applied to AI evaluation.

False precision. Your judge says Policy A scores 4.2 and Policy B scores 4.0. Is that a real difference or noise? Without calibrating to a ground truth scale, you can't construct valid confidence intervals. In practice, uncalibrated judge scores produce confidence intervals with near-zero coverage: your “95% CI” almost never contains the true business outcome.

Scale arbitrariness. Switch your judge from GPT-4o to Claude and your numbers change — not because your product got better or worse, but because the ruler changed. Without anchoring to a business outcome, you can't compare across judges, across time, or across evaluation setups.

The bias enumeration literature keeps growing — verbosity bias, self-preference bias, position bias, authority bias, style bias — with each paper identifying a specific distortion and proposing a specific mitigation. But here's the question nobody seems to ask: why would you expect these to be the only biases? LLM judges are complex functions of their inputs, sensitive to phrasing, formatting, tone, length, vocabulary, and countless other features that correlate with but don't determine quality. Every one of those sensitivities is a potential bias. Most of them don't have names. Most of them never will.

You can't patch what you can't enumerate. And you will never finish enumerating.

Calibration sidesteps the entire problem. Learn the mapping from what the judge says to what's actually true, absorbing every distortion — named and unnamed — in a single step. One mapping. Every bias corrected at once. The approach scales with the cost of oracle labels (finite and plannable), not with the number of biases (infinite).

You don't need to measure your business outcome on every sample. You need to measure it on a small slice — typically 5 to 25% — and learn the relationship. Then apply it everywhere.

The CJE workflow

CJE is a method and an open-source library (pip install cje-eval) that bridges judge scores to business outcomes. Four steps.

The confidence intervals deserve emphasis. Most eval frameworks don't decompose your uncertainty into its two sources. They tell you your judge “agrees with humans 78% of the time” and leave you to guess how many oracle labels you need. That's like managing a budget without knowing your line items. Under the hood, CJE uses cross-fitted calibration and a debiased estimator that corrects for error in the learned mapping. The statistical backbone — label a slice, learn the mapping, debias the rest with valid intervals — is prediction-powered inference (Angelopoulos et al., 2023) and, before that, surrogate-index estimation (Athey, Chetty, Imbens & Kang); CJE packages that machinery for judge calibration and adds transport audits on top. The result: confidence intervals that achieve proper 95% coverage, not the near-zero coverage you get from treating judge scores as ground truth.

Naive intervals account only for evaluation variance. Calibration-aware intervals account for both sources — wider, but honest.

Here's the complete workflow in code:

from cje import analyze_dataset

results = analyze_dataset(
    fresh_draws_data={
        "prompt_v1": [
            {"prompt_id": "1", "judge_score": 0.85, "oracle_label": 0.9},
            {"prompt_id": "2", "judge_score": 0.72, "oracle_label": 0.7},
            {"prompt_id": "3", "judge_score": 0.68},  # no oracle label needed
            {"prompt_id": "4", "judge_score": 0.79},
        ],
        "prompt_v2": [
            {"prompt_id": "1", "judge_score": 0.78, "oracle_label": 0.82},
            {"prompt_id": "2", "judge_score": 0.81, "oracle_label": 0.79},
            {"prompt_id": "3", "judge_score": 0.75},
            {"prompt_id": "4", "judge_score": 0.83},
        ],
    }
)

# Calibrated estimates with honest confidence intervals
for i, policy in enumerate(results.metadata['target_policies']):
    lo, hi = results.ci()[i]
    print(f"{policy}: {results.estimates[i]:.3f} [{lo:.3f}, {hi:.3f}]")

# Statistical comparison with proper variance accounting
comp = results.compare_policies(0, 1)
print(f"Difference: {comp['difference']:.3f}, p={comp['p_value']:.3f}")

One function call handles calibration fitting, mode selection, and inference. Most samples don't have oracle_label — that's the point. You label a small slice, CJE learns the mapping, and applies it to the rest.

The CJE loop: score broadly, calibrate on a small oracle slice, estimate with uncertainty, monitor for drift.

What this looks like in practice

To make this concrete, consider a real validation on medical AI evaluation — a domain where the gap between “judge opinion” and “business outcome” has direct patient safety implications.

HealthBench (OpenAI, 2025) is a benchmark of roughly 5,000 health conversations with grading rubrics attached. We used its meta-evaluation subset: 29,511 criterion-level judgments where physicians assessed whether a specific rubric criterion was met — those physician consensus labels are the ground-truth business outcome here. At typical physician-annotation rates, labeling at that scale plausibly runs to six figures.

We ran two popular LLM judges on the full dataset and compared their scores to the physician labels.

The problem went deeper than overall bias. On specific clinical criteria — medication safety, diagnostic accuracy, treatment appropriateness — the two judges disagreed with each other by up to 73 percentage points. They weren't just wrong in the same direction; they were wrong in different directions on different slices. And you would have had no way to know, because both judges looked internally consistent.

Then we applied CJE with just 5% of the oracle labels — roughly 1,400 physician assessments out of 29,511. (Full methodology, judge configurations, and the reproduction repo live in the HealthBench judge audit.)

After calibration, both judges' aggregate estimates converged to the physician ground truth. The 24.5-point overconfidence disappeared, and the two judges' aggregates — sharply different on raw scores — converged to overlapping calibrated uncertainty bands (0.692 vs 0.685). One caveat from the underlying study: the 73-point disagreement lives in specific category-level criteria (hedging and seeking-context), and calibration aligned the judges' aggregates, not that category-level divergence. (The CI-coverage result — nominal 95% intervals going from 0% to ~95% actual coverage — comes from the Arena benchmark below, whose multi-seed design can measure coverage; the HealthBench study is a single run.)

The cost trade-off: labeling 5% instead of 100% — a ~20x labeling-cost reduction plus pennies for LLM judge scoring on the rest — with the calibrated aggregate landing within 0.6–2.1 points of full-label ground truth in this single-seed study. (Policy-ranking accuracy is the Arena result below; the HealthBench study ranked no policies.) And crucially, the calibrated pipeline gives you something the raw judges never could: valid confidence intervals that tell you how much to trust each estimate. If the interval is wide, you know you need more data. If it's tight, you can make decisions with confidence.

A separate validation on 5,000 evaluations drawn from Chatbot Arena — the largest public source of human LLM preference votes — showed 94% pairwise ranking accuracy on average (up to 99% in the best configuration), using just 5% oracle labels. For comparison, importance-weighted approaches (the standard tool for off-policy evaluation) achieved near-random accuracy on the same data — the action space of LLM outputs is so large that policies have near-zero overlap, making reweighting useless in practice. The full methodology is published on arXiv (2512.11150).

Knowing when calibration breaks

Calibration learned on one distribution can break silently when the distribution shifts — new prompt types, different users, model updates. This is where most ad hoc calibration fails. You get a calibrated number but no way to know when it stops being trustworthy.

CJE includes a built-in safety net: transportability auditing. Periodically label a small batch of new samples with your oracle (40–60 is often enough) and ask: does the calibration I learned still hold?

from cje.diagnostics import audit_transportability

# new_probe_data: fresh held-out rows with judge_score and oracle_label
# (never used to fit the calibrator)
# e.g. [{"judge_score": 0.82, "oracle_label": 0.79}, ...]
audit = audit_transportability(
    results.calibrator,
    new_probe_data,
    delta_max=0.05,  # predeclared drift margin, in business-outcome units
)
print(audit.summary())
# Residual transport: PASS | N=60 (60 clusters) | delta: +0.008 (CI: [-0.023, +0.039]) | margin: +/-0.050

The audit doesn't test whether drift is exactly zero — on enough data, it never is. Instead, you declare how much calibration drift your decision can tolerate: a margin (delta_max) in the units of your business outcome, like ±0.05 on a 0–1 satisfaction scale. The audit then grades a confidence interval for the mean gap between calibrated predictions and oracle labels against that margin. CI wholly inside the margin — PASS. Wholly outside — FAIL, recalibrate before making decisions. Straddling the boundary — INCONCLUSIVE, the probe can't tell yet. Skip the margin entirely and the audit is NOT_GRADED: descriptive numbers that can never pass or fail. The library warns when you do this, because a drift monitor with no declared margin silently never fails — the dangerous direction.

Transport audit over time. Weekly probes catch calibration drift before it reaches stakeholders.

• PASS: Drift is established inside your margin. Keep using your calibration; fold the new probe labels into your history. (A pass also requires enough independent probe prompts — roughly 20 — so a tiny probe can't certify one.)
• INCONCLUSIVE: The probe can't place the drift inside or outside the margin. Increase probe size next cycle and inspect which score ranges are drifting.
• FAIL: Drift exceeds your margin — calibration no longer transfers. Collect 100–200 fresh oracle labels and refit. A FAIL counts even from a small probe: an interval wholly beyond the margin is decisive evidence of bias, not low power.

Think of it as a unit test for your evaluation pipeline. You run it weekly, and it tells you whether your numbers are still trustworthy — before a stakeholder has to ask.

When CJE won't help

No oracle labels at all. You need at least some ground truth to learn from. If you genuinely cannot measure your business outcome on any samples — no human ratings, no downstream metrics, no stronger reference model — then there's nothing to calibrate to. Start by defining what “good” means and measuring it on 50–100 samples.

The outcome keeps changing. If what “good” means shifts faster than you can collect labels — user preferences evolving week to week, evaluation criteria under active debate — then calibration chases a moving target. The transport audit will catch this (repeated FAIL verdicts), but the fix is stabilizing your evaluation construct, not recalibrating more often.

The judge's ordering is broken. Isotonic regression assumes higher scores generally mean better outcomes. If the judge has a fundamentally broken ordering — systematically preferring bad outputs — calibration can't rescue it. Fix the judge first.

Unrepresentative oracle sample. If your oracle labels come from a narrow slice — only easy prompts, only one domain, only one user segment — the calibration may not generalize to the full distribution. The transport audit will flag this, but prevention is better: sample thoughtfully across the full range of your data.

Bad eval design. CJE calibrates a judge to a business outcome. If neither your judge nor your oracle actually measures what matters for product decisions, calibration won't help. Garbage in, calibrated garbage out. The first step is always defining the right outcome.

What to do Monday morning

If you're making shipping decisions based on LLM judge scores, here's a concrete protocol:

The cje-eval library is open source on PyPI. The full methodology is on arXiv (2512.11150). If you'd like to pilot CJE with your evaluation team, reach out — we're working with teams on adoption and welcome the conversation.