Your AI Metrics Are Lying to You

When a meme reveals broken evaluation

If you've seen users banning their AI assistant from saying "You're absolutely right!", you've seen a metric failure in the wild.

Image credit: u/ProperAd3683 on r/ClaudeAI

What likely delighted early raters ("So polite! 10/10!") quickly became a symbol of untrustworthy sycophancy. The backlash was swift and public—developers couldn't get Claude Code to stop, no matter what they tried.

This wasn't just a quirk. It was a symptom of broken evaluation.

The Deliberation Ladder

Think of evaluation as climbing a ladder toward an ideal you can't directly observe. Each rung represents a different quality of signal—from instant metrics at the bottom to careful expert judgments higher up. The top rung (Y*) is what you really care about, but it's almost always unobservable. Lower rungs are cheap to measure at scale but far from Y*; higher rungs are closer to Y* but expensive, so you can only afford a small sample. Evaluation becomes an optimization problem: how to allocate your budget across rungs to best estimate Y*?

Preference inversion for Claude Code

The behavior that maxed your surrogate scores (S) killed productivity (Y). This is preference inversion: what scores highest on your cheap metric predicts lower value on what actually matters.

Image credit: absolutelyright.lol

The revolt was swift and public. Developers documented the damage:

• Trust collapse: "It's pure reward hacking... it says it unconditionally, when it can't be known, and actually reveals deep deception."
• Lost critical feedback: Users needed the assistant to challenge bad assumptions and catch errors, but instead got unconditional agreement—exactly when they needed pushback most.
• Desperate workarounds: Developers resorted to banning the phrase in their config files, trying (and failing) to stop behavior that early raters had rewarded.

The discussion spread across communities, from the Reddit thread to a 170+ comment GitHub issue to a Hacker News discussion with hundreds of developers sharing similar frustrations.

The metric that looked good in evaluations—warm, affirming language—became the exact thing that destroyed the product's value for actual developers. High S, low Y.

Here's what the ladder looks like for three common LLM system archetypes, showing where preference inversions tend to appear.

These are illustrative examples. Deciding on S, Y, and Y* is domain-specific and requires deep domain knowledge and careful consideration of tradeoffs.

Recent evidence: The AI coding complexity trap

A 2025 METR randomized study of experienced open-source developers found they were 19% slower when using AI tools (Cursor + Claude 3.5/3.7), even though they predicted being 24% faster. This S vs Y gap persisted even after experiencing the slowdown—developers still felt 20% faster.

The long-term failure (Y*) is even more striking: developers report that AI-assisted code often becomes unmaintainable within weeks. The complexity exceeds the assistant's context window, bug fixes introduce new bugs, and in extreme cases, the entire codebase needs to be deleted and rebuilt.

Read the METR study →

How does a meme like this happen?

We don't know exactly what happened with Claude and the "You're absolutely right!" issue. These are genuinely hard problems, the technology is moving fast, and we have great respect for Anthropic's team. But there are three common classes of evaluation failure that can produce this kind of breakdown:

Class 1: Wrong or no calibration—never measured the right outcome (Y)

During RLHF training, raters gave high scores (S) to warm, affirming responses. The team shipped based on those scores, assuming they meant users would be productive. But they never actually measured whether users who got affirmation accomplished more—caught bugs, made better decisions, avoided bad advice. Alternatively, they may have measured some Y (like "user says they're happy"), but that Y wasn't actually correlated with Y* (long-term developer productivity). Either way, the S→Y* chain was broken from day one.

Class 2: Wrong population—measured Y on a sample that didn't represent the true distribution of users / tasks

The team did calibrate S→Y, but on the wrong slice: examples from "therapy" use cases (emotional support, life advice) where affirmation feels productive, or simple tasks where encouragement helps beginners. But most actual users are writing code, debugging complex systems, or making high-stakes decisions where affirmation is toxic. The S→Y mapping doesn't transport to the real population or task distribution.

Class 3: Temporal decay—the relationship between S and Y changed over time

Maybe affirmation did help initially—new users felt supported and tried more tasks. But over weeks or months, as users adapted, the behavior became wallpaper, then a liability. User preferences shifted. The S→Y mapping that was valid in January broke by March, but no monitoring caught the flip.

Each failure class needs a different fix: Class 1 needs you to identify and measure the right Y and calibrate S against it. Class 2 needs representative sampling from your actual user population. Class 3 needs continuous monitoring and periodic recalibration.

The solution: Causal Judge Evaluation (CJE)

CJE is the framework that addresses all three failure classes systematically. Think of it as building a calibration curve: you measure how your cheap metric (S) actually predicts what matters (Y), then use that mapping to aim at your North Star (Y*).

How it works: The oracle slice

The key insight: you don't need expensive oracle labels on every example. Instead, collect them on a small "calibration slice" (~500 examples), learn how cheap metrics predict real outcomes on that slice, then apply that mapping at scale. You get less bias (because Y is closer to Y* than raw S) and less variance (because you leverage thousands of cheap labels instead of just 500 expensive ones).

How does CJE learn S→Y?

A great starting point: combine an LLM-judge score with response length. In theory, you can use more features if your data supports it, but in eval settings you often have small data, so judge + response length hits a sweet spot (you don't want too many predictors relative to data points).

The approach: learn a flexible model that combines your features (e.g., judge score and response length) to predict oracle outcomes, then ensure the final predictions are monotone in the original judge score and stay on the right scale. This lets you capture patterns like "long responses fool the judge" while maintaining sensible behavior.

In the CJE library: If S has all the cheap information you want and you think it maps monotonically to Y, just fit isotonic regression. If you want to learn flexible relationships between multiple cheap signals and Y (e.g., combine judge score + response length), use the two-stage approach—the cje-eval library uses splines for the first stage, then isotonic regression for the second.

Two-stage calibration: Stage 1 learns g(S, response_length), Stage 2 applies isotonic to preserve monotonicity and scale.

The fundamental challenge: You can never observe Y* (the true target—what an idealized deliberator would choose) directly. But you can collect a sample of (S, Y) pairs, learn how S predicts Y, then apply that mapping to abundant S data at scale. Since Y approximates Y*, this lets you aim abundant S at Y*.

What are S and Y? They're domain-specific and relative—human labels might be S in one context and Y in another; same with LLM-judges. Learning the S→Y mapping gives you both less bias (because Y is closer to Y*) and less variance (because you leverage the large-sample properties of S).

The loop has two phases: an inner calibration loop where you iterate until precision is good enough, and an outer monitoring loop where you watch for drift and recalibrate as needed. Let's walk through the key operational pieces.

Evidence: This approach works

In a benchmark on 5,000 Chatbot Arena prompts evaluating 5 LLM policies, direct model with two-stage calibration achieved 94% pairwise accuracy and 89% top-1 accuracy, using a judge that was 16× cheaper than oracle labels ($0.00014 vs $0.0022 per call).

Ranking performance comparison (n=1000, 25% oracle coverage). Direct+cov consistently outperforms off-policy methods.

Two cheap guardrails to prevent embarrassing memes

Your S→Y mapping can drift over time or fail to transport to new users, tasks, or model versions. Two lightweight checks catch this early:

1. Watch for drift over time

Periodically collect small batches of new (S, Y) pairs from current production. Apply your existing S→Y mapping (learned on old data) to predict Ŷ. Compute residuals: Y − Ŷ.

Alert if: Mean residuals go consistently negative (error bar doesn't overlap 0) for 2+ checks in a row. That means your old mapping is systematically over-predicting quality on new data—time to recalibrate.

2. Check transportability to new cohorts / policies

When launching a new policy, domain, or user segment, collect (S, Y) samples from that cohort. Apply your existing mapping and compare residuals to your baseline cohort.

Alert if: The new cohort has systematically different residuals (different mean, error bars don't overlap). Either add cohort as a feature in your mapping or fit a cohort-specific calibrator.

Transportability test procedure

1. Sample ~100–200 (S, Y) pairs per cohort.
2. Apply old f(S) → Ŷ; compute residuals Y − Ŷ and confidence intervals.
3. If cohorts differ materially (calibration curves diverge, non-overlapping CIs), re-fit or stratify the calibrator.

Example from our Arena benchmark. Click to view full-size:

Use residuals to improve S

Your residuals (Y − Ŷ) aren't just for drift detection—they're also a diagnostic tool for improving your cheap metric S. If your average residual is significantly below zero, S is systematically over-predicting quality. In this case, inspect the examples with the largest negative residuals to find patterns in what's fooling S.

This feedback loop—calibrate, inspect residuals, improve S, recalibrate—is how you iteratively build better evaluation systems.

Why your estimate has honest uncertainty

Standard approaches account for evaluation uncertainty—the variance from estimating your policy value on a finite sample of evaluations. Even if you had a perfect calibration function, you'd still have uncertainty from sampling.

But there's a second source most frameworks ignore: calibration uncertainty. You learned your S→Y mapping from a finite set of oracle labels. Different oracle samples (or different cross-validation folds) would give you a slightly different calibration function, and thus a different policy estimate. This is real uncertainty, and ignoring it produces overconfident confidence intervals.

Planning ahead: How much data do you actually need?

Before collecting data, you should plan proactively by deciding your target precision upfront, then calculating the sample size you need to achieve it. This prevents collecting too little data (can't detect real differences) or too much (wasting money on unnecessary precision).

The key metric is Minimum Detectable Effect (MDE): the smallest true policy difference you can reliably detect (80% power, 95% confidence). If a 1.5% improvement in satisfaction is worth shipping, you need enough data to distinguish 1.5% from noise.

Panel A: RMSE (accuracy of estimates). Panel B: Standard errors (width of confidence intervals). Panel C: MDE contours—the smallest effect you can detect. Darker cells = need more oracle labels. Dashed lines show common detection thresholds.

This visualization is from our Arena experiment benchmark. These numbers are specific to that dataset—your domain will differ.

Can't I Just Do Error Analysis and Assertion-Based Testing (for evals)?

It depends on your scale, complexity, and the stakes of your problem. Error analysis and assertion-based testing are essential starting points, but they're not sufficient as systems grow more complex or stakes become higher. Here's why:

• Only discourages visible bad behavior — doesn't encourage good behavior or catch subtle failures like "You're absolutely right!" that quietly tank long-term value. Works well for single, well-defined tasks; breaks down for complex systems like general-purpose AI assistants.
• It's brittle — binary pass/fail doesn't capture nuance or shades of grey, especially in complex systems with second-order effects.
• No statistical grounding — how many assertions do you need? Hard to make principled decisions when your mandate is just "find errors and assert they don't happen."

The optimal level of evaluation rigor depends on scale, complexity, and stakes. Here's a practical progression (each stage is inclusive of the previous):

Error analysis is one of the best ways to get S or Y labels. If users are obviously frustrated by a behavior (getting the date wrong, using the wrong name, stalling out), removing it will almost always map correctly to Y*. And assertion tests catch known failure modes. But calibration gives you the S→Y mapping for everything else—especially critical in complex systems like general-purpose AI assistants where S→Y relationships vary across different users, tasks, and contexts. You need both: error analysis finds failures, calibration tells you how your metrics predict outcomes.

What changes when you ship on calibrated Y

• Compare policies without expensive experiments. Once calibrated, evaluate new variants using cheap S labels instead of waiting months for retention data or expert audits.
• Catch inversions before they become memes. Calibration tells you when high S predicts low Y—stopping "You're absolutely right!" before it tanks user trust.
• Style tricks stop working. Verbosity, flattery, emoji-ification no longer masquerade as quality because you're measuring against real outcomes, not vibes.
• Climb toward Y* iteratively. As you collect better Y labels (longer-horizon outcomes, expert review), your decisions improve without retraining the whole system.

Ship value, not memes

CJE gives you honest confidence intervals on what actually matters. You'll catch inversions early, make faster decisions with fewer expensive experiments, and stop shipping behavior that scores high on vibes but tanks on outcomes.

Start with ~2K S labels and 500 Y labels. Calibrate. Deploy. Monitor periodically for drift. Recalibrate when relationships change. Watch memes about your product fade into the past.

Related Reading