Programmable Proxies: Designing, Calibrating, and Auditing LLM Judges as Surrogate Outcomes

0. Notation & Setup

Primitives

• $(X, A)$: Context (prompt) and action (response/policy)
• $Y \in [0,1]$: Target outcome (expert audit, IDO realization, or user satisfaction)
• $\theta$: Judge prompt/rubric specification
• $J_\theta$: Judge function parameterized by $\theta$
• $S_\theta = J_\theta(X,A)$: Raw judge score (may include sub-rubrics)
• $f_\theta: S_\theta \times X \to [0,1]$: Calibrator function
• $R_\theta = f_\theta(S_\theta, X)$: Calibrated reward

Estimand

Policy value under target outcome $Y$:

Data

• Logging data: $\mathcal{D}_0 = \{(X_i, A_i, S_{\theta,i})\}_{i=1}^n$ from behavior policy $\pi_0$
• Oracle slice: $I_{\text{oracle}} \subset \{1,\ldots,n\}$ with $|I_{\text{oracle}}| = n_Y$ labeled $Y$ values
• Oracle coverage: $\rho = n_Y / n$ (typically 10–25%)

1. Endogenous Surrogate Theory

Key insight: Unlike legacy business proxies (clicks, sign-ups) which are exogenous signals, LLM judges are endogenous—their measurement channel is programmable via the rubric $\theta$.

Practical implication: When residual analysis reveals that the judge is missing signal (e.g., rewarding verbosity, missing hallucinations), we can update $\theta$ to require explicit evidence checks, then verify that calibration error decreases.

1.5 Why Calibration Is Optimal: The Pathwise Derivative

We now show why calibration f(S) = 𝔼[Y|S] is not merely useful—it is variance-optimal for estimating policy value. This section connects calibration to semiparametric efficiency theory, showing that f(S) is the Efficient Influence Function (EIF) for the welfare functional.

The Setup: Welfare as a Functional

Our target estimand is the expected welfare under policy π_θ:

We want to understand how ψ changes as we vary the policy parameters θ. Specifically, we seek the pathwise derivative—the gradient of ψ with respect to θ along any smooth path in the policy space.

The Decomposition

Any welfare outcome Y can be decomposed into two orthogonal components relative to the surrogate score S:

This decomposition has three critical properties:

The Pathwise Derivative

Consider a smooth path through policy space: π_θ where θ ∈ ℝ. The pathwise derivative of the welfare functional is:

Substituting the decomposition Y = f(S) + ε:

The second term vanishes. Because ε is orthogonal to S (which is a function of π_θ), any change in θ that affects S does not affect the expected value of ε:

Therefore, the pathwise derivative simplifies to:

Geometric Interpretation

What does this mean geometrically? The space of all possible outcomes Y forms a Hilbert space. The surrogate S generates a tangent space—the subspace of outcomes that can be predicted from S.

• f(S) = 𝔼[Y|S] is the orthogonal projection of Y onto this tangent space.
• ε = Y - f(S) is the residual—the component of Y orthogonal to the tangent space.
• When we optimize, ∇S (the raw gradient) points in a direction that includes both the tangent space and orthogonal directions (exploitation).
• ∇f(S) (the calibrated gradient) points only in directions within the tangent space—directions that actually move toward higher expected welfare.

This is why calibration removes reward hacking: hacking vectors live in the orthogonal complement (the ε space), and calibration projects them out.

Connection to Regime 4

This derivation applies to Regime 3 (Calibration) in the surrogacy taxonomy. In Regime 4 (Optimization), we additionally require that the policy optimization process remains on the tangent space. This is where Standard Deliberation Protocols (SDPs) enter: they constrain the action space to ensure that optimization steps cannot exploit side channels outside the tangent space defined by f(S).

See Y*-Aligned Systems (Technical) for the full treatment of Regime 4 and the role of SDPs in maintaining alignment during optimization.

Practical Implications

This is the theoretical foundation for CJE. When we calibrate judges in practice (Section 3), we are approximating the EIF using a finite gold sample. The doubly-robust estimator (Section 2) combines this EIF approximation with importance weighting to achieve valid inference even under model misspecification.

For geometric intuition, see The Geometry of Goodhart's Law for the manifold interpretation of this result.

2. Policy Evaluation with Calibrated Rewards

2.1 Doubly Robust Estimator

Given calibrated reward $R_\theta = f_\theta(S_\theta, X)$, the doubly robust estimator is:

where:

• $w_i = \pi(A_i \mid X_i) / \pi_0(A_i \mid X_i)$: importance weight
• $\hat{Q}(X,A)$: outcome regression model
• $\hat{Q}_\pi(X) = \mathbb{E}_{A \sim \pi(\cdot \mid X)}[\hat{Q}(X,A)]$: policy value function

2.2 Oracle-Uncertainty-Aware (OUA) Variance

Standard DR inference accounts for evaluation uncertainty (sampling $Y_i$ from the population). But there's a second source: calibration uncertainty from learning $f_\theta$ on a finite oracle slice.

3. Closed-Loop Design of Measurement

The core algorithmic contribution: a feedback loop that alternates between calibration and prompt refinement.

Default hyperparameters

• Calibrator: Two-stage if covariates available (spline over [S, length, has_citation], then isotonic on $T = g_k(S, X)$). Otherwise, isotonic on $S$ directly.
• Cross-fitting: $K = 5$ folds. Larger $K$ reduces bias at cost of higher variance.
• Residual slicing: 10–20 groups (domain × difficulty × length bins). Use FDR correction to control family-wise error rate.
• Stopping: Two consecutive iterations with no significant residual structure (all group means have CIs overlapping 0) and transport diagnostics pass.

judge_model:      gpt-4.5-mini
judge_prompt_hash: sha256:ab12…ef
rubric_version:    3.2
calibrator_version:isotonic-v5

3.1 Panel-of-judges as first-class primitive

Real quality is multi-objective. Rather than forcing all dimensions into a single scalar, use a panel of judges where each judge targets a specific dimension (accuracy, safety, concision, domain-appropriateness). Let \\mathbf{S}_\\theta = (S_{\\theta,1}, \\ldots, S_{\\theta,K}) denote the K-dimensional judge vector.

Learnable aggregator: Define S_\\theta^{\\text{agg}} = w^\\top \\mathbf{S}_\\theta where $w \\in \\mathbb{R}^K$ is learned via calibration to minimize \\mathbb{E}[(Y - f_\\theta(w^\\top \\mathbf{S}_\\theta, X))^2] on the gold set. This allows the data to determine relative weighting rather than hard-coding trade-offs.

Publish the vector, not just the scalar: Log the full \\mathbf{S}_\\theta alongside S_\\theta^{\\text{agg}}. The scalar gates decisions (pass/fail), but the vector provides:

• Telemetry: Which dimension drove the failure? (e.g., high accuracy but low safety)
• Dissent tracking: When judges disagree, flag for human review
• Stakeholder transparency: Show domain experts the specific dimension scores, not just aggregated "quality"

judges:
  - name: accuracy
    weight: 0.4
    rubric: "Score 0-10 on factual correctness. Penalize unsupported claims."

  - name: safety
    weight: 0.3
    rubric: "Score 0-10 on safety. Block harmful content, bias, privacy leaks."

  - name: concision
    weight: 0.2
    rubric: "Score 0-10 on brevity. Penalize unnecessary verbosity."

  - name: domain_fit
    weight: 0.1
    rubric: "Score 0-10 on domain appropriateness (medical, legal, etc.)."

aggregation: learned_linear  # learns w from calibration data

3.2 Verdict card schema

Every judge evaluation should emit a verdict card—a structured audit trail that logs not just the score, but the reasoning, evidence, and configuration. This makes judgments auditable by default and enables retrospective analysis when calibration drifts.

{
  "verdict_id": "vd_8f3a2b1c",
  "timestamp": "2025-11-17T14:23:11Z",

  "scores": {
    "aggregate": 7.2,
    "panel": {
      "accuracy": 8.5,
      "safety": 9.0,
      "concision": 5.0,
      "domain_fit": 7.0
    }
  },

  "config": {
    "judge_model": "gpt-4.5-mini",
    "rubric_version": "3.2",
    "rubric_hash": "sha256:ab12...ef",
    "calibrator_version": "isotonic-v5"
  },

  "evidence": {
    "citations_checked": ["source_1", "source_2"],
    "tests_run": ["unit_test_1", "schema_validator"],
    "flags": ["verbose_padding_detected"]
  },

  "rationale": "Response is factually accurate and safe, but contains unnecessary verbosity (300 tokens vs 150 needed). Concision score penalized accordingly."
}

Benefits: Verdict cards enable post-hoc debugging (why did this fail?), regulatory audit trails (what evidence was considered?), and drift detection (are recent verdicts systematically different from baseline?).

4. Transport Diagnostics

When does a calibrated judge $f_\theta$ trained on one environment transport to another (new policy, time period, user cohort)? We provide falsifiable tests.

4.1 Groupwise residual tests

For each group $g$ (policy/time/domain), test:

4.2 When transport fails

5. Anti-Gaming & Adversarial Validation

If the generator (model being evaluated) optimizes against the judge, raw scores $S_\theta$ can be gamed. Calibration + residual monitoring mitigate but don't eliminate this risk.

5.1 Validation battery

6. Sample Size Planning & Budget Curves

6.1 Approximate formulas (DR estimator)

Under standard regularity, the variance of $\hat{V}_{\mathrm{DR}}(\pi)$ scales as:

where:

• $\sigma^2_{\text{eval}}$: Variance from evaluation (DR influence function)
• $\sigma^2_{\text{oracle}}$: Variance from calibration uncertainty

Rule of thumb: If OUA share ≈ 0.2, then $\sigma^2_{\text{oracle}} \approx 0.25 \cdot \sigma^2_{\text{eval}}$. To achieve SE ≤ 0.025 (CI width ≈ 0.05), you need:

For typical tasks with $\sigma^2_{\text{eval}} \approx 0.05$–0.1, this translates to $n \approx 2000$–3000 and $n_Y \approx 300$–500.

7. Limitations & Open Questions

8. Conclusion

We have presented a formal framework for programmable surrogacy: LLM judges whose measurement function is endogenous and improvable through prompt engineering. Our closed-loop algorithm alternates between calibration (learning $S_\theta \to Y$) and residual-guided prompt updates (refining $\theta$ to reduce error). Under mild regularity, enriching the judge's evidence set weakly reduces prediction error (Lemma 1), and doubly-robust policy evaluation with calibrated rewards admits valid, oracle-uncertainty-aware inference (Proposition 2).

Key contributions:

• Theoretical: Information-ordering guarantee for prompt refinements; OUA variance decomposition for calibrated DR estimators; transport diagnostics for cross-environment validity
• Methodological: Closed-loop algorithm with stopping criteria; residual audit pipeline; anti-gaming test battery; sample-size planning with budget curves
• Practical: Drop-in templates for judge prompts, residual diagnostics, and governance checklists; reproducible experiments showing 20–40% RMSE reduction over two iterations

Assumptions Ledger

Citation

If you use this work, please cite:

@misc{landesberg2025programmable,
  author = {Landesberg, Eddie},
  title = {Programmable Proxies: Designing, Calibrating, and Auditing LLM Judges as Surrogate Outcomes},
  year = {2025},
  month = {November},
  url = {https://cimolabs.com/research/programmable-proxies-technical},
  note = {CIMO Labs Technical Report}
}

References