Net IDO Impact: Evaluating Foundation Models Against Realistic Alternatives

0. Motivation and problem setup

Standard evaluation measures policy value $V(\pi) = \mathbb{E}_{X}\mathbb{E}_{a \sim \pi(\cdot \mid X)}[Y^*(X,a)]$in isolation. But practitioners need to answer: compared to what?

A foundation model's value is not absolute—it's the net advantage over realistic alternatives(other models, software tools, human workflows) net of switching costs, latency, and learning curves. Moreover, FMs reshape the task landscape: they create new tasks (work that was previously infeasible),compete on existing tasks (work users were already doing), and may eliminate tasks(work that becomes obsolete or deprecated post-FM).

We formalize this as Net IDO Impact (NII), which decomposes into:

• Competition (exploitation): Net gains/losses on the pre-FM task distribution, re-weighted by post-FM prevalence. (Task elimination is captured via $w(t) \approx 0$.)
• Creation (exploration): Value on genuinely new tasks that have negligible pre-FM density.

1. Formal definitions

1.1. Spaces and measures

• Users: Measurable space $(\mathcal{U}, \Sigma_{\mathcal{U}})$with probability measure $P_U$.
• Tasks: Measurable space $(\mathcal{T}, \Sigma_{\mathcal{T}})$.
• Pre-/Post-FM task measures: $P_T^{0}$ and $P_T^{FM}$ on $(\mathcal{T}, \Sigma_{\mathcal{T}})$.
• Contexts and actions: Spaces $\mathcal{X}$, $\mathcal{A}$. For each $(u,t)$, let $P_{X \mid u,t}$ be the context distribution (prompts, histories) when that user does that task.

1.2. IDO and policy values

• Idealized Deliberation Oracle (IDO). For each $(u,t)$, $Y^*_{u,t}: \mathcal{X} \times \mathcal{A} \to [0,1]$ is measurable.
• Value of a policy $\pi$ for $(u,t)$:
  $$V(\pi; u,t) = \mathbb{E}_{x \sim P_{X \mid u,t}} \; \mathbb{E}_{a \sim \pi(\cdot \mid x)} \big[ Y^*_{u,t}(x,a) \big]. \tag{1}$$
• FM adapted policy and value. Given adaptation (prompting, fine-tuning) by $u$ for $t$, let $\pi_{FM,u,t}$ be the induced policy. Then
  $$V_{FM}(u,t) = V(\pi_{FM,u,t}; u,t). \tag{2}$$
• Effective alternative value. Let $\mathcal{Alt}(u,t)$ be feasible alternatives (other models, software, manual workflows). With friction/switching cost $C_s(\pi; u,t) \in [0,1]$ measured on the same utility scale:
  $$V_{alt}(u,t) = \sup_{\pi \in \mathcal{Alt}(u,t)} \Big\{ V(\pi; u,t) - C_s(\pi; u,t) \Big\}. \tag{3}$$

  Optionally include a cost term $C_{FM}(u,t)$ inside $V_{FM}$ to net out access/usage friction of the FM itself.

• Net advantage at $(u,t)$:
  $$\Delta(u,t) = V_{FM}(u,t) - V_{alt}(u,t). \tag{4}$$
• Positive/negative parts: $(x)_+ = \max(0,x)$, $(x)_- = \max(0,-x)$, with $x = x_+ - x_-$.

2. Net IDO Impact (NII)

We define three related quantities to capture gains, losses, and net welfare:

3. Dynamic task space decomposition

3.1. Lebesgue decomposition

The post-FM task measure can be uniquely decomposed into a part absolutely continuous w.r.t. $P_T^{0}$ and a singular part:

where $w = \frac{dP_T^{FM}}{dP_T^{0}}$ is the density ratio on the shared support and $P_T^{\perp}$ is the singular (creation) measure. Note:

3.2. Competition-Creation decomposition

Substituting equation (8) into equation (5):

The simplification in the creation integral (replacing $(\Delta)_+$with $V_{FM}$) corresponds to assuming $V_{alt}(u,t) \approx 0$for $t \in \operatorname{supp}(P_T^{\perp})$. If nascent alternatives exist for new tasks, use $V_{FM} - V_{alt}$ in the creation term as well.

4. Identification via surrogacy

Under the surrogacy assumptions S1-S2 from the IDO framework, let $R^{(k)} = f_k(S^{(k)}, X)$ be the calibrated reward on the IDO scale. If S-admissibility fails (i.e., selection into $Y^*$ depends on unobserved confounders), recalibration is required; see IDO appendix §3 for transport tests.

For any policy $\pi$, use Direct Method, IPS, or DR estimators to obtain $\widehat{V}(\pi; u,t)$ from logs or fresh draws. Then:

4.1. Components needed for identification

• Alternatives menu. Specify a realistic set $\mathcal{Alt}(u,t)$(e.g., another model, toolchain, human workflow) and measure friction $C_s$on the $[0,1]$ utility scale. For example, $C_s = \alpha \cdot \text{latency} + \beta \cdot \text{price} + \gamma \cdot \text{onboarding}$, with coefficients calibrated to convert time/cost into utility.
• Density ratio. Estimate $\hat{w}(t) \approx \frac{dP_T^{FM}}{dP_T^{0}}(t)$using a discriminator (e.g., logistic regression, KLIEP, Bregman divergence methods). Calibrate so that $\int \hat{w}\,dP_T^{0} \approx 1 - \widehat{M}_{\text{create}}$.
• Novelty / creation measure. Flag tasks with negligible pre-FM density as $\widehat{\mathcal{T}}_{\perp}$ to estimate the creation mass $\widehat{M}_{\text{create}} = P_T^{FM}(\widehat{\mathcal{T}}_{\perp})$.

5. Estimation

5.1. Plug-in Monte Carlo estimator

Sampling notation: Let $N_0$ = number of pre-FM samples $(u_i, t_i) \sim P_U \otimes P_T^{0}$, $N_1$ = number of post-FM samples $(u_j, t_j) \sim P_U \otimes P_T^{FM}$, and $N_\perp$ = number of post-FM samples flagged as creation tasks (i.e., $t_j \in \widehat{\mathcal{T}}_\perp$).

Use Direct/IPS/DR estimators (preferably DR for stability) to compute each $\widehat{V}$. Similarly estimate:

5.2. Uncertainty quantification

6. Modeling choices and assumptions

Several design decisions require explicit specification:

7. Worked example (discrete case)

Consider a simple discrete setting to illustrate the calculation:

Setup

• Users: $P_U(A) = 0.6, \; P_U(B) = 0.4$
• Pre-FM tasks: $t_1$ (translation, 0.7), $t_2$ (formatting, 0.3)
• Post-FM: $w(t_1) = 0.8, \; w(t_2) = 1.2$, plus new task $t_3$ with creation mass $P_T^{\perp}(\{t_3\}) = 0.08$
• Check: $0.7 \times 0.8 + 0.3 \times 1.2 = 0.92$; $0.92 + 0.08 = 1.0$ ✓

Values (net of friction)

Competition weights

For $t_1$: $0.7 \times 0.8 = 0.56$

• User A: $0.6 \times 0.56 = 0.336$
• User B: $0.4 \times 0.56 = 0.224$

For $t_2$: $0.3 \times 1.2 = 0.36$

• User A: $0.6 \times 0.36 = 0.216$
• User B: $0.4 \times 0.36 = 0.144$

Creation weights

$P_T^{\perp}(\{t_3\}) = 0.08$

• User A: $0.6 \times 0.08 = 0.048$
• User B: $0.4 \times 0.08 = 0.032$

Calculation

Competition gains:

• A, $t_1$: $0.336 \times 0.05 = 0.0168$
• A, $t_2$: $0.216 \times 0.20 = 0.0432$
• B, $t_2$: $0.144 \times 0.07 = 0.01008$

Creation gains:

• A, $t_3$: $0.048 \times 0.60 = 0.0288$
• B, $t_3$: $0.032 \times 0.72 = 0.02304$

Loss (for net calculation):

• B, $t_1$: $0.224 \times 0.08 = 0.01792$

8. What to report

9. Diagnostics and guardrails

Beyond the standard IDO diagnostics (transport tests, ESS, coverage), NII requires additional checks:

This extends the IDO measurement framework from single-policy evaluation to competitive benchmarking, turning "is this model better?" into a rigorous, measurable question with estimators, confidence intervals, and falsifiable diagnostics.

Citation

If you use this work, please cite:

@misc{landesberg2025nii,
  author = {Landesberg, Eddie},
  title = {Net IDO Impact: Evaluating Foundation Models Against Realistic Alternatives},
  year = {2025},
  month = {November},
  url = {https://cimolabs.com/blog/net-ido-impact},
  note = {CIMO Labs Technical Report}
}

References