Belief Cartography: Turning LLM Product Intuitions into an Experiment Portfolio

The idea

Every product team sits on event streams that nobody fully reads: clickstreams, usage logs, support touches. And every team has a backlog of intervention ideas ranked mostly by whoever argued best in the planning meeting. The tempting shortcut is to ask an LLM to do both jobs — read each user's journey, describe where they are, propose the intervention with the highest expected value, and estimate the customer's lifetime value in the current world and in the counterfactual world where the intervention ships.

Taken literally, that shortcut is unsound. The judge has no identification: there is no data from the counterfactual regime, so both “estimates” are fabrications, and their difference — the lift — is a difference of two hallucinations. But there is a sound version, and it needs only one reframe: the pipeline's output is a map of the judge's beliefs, not a map of reality. Beliefs are cheap to elicit at scale, they turn out to correlate with real experimental outcomes, and a belief surface over user-journey space is exactly the right prior for deciding which experiments to run first.

The one-sentence version: structured elicitation of an LLM's beliefs into a surrogate objective over user-journey space, optimized under population density, and settled by experiment. We spent one day and $22.74 of API calls stress-testing every clause of that sentence — one simulation with known ground truth, one synthetic pilot with planted archetypes, and two real datasets: 500 experiments from the Upworthy Research Archive and 220 student journeys with realized outcomes from OULAD.

How good does the judge have to be?

Before touching real data, we built a world where we control everything: a 2D journey space with a mixture-of-Gaussians population, planted opportunity pockets (a real winner in a dense region, a hidden gem in a sparse one), and — crucially — a folklore bump: a region where the judge believes there is value and there is none, planted in the densest segment, because that is exactly where LLM tropes live. The judge's beliefs mix truth, folklore, systematic error, and query noise; a knob sets how correlated its region ranking is with reality.

The pipeline then plays a sequential game: partition users into regions, take the judge's believed-lift ranking as a Bayesian prior (location and scale calibrated from “historical” effect sizes — the judge supplies ranking only), and allocate a budget of eight segment-level experiments by upper confidence bound. The question: how correlated does the judge's ranking have to be with truth before this beats not having a judge at all?

The answer is: not very good. The break-even sits around rank correlation ρ ≈ 0.15–0.2. A mediocre judge (ρ between 0.15 and 0.45) finds the true best segment 79% of the time versus 56% for the same adaptive search without the prior, and roughly halves cumulative regret. A good judge finds the winner in about 2 experiments instead of 6. And the downside is tiny: because the prior carries only a ranking and the experiments keep arbitrating, even a useless judge costs almost nothing — the loop washes it out. Asymmetric payoff: that is the strongest structural argument for the method.

One caveat the simulation taught us that we only appreciated later: the wash-out guarantee depends on experiments being informative. Segment-level experiments aggregate many users and produce tight estimates; if your “experiments” reveal one noisy observation at a time, a wrong prior stands for a long time.

Are LLM priors real? 500 actual A/B tests

The load-bearing premise is that judge beliefs correlate with real experimental outcomes. The Upworthy Research Archive — 32,487 real headline A/B tests from 2013–2015 — lets us measure that directly. We selected 500 well-powered tests (same image across arms so the headline is the isolated treatment, |z| ≥ 2 so the ground-truth direction is reliable, presentation order randomized), showed the judge both headlines, and asked it to predict the winner and the relative CTR lift.

• Haiku: 65.2% ± 4.2% directional accuracy (chance is 50%), signed believed-vs-realized lift Spearman 0.36. Sonnet on a 100-test subset: 67%, ρ = 0.35.
• Confidence is informative: accuracy rises from 45% in the lowest-confidence bin to 73% in the highest. Re-asking the same question five times is informative too: on pairs where the judge flips its answer across resamples, error rates are 50% versus 33% on stable pairs.
• Magnitude is not: the judge's ability to rank gap sizes (direction-free) is weak, ρ ≈ 0.13–0.19. Judges rank winners; they cannot size effects. Take rankings, never levels — the same conclusion CJE reaches for policy evaluation.
• No contamination detected: accuracy is flat across test-prominence quartiles, and a completion probe found zero near-verbatim recall on deployed and never-deployed headlines alike.
• But watch for position bias: the judge picked the second-presented headline 60% of the time regardless of content (accuracy 55% when the winner sat in slot A vs 76% in slot B). Our order randomization keeps the aggregate honest, but single-order querying leaves accuracy on the table — in deployment, query both orders and average.

Context makes the 65% more interesting than it looks. The LOLA paper ran essentially this evaluation on the same corpus in 2024: GPT-3.5 performed at chance and GPT-4 only marginally better. Today's cheapest frontier-lab model clears the simulation's break-even with room to spare, at roughly $0.01 per judgment. Every negative result of the form “LLMs can't predict experiments” carries a model-generation expiry date, and several have already expired.

The judge never says no

Here is the failure mode that makes the settlement layer non-optional. Across every dataset we tested, the judge proposed a positive-lift intervention for essentially every entity it saw — including the ones doing great:

• In a synthetic SaaS pilot with planted ground truth, 0 of 14 healthy accounts got “none” — and the healthy cluster ranked #2 on believed total value ($10.8k believed; $350 planted).
• On 220 real OULAD student journeys, 0 of 110 observably-thriving students (all assessments submitted, scores ≥70, consistently active) got “none.” Mean believed gain on these students: +16.7 percentage points of completion probability — including the 87 who went on to finish well with no intervention at all.
• Folklore shows up at cluster level too: a “silent dropout risk — post-assessment cliff” segment carried one of the highest believed gains, while its realized bad-outcome rate (0.29) was below the population base rate (0.42). The trope reads as danger; the reality is benign coasting.

This is not a prompt bug; it is what a helpful assistant trained on product-advice text is. In the simulation, once folklore gets moderately strong, the naive believed-lift × density map picks the folklore region as its top opportunity 60% of the time. An unaudited LLM opportunity map is a confident, legible, well-written atlas of product folklore. The experiments are what turn it into causal knowledge — and in the simulation, two to three segment-level experiments are enough to correct it.

What else broke (and what it teaches)

Three more results from the real-data rounds, all of which sharpened the method's boundaries:

1. A map of a flat surface is worthless. We replayed the full cartography loop on Upworthy: cluster the content space, aggregate judge beliefs per region, run the prior-guided experiment loop against real archived outcomes. It did not beat a flat prior — because Upworthy's opportunity surface is nearly flat. Mean realized lift varies between content regions by only 0.16 while varying within regions by 0.87; the top two regions sit within 5% of each other. When every region pays about the same, there is nothing for a map to find, and any mass-aware adaptive method captures ~97% of achievable value. Measure opportunity heterogeneity across segments before building any map. Product journey-space tends to be heterogeneous (our pilot's planted spread: $0 to $28k per segment); content-style spaces may not be.

2. Reading is not the edge. On OULAD, the judge's withdrawal-risk predictions from raw logs are genuinely calibrated (AUC 0.715; predicted-risk bins realize monotonically: 0.26 / 0.52 / 0.80). They are also strictly dominated by a five-feature logistic regression (AUC 0.839), and add nothing on top of it (0.841). Where the signal is legible in simple counts, a boring model beats the judge at prediction. The judge's unique contribution is upstream and downstream of prediction: generating the segment × intervention hypotheses and articulating them — which is precisely the part observational data cannot verify. That verification requires experiments.

3. Explanations confabulate — and the audit catches it. When we asked the judge to articulate the common thread of a top segment against easy contrast examples, it produced a rule that overfit to exemplar constants (“12–14 seats, $588–686 MRR”) and classified held-out members barely above chance (8/12, in-cluster recall 2/6). Re-run with hard negatives (nearest out-of-cluster neighbors) and an explicit no-constants instruction: a scale-free behavioral rule whose in-cluster recall went to 6/6 (the held-out negatives were easy in both versions — we quote the recall, not a 12/12). On real OULAD clusters — which are fuzzier (silhouette 0.06) — the same audit scored 9/12, and the rule it produced is the kind of thing a product team can actually ship against: “scored well early, still engaged, but missed the first high-stakes assignment — a motivational gap, not a skill gap → timed nudge plus a no-penalty extension.” Validate every narrative as a classifier before it goes in a deck.

The audit stack

If you build any version of this — and several teams are quietly building exactly this — the audits are not optional hygiene. They are what separates a prioritization engine from a mythology generator:

Relation to prior work

The pieces adjacent to this have real literatures. LOLA combines LLM predictions with bandits to allocate traffic among given content variants — the settlement half of this method, in production. A recent surrogacy framework for LLM-based A/B testing gives identification conditions for calibrating LLM outputs to human treatment effects (raw outputs recover ~39% of the effect on Upworthy; calibration closes the gap) — the rigorous measurement-side complement to our deliberately weaker ranking-only requirement. LLM forecasting of field experiments shows domain-dependent skill (78% directional accuracy on economics experiments, with systematic failures on sensitive social domains) — reinforcing per-domain ranking audits. The text-as-data causal inference literature (Egami et al.; Feder et al.) covers the LLM-as-measurement role for observational settings. What we have not seen claimed: eliciting beliefs over user-journey space from observational logs to generate segment × intervention hypotheses, density-weighted into an experiment portfolio, with the audit stack as a first-class part of the method. That strip is what this post stakes out.

When to use this (and when not to)

Reproducibility & what's next

Everything above ran in one day on public data: the Upworthy exploratory split (respecting the archive's exploratory/confirmatory protocol) and OULAD, with Claude Haiku 4.5 as the workhorse judge and Sonnet for the model-gradient subset and thread articulation. Total model spend: $22.74. The one claim public data cannot settle is the one that matters most commercially: whether the judge's generated interventions beat a product team's backlog when both are tested. That requires a partner with real journeys and a real experiment pipeline. If that's you, get in touch.