Causal Inference Crisis: Opt-In Bias Skews AI Feature Metrics – Propensity Scores Offer Solution

Breaking: Product Experimentation Teams Face Systematic Bias in AI Feature Tracking

Product teams measuring the impact of LLM-based features like AI assistants are discovering a critical flaw: users who opt into these features are fundamentally different from those who do not, rendering naive comparisons statistically invalid.

According to a detailed technical analysis released today, the so-called 'Opt-In Trap' skews key metrics by conflating the feature's causal effect with pre-existing user engagement differences. One example cited: users who enable an 'agent mode' show 21 percentage points more task completion, yet this gap partly reflects that heavy users opt in more frequently.

"When users click 'Try our AI assistant,' the volunteers aren't a random sample," said Rudrendu Paul, data scientist and author of the study. "Any dashboard comparison between those who toggle on and those who don't embeds selection bias."

The Opt-In Trap Defined

The core issue emerges when generative AI features (smart replies, code suggestions, agent modes) are deployed behind user-controlled toggles. The toggle creates self-selected groups that differ on engagement, skill, or motivation before the experiment begins.

"A t-test on dashboard numbers cannot fix pre-existing group differences," Paul explained. "This is not a standard A/B test, where randomization ensures equivalent populations."

Propensity Score Methods as a Solution

To separate adoption bias from true feature effect, the research advocates for propensity score methods. These statistical techniques reweight or rematch user groups so that they appear comparable on observable characteristics, approximating a randomized experiment.

The analysis walks through a full pipeline on a 50,000-user synthetic SaaS dataset where the ground-truth causal effect is known. Steps include propensity estimation, inverse-probability weighting, nearest-neighbor matching, balance diagnostics, and bootstrap confidence intervals.

Tutorial and Companion Code

All code runs end-to-end in a companion notebook available on GitHub (file psm_demo.ipynb with pre-executed outputs). The tutorial is designed for data scientists working directly with LLM product features.

Key prerequisites include Python 3.8+, pandas, numpy, scikit-learn, and statsmodels. The synthetic dataset mimics a SaaS platform with user engagement metrics and a true causal effect embedded.

When Propensity Score Methods Fail

The research also explicitly addresses limitations: propensity scores rely on the ignorability assumption (no unmeasured confounders). When hidden factors influence both opt-in and outcomes, the method silently breaks. "If users who opt in are also more likely to use other advanced features not captured in your data, your adjusted estimate remains biased," Paul cautioned.

Background

As generative AI products proliferate, product teams increasingly deploy features behind toggles to gather user feedback. However, standard A/B testing infrastructure often cannot randomize exposure, because opt-in is inherently voluntary. This measurement gap has led to widespread misinterpretation of feature performance metrics across the industry.

Propensity scores have been used for decades in medical and social sciences to address selection bias. Their application to LLM product experimentation is relatively new, and the current tutorial aims to bridge that gap.

What This Means

For product managers and data scientists, the message is clear: dashboard comparisons of opt-in vs. non-opt-in users are unreliable. Adoption of rigorous causal inference methods—starting with propensity score weighting or matching—is necessary to avoid acting on spurious results.

"The 21-point gap is not the agent's effect alone," Paul emphasized. "Until teams adjust for selection, they risk making product decisions based on inflated or misleading metrics."

The full tutorial and code are expected to serve as a template for teams building causal inference pipelines for any LLM-based feature deployed behind a toggle.