Pricing Experiment Exposure Contamination and Assignment Integrity

Overview

A pricing experiment can produce misleading results if the same account sees different prices across devices, sales staff override the assigned offer, or a separate promotion overlaps with the test. A large dataset does not repair inconsistent treatment. Define the experimental unit—visitor, account, household, store, or another entity—according to how the buying decision is made, and record eligibility before observing outcomes.

1. Diagnose the actual operating condition

Keep assignment, displayed price, and charged price as separate fields. A correct assignment does not prove correct exposure, and a displayed price does not prove that checkout charged the same amount. Map all touchpoints that can show the offer, including product pages, email, sales quotes, support messages, cart, and checkout. Define which channels are included and which must remain constant.

2. Define ownership and evidence

Monitor multiple variants for the same experimental unit, coupon bypasses, cached pages, sales overrides, and inconsistent session values. Preserve the reason for every contaminated record instead of silently removing it. Report the assigned population and observed contamination rate. A post-hoc filtered analysis may be useful as a sensitivity check, but it must not be presented as the original planned analysis.

3. Apply controls before release

Set customer-protection stop conditions before launch. Examples include a tested price appearing for an ineligible customer, checkout charging an incorrect amount, a sudden complaint spike, or a failure in assignment logic. Name who can pause the test and how the previous state will be restored. Do not change stop rules simply because early results are commercially inconvenient.

4. Handle exceptions without guesswork

At close, archive the configuration, reconcile exposure, and remove temporary promotion settings. Report assigned counts, correct exposure, contamination, checkout failures, and the pre-agreed outcome measures. A statistically strong result cannot compensate for broken assignment; describe the limitation and repair the experiment system before the next test.

Worked example

A subscription price test assigns variants by account, but a sales email uses a static audience segment. Monitoring detects that several accounts saw a different price from their assignment. The experiment owner pauses the email, records the affected exposure, and repairs the shared assignment source before restarting with test accounts.

Implementation checklist

• Define experimental unit and eligibility before launch.
• Store assignment, exposure, and charged price separately.
• Monitor cross-variant exposure across channels.
• Set customer-protection stop conditions in advance.
• Report contamination and follow the pre-agreed analysis plan.

Key takeaways and next steps

Experiment quality depends on consistent exposure, not merely random assignment. Report contamination honestly, and stop a test when customers receive an unintended price. Validate every channel that can change the offer.

References and further reading

• Microsoft Research, experimentation resources: https://www.microsoft.com/en-us/research/
• Google Analytics Help: https://support.google.com/analytics/