Marketing Landing Page Experiment Holdout Integrity and Contamination Control

Executive summary

What a holdout is meant to protect

A holdout group provides a comparison against which a change can be evaluated. If visitors assigned to the control experience later see the treatment through another campaign, shared device, cached page, or sales-assisted link, the comparison becomes less clear. Contamination does not always invalidate an experiment completely, but it can dilute differences and make the result harder to interpret. The operating objective is to define the unit of assignment, preserve that assignment, and record exposures that could compromise the intended comparison.

Choose the correct unit of randomisation

The unit might be a browser, visitor, account, household, or lead, depending on the product and buying journey. Randomising by browser can be convenient, but the same person may switch devices. Randomising by account may be more appropriate for a B2B purchase where colleagues share information. The choice should be made before launch and documented with its trade-offs. Do not switch the unit mid-test because results look inconvenient; that changes the meaning of the experiment.

Keep assignment stable across visits

A visitor should normally continue to receive the same variant during the defined experiment window. Implement a durable assignment method appropriate to consent, privacy, and technical constraints. Document what happens when cookies are cleared, a visitor logs in, or an account is merged. If assignment cannot be preserved, quantify how often this occurs and decide whether the limitation is acceptable. Do not use hidden fingerprinting or other intrusive tracking to force identity continuity.

Map possible contamination paths

Potential paths include paid ads pointing directly to the treatment page, email links that bypass assignment logic, sales representatives sharing a preview URL, retargeting ads showing treatment-specific claims, browser caches, content delivery rules, and internal staff testing. Create a pre-launch map of these paths and assign an owner to each control. A treatment-specific URL should not become a public alternative route unless the experiment design explicitly supports that exposure.

Define exposure logging and quality checks

Log assignment, eligible page view, actual variant rendered, timestamp, experiment ID, and relevant exclusion reason. Distinguish assignment from exposure: a visitor can be assigned to treatment but never load the page. Check sample ratios, missing variant events, duplicate identifiers, unexpected redirects, and variant-specific performance failures. Establish alerts for material imbalances before the team reviews conversion outcomes, so technical defects are not mistaken for user preference.

Handle internal and external sharing

Provide a safe preview process for reviewers that does not add staff sessions to the experiment population. Sales and support teams should know whether a link is a preview, a production variant, or a stable public destination. If a customer receives a variant through a direct link, record the exposure when possible and assess whether it affects the chosen unit. Avoid relying on informal instructions such as “please do not share this link” as the only control.

Predefine exclusions and stopping rules

Decide before launch how to treat bots, employees, duplicate records, ineligible traffic, and known test accounts. Define the primary metric, guardrail metrics, minimum observation period, and stopping conditions. If a severe defect requires stopping the test, record the reason and do not describe the interrupted result as a clean winner. Repeatedly checking results and stopping when a preferred variant leads can create misleading conclusions even if assignment integrity is perfect.

Interpret contaminated results honestly

If contamination is discovered, estimate its extent where data allows, describe the likely direction of bias, and distinguish exploratory evidence from confirmatory evidence. A contaminated result may still reveal a usability issue or a technical failure, but the confidence in causal interpretation is reduced. Do not simply remove inconvenient users after seeing outcomes unless the exclusion rule was pre-specified or the analysis is clearly labelled exploratory. Document sensitivity analyses and the assumptions behind them.

A worked example: a pricing page test

A small SaaS team tests a clearer pricing comparison. The control and treatment are assigned at account level, but a salesperson shares a direct treatment link with several control accounts. The team records the incident, estimates the affected accounts, and reports the primary analysis with a contamination caveat. It does not quietly delete those accounts to improve the result. For the next test, sales receives a review environment and the CRM flags experiment participation before representatives send product links.

Implementation checklist and takeaways

Write down the assignment unit, persistence method, exposure definition, contamination map, exclusions, primary metric, and stopping rules before launch. Monitor assignment and exposure quality independently from business outcomes. When a problem occurs, preserve the evidence and qualify the conclusion rather than hiding the limitation. Experimental integrity is an operational discipline: a modest test with clear controls is more valuable than a large test whose exposure history cannot be trusted.