Design a Local Marketing Holdout Test Before Changing Spend

When a business changes its marketing budget and enquiries move, the team may credit or blame the campaign without knowing what would have happened otherwise. A holdout design creates a comparison that helps estimate incremental impact, but it requires careful planning and may not be practical for every small business. This guide covers the decisions to make before a local marketing test, from the unit of comparison to the risk of spillover between areas.

Write the decision and hypothesis

State the decision the test will inform, such as whether to expand a paid campaign to additional service areas. Define the expected mechanism and primary outcome before launch. Avoid vague hypotheses like 'marketing will improve'. Choose a business outcome that can be measured consistently, such as qualified enquiries or completed bookings, and identify the minimum operationally meaningful change if the business can set one responsibly.

Choose a feasible unit of assignment

Possible units include geographic areas, eligible time blocks, campaigns, or customer groups, but each creates different risks. Neighbouring areas may share customers, and ads may be seen outside intended boundaries. Time-based tests may be affected by weekday patterns, holidays, and changing demand. Select the unit that the business can assign and measure reliably; do not use a complex design merely because it sounds scientific.

Check whether the sample can answer the question

Estimate the likely volume of eligible outcomes and the duration needed to observe them. If the business receives very few leads, a short test may be unable to distinguish a real effect from ordinary variation. Do not invent a statistical power claim without the required assumptions. When a controlled test is not feasible, choose a transparent observational comparison and state that it cannot isolate the campaign's causal effect.

Protect customer access and operational fairness

A holdout should not remove essential information or deny a customer service they would normally receive. Get approval for budget and geographic restrictions, ensure staff can handle demand, and define stop conditions for harm or severe operational disruption. Record other campaigns, pricing changes, and availability issues that might contaminate the comparison. If treatment and control areas exchange customers, document the spillover risk.

Predefine analysis and decision rules

Specify the start and end dates, exclusion rules, outcome window, missing-data treatment, and comparison method before results arrive. Keep an experiment log and avoid changing the main metric after seeing the data. At the end, report the estimate, sample size, uncertainty, implementation deviations, and practical implications. A statistically uncertain result is not necessarily proof of no effect; it may indicate the test was too small or noisy.

Worked example

Hypothetical example: a multi-town service company considers a paid campaign expansion. It selects comparable service zones, verifies lead destinations, predefines qualified enquiries as the primary outcome, and records that cross-zone customer movement may blur the contrast.

Document deviations as they happen

Staff may forget to apply the treatment, campaigns may spend outside the intended area, or the website may change for everyone. Record each deviation with date and likely effect on the comparison. A transparent imperfect test is more informative than an apparently clean result whose exceptions were hidden.

Choose the simplest valid design

A local business does not need a complicated experiment if a smaller, feasible comparison answers the decision. If random assignment is not possible, say so and use the strongest practical alternative. The credibility of the conclusion depends more on clear assumptions and consistent measurement than on technical language.

Frequently asked questions

Can every local business run a holdout test?

No. Low volume, geographic overlap, and operational constraints may make a controlled design impractical.

Can I change the main metric halfway through?

You can add exploratory analyses, but label them separately and preserve the predeclared primary outcome.

What if the result is inconclusive?

Report the uncertainty honestly and decide whether more data or a different design is worth the cost.

Decision rule

Decision rule: run a holdout only when assignment, outcome measurement, operational risk, and the expected sample can support a meaningful comparison.

## Keep the analysis plan accessible

Store the plan somewhere the people implementing the test can read it before launch. Include who may change targeting, how budget is controlled, what happens if a tracking failure occurs, and which dates are excluded for a documented reason. If the plan must change, record the date and rationale before reviewing the final outcome. A transparent record helps stakeholders understand the limits of the estimate and makes a later replication more useful.

Evidence to retain

For the holdout test, save the predeclared outcome, assignment rule, stop conditions, and deviations before interpreting the final result.