Are Cart Recovery Calls Incremental? How to Run a Holdout Test

Randomly keep a share of eligible abandoned checkouts out of the calling step, send both groups the same emails and texts, and compare the order rate after a fixed window. The difference between the two groups, not the revenue attributed to calls, shows how many orders the calls added.

Some shoppers return on their own, and a last-contact attribution model credits the last contact before the purchase. That can help you understand the path to an order without telling you whether the order would have happened anyway. A holdout gives you a comparable audience that receives the recovery program without the call.

Use the attributed report for everyday campaign review and the experiment for the spending decision. Keep the question narrow: how much does adding this calling step change orders and contribution for the eligible audience?

Define eligibility before assigning a checkout to a group. Use the same cart rules, permitted hours and calling consent requirements for everyone who enters the test. Then assign each eligible checkout at random before any call can take place.

Hold out 10% to 20% of eligible checkouts, leaving most in the calling program while building a comparison group. Choose the share before launch and keep the assignment attached to the checkout. A larger holdout builds the comparison audience faster but leaves more checkouts without the calling step during the test.

Never move a checkout between groups after seeing whether the shopper answers, clicks or purchases. Keep an assignment record so you can check that the original split survives through reporting. If eligibility changes during the test, record the change and consider whether the periods can still be compared fairly.

Decide how repeat abandoned checkouts from the same shopper will be handled before launch. Avoid a situation where that shopper is held out in one tool but called through another checkout or campaign. Review overlapping campaigns with the team that owns the recovery sequence.

Keep emails, texts, discounts and stop rules identical. The only difference should be the calling step. If the calling group receives a better offer or an extra message that the holdout does not receive, the result measures those differences together.

Check the scheduled messages in every recovery tool before starting. A call-triggered SMS can create an extra difference unless the comparison program handles the equivalent message consistently. Write down exactly what each group receives and how you will interpret the result if a follow-up is part of the calling step.

Apply stop-on-order and opt-out handling to both groups. A recovered shopper should not keep receiving pending outreach simply because their experiment assignment remains on record. Preserve the assignment for measurement while stopping the contact as intended.

Record changes that cannot wait, such as a stock issue or a necessary policy update. You will need that context when interpreting a surprising result.

Run the test for at least two full weeks so every weekday is covered, avoid major promotions and continue until each group has enough orders to compare. Choose an observation window for purchases and apply it to every assigned checkout. The time you spend enrolling checkouts and the time each checkout has to purchase are separate decisions.

Do not compare a newly enrolled checkout with one that has already had its full purchase window. Wait for the final enrolled audience to complete the window before producing the final result. Otherwise the newest checkouts can make the order rate look lower simply because they have had less time.

Keep running when the order counts are too small to distinguish a useful difference. Review the experiment's uncertainty alongside the contribution calculation rather than stopping as soon as the called group appears ahead. A busy promotion may also change the audience enough that you need a separate test for ordinary trading periods.

Use all checkouts originally assigned to each group, including those whose calls were unanswered. Calculate orders within the same observation window, then compare the rates:

• Order rate = orders within the window ÷ checkouts in the group.
• Incremental orders = (called-group rate − holdout rate) × checkouts in the called group.
• Cost per incremental order = calling cost ÷ incremental orders.
• Incremental contribution = incremental orders × contribution per order − calling cost.

Use contribution after product cost, fulfillment, fees, discounts and expected returns. Check your actual calling spend against the plans and answered-conversation costs, including answered conversations that did not produce an order (Loop Voice pricing data, 2026). If the estimated incremental orders are zero or negative, report that outcome rather than presenting a positive cost per added order.

Small differences can be noise, particularly when the holdout produces few orders. Use a two-proportion significance test or a confidence interval for the difference in order rates. The uncertainty range helps you see whether the result distinguishes a meaningful improvement from an unclear outcome.

Ask someone comfortable with experiment analysis to review the counts and method if you do not normally run these tests. Keep the assigned-checkout counts, order counts and observation window available for that review. A percentage without its underlying audience size makes the result harder to assess.

When the result is unclear, continue collecting comparable data or rerun the test. Avoid repeatedly changing the offer, audience and call content while expecting the same experiment to answer the original question. Once you make a substantial change, treat the next comparison as a test of that new program.

Comparing only answered calls selects the people who picked up rather than everyone assigned to the calling step. That can flatter the result by excluding checkouts where the program never reached the shopper. Keep unanswered calls in the called group's denominator.

Calling holdout shoppers through another tool removes the intended difference between groups. Audit overlapping win-back and recovery campaigns, and make the exclusion visible to the people running them. Keep the observation windows identical so late orders are treated the same way.

Changing the offer midway through the test makes the result harder to interpret. If a change is unavoidable, record when it happened and which checkouts experienced it. Check randomization and stop rules before concluding that an unexpected result came from the call script.

Loop Voice attribution ties recovered revenue to calls for day-to-day reporting; customer results show how stores use that reporting (Loop Voice data, 2026). Run a holdout a few times a year to check how much the calling step adds, keeping the group assignment in your experiment records and comparing contribution after costs.