How to A/B Test Shopify Upsell Popups Without Lying to Yourself

What to test (offer, trigger, frequency), what not to test (button colour in week one), and the sample size trap that makes popup tests look like winners.

1 min read CRO & Analytics Monet
Editorial illustration for How to A/B Test Shopify Upsell Popups Without Lying to Yourself

Key takeaways

  • Test one variable: the SKU, the trigger, or the cap — not all three. A "new popup" vs "old popup" test teaches you nothing.
  • Primary metric is attach rate of the offered SKU, plus checkout initiation, so you catch tests that add items and lose checkouts.
  • Do not call a winner on 200 sessions. Overlay tests are noisy; wait for a sample you would defend in a meeting.
  • Stop tests that hurt add-to-cart-to-checkout even if attach rate is up. A bigger basket that does not pay is not a win.

Popup tests fail in a particular way: the overlay gets more clicks, AOV ticks up in the test group, checkout initiation quietly drops, and someone ships the winner. Two weeks later revenue is flat and nobody connects it.

You have to measure the thing you wanted (they added the SKU) and the thing you cannot afford to lose (they still paid).

One variable

Good tests:

  • This complementary SKU vs that one, same trigger
  • Add-to-cart trigger vs exit-intent, same SKU
  • Session cap of one vs two, same SKU and trigger

Bad tests:

  • New design + new SKU + new trigger vs the old everything
  • Button colour before the offer is even the right product

Metrics

Must-haves:

  • Offer impressions
  • Offer adds (attach)
  • Checkout initiation rate
  • Conversion rate to paid order
  • Revenue per session

If attach is up and checkout initiation is down, you did not find a better upsell. You found a speed bump.

Sample size

Overlays fire on a subset of sessions. Your real n is impressions, not store sessions. A week of a small store can be hundreds of impressions, not thousands. Do not declare a 12% lift on 180 views.

If you cannot power the test, do not run it. Make a merchandising decision and watch revenue per session for two weeks.

Guardrails

Hard-stop the test if checkout initiation drops beyond a threshold you set in advance. Pre-commitment is the only thing that stops a team from "giving it another day" on a losing overlay.

A/B testing is not a personality. It is a way to stop arguing. Use it when you have enough traffic to be wrong in public. Use judgement when you do not.

Frequently asked questions

Should I test popup vs no popup first?

Yes, if you do not already know the overlay is net-positive. Many stores skip this and optimise a thing that should not exist.

Can I use Shopify's reports for this?

Not for overlay-level attach. You need the app's offer analytics or a custom event (offer_shown, offer_added) in your pixel. Order value alone hides whether the popup did the work.

How long should a test run?

Through at least one weekly cycle (weekday vs weekend mix). Stopping on a Tuesday because it "looks good" is how you ship noise.

What if traffic is low?

Run fewer tests. Sequential changes with a long observation window beat a 12-variant experiment you cannot power.

Ninety9 Team

We build 5 conversion apps used by Shopify merchants in Bulgaria and beyond. Everything we write here comes out of what we see in real store data.

Keep reading

Related articles