A/B testing
Mar 12, 2024

Definition
A/B testing, also called split testing, compares two versions of a page, email or campaign by showing each one to a random share of your visitors and measuring which gets more clicks, signups or sales. Version A is usually what you run today. Version B changes one thing, such as the headline, the offer or the timing.
Why it matters
Most marketing decisions get made in a meeting. Someone prefers the green button, someone else wants a bigger discount, and the loudest opinion goes live. You never learn whether the other idea would have won.
An A/B test settles the question with your own visitors. Both versions run at the same time, so a sale, a newsletter send or a change in season affects both equally. The difference you see comes from the change you made.
Testing also catches changes that only look like improvements: a bigger discount can raise signups and lower the margin on every order that follows.
How it works
- Write one change down as a hypothesis. "A fixed amount off converts better than a percentage" is testable. "Make the popup better" is not.
- Split the traffic. Visitors are randomly assigned to A or B.
- Choose the metric before you start. Pick the one closest to the outcome you care about: clicks, signups, purchases or revenue.
- Run it long enough. Stopping early often picks a winner that fades a week later.
- Check that the difference is real. Statistical significance tells you how likely the result is to hold.
There are two ways to run the test itself:
| Client-side testing | Server-side testing | |
|---|---|---|
| Where the change happens | In the visitor's browser, after the page loads | On your server, before the page reaches the browser |
| Good for | Copy, offers, images, calls to action, campaigns | Pricing logic, checkout flows, search |
| Setup | Fast, often without a developer | Needs developer time |
| Watch out for | Flicker when changing content at the top of the page | Slower to launch |
Triggerbee runs in the browser through a JavaScript tracking script, so its tests are client-side.
How you use it in Triggerbee
You can create an A/B test from any campaign. Variant A is the campaign you built. "Create Challenger" copies it as variant B, and you change one thing in the editor.
In the test settings you choose:
- Hypothesis: what you are testing, such as "percentage discount or fixed discount."
- Length: 7, 14 or 28 days. With fewer than 1,000 conversions a month, Triggerbee recommends at least 14 days.
- Winning metric: conversions or clicks. If the campaign collects email addresses, choose conversions.
Audience, triggers and URL targeting work as on any campaign, so you can test on returning visitors only. Regular variants cannot be used inside an A/B test, so run separate tests for desktop and mobile. A test cannot be paused, but you can stop it at any time.
When the test ends, Triggerbee emails you the winner, based on conversions or clicks relative to views. On the performance page you can compare each variant's funnel. Before you start, the A/B test calculator estimates how many days your test needs to reach a significant result.
Example
A fashion retailer runs a newsletter popup that offers 10% off the first order. The team wants to know if a fixed amount works better, so it creates a challenger that offers $10 off, with the same design, audience and trigger. It runs the test for 14 days with conversions as the winning metric.
When the test ends, the team also checks which variant led to more first orders, because a version that collects more emails but fewer buyers is the wrong winner.
Common mistakes
- Changing several things at once. If B has a new headline, image and offer, you cannot tell which change made the difference.
- Reading small gaps as wins. Triggerbee's rule of thumb: below 1,000 conversions a month, look for a difference of at least 15% between variants. Above 10,000 a month, at least 5%. Most tests do not produce a clear winner.
For test ideas, see 30+ things you should A/B test.