A/B Testing for Beginners: How to Set Up Your First Test

Warning: fileperms(): stat failed for /www/.wordpress/ in /wordpress/wp-admin/includes/file.php on line 2225

A/B testing is the most rigorous way to know whether a change to your website actually improves conversion or just feels like it does. Done properly, it removes guesswork. Done badly, it actively misleads you, often in directions that feel right but cost you money for months before anyone notices. Most teams getting started with A/B testing make the same handful of mistakes, and most of them are avoidable with a clearer process upfront.

After a decade running CRO across ecommerce and service businesses, I’ve watched plenty of teams treat A/B testing like opinion-settling. “I think the green button works better; let’s test it.” The teams that actually move conversion rate treat A/B testing differently — as a structured way to test hypotheses they’ve formed from data, with statistical rigour, on changes large enough to matter. This guide is how I explain A/B testing to clients setting up their first test.

This guide covers what A/B testing is, when you can and can’t run a meaningful one, how to set up your first test step by step, how to calculate the sample size you need, how to read the results without getting fooled by noise, and the common mistakes that produce wrong conclusions. It pairs with our 7-step CRO framework and our guide to CRO tools.

What A/B Testing Is (And Isn’t)

An A/B test is a controlled experiment where two versions of a page or element (A = control, B = variant) are served to randomly assigned segments of your traffic, and the conversion rates of each are compared to determine if the variant produced a statistically significant improvement.

The key word is controlled. Both variants run simultaneously, to comparable audiences, with everything else held constant. That’s what separates A/B testing from “I changed the page, conversions went up, the change worked”. The before/after comparison is contaminated by everything else that changed in the same period (seasonality, marketing campaigns, traffic mix, day of week). A/B testing controls for all of that.

What A/B testing is NOT:

• A way to make small businesses with low traffic suddenly understand their customers
• A substitute for talking to customers
• A way to validate a change that has already been pushed live
• An excuse to test every micro-decision instead of forming opinions

When You Can (and Can’t) Run Meaningful A/B Tests

This is the question most teams skip. A/B testing requires enough traffic and enough conversion events to reach statistical significance within a reasonable time window. Without volume, your test will either never reach significance or will produce results that look conclusive but are actually noise.

Rough guideline using a standard A/B test sample size calculator: to detect a 10% relative lift on a baseline 3% conversion rate at 95% confidence, you need roughly 10,000-15,000 sessions per variant. That’s 20,000-30,000 total sessions for the test.

Practical implications:

• High-traffic pages (homepage, top 3 landing pages, checkout pages on busy ecom stores): A/B testing is feasible. Tests reach significance in 2-4 weeks.
• Medium-traffic pages (top 10-20 collection pages, top service pages): Possible but tests take longer (4-8 weeks). Test changes large enough to detect with the available volume.
• Low-traffic pages: A/B testing is not the right tool. Use qualitative research (session recordings, heatmaps, user interviews) and ship changes informed by that research without formal testing.

If your entire site does fewer than 5,000 monthly sessions, formal A/B testing usually isn’t the right place to spend your CRO effort. Focus on qualitative research and the seven-step process from our CRO framework guide.

Step-by-Step: Setting Up Your First A/B Test

For a team running A/B testing for the first time, the process below works well. Each step exists for a reason.

1. Choose What to Test (Based on Data, Not Opinion)

Your first test should be on a high-traffic page with a clear hypothesis backed by real data. Bad first test: “let’s try a different homepage hero image”. Good first test: “GA4 shows 60% of homepage visitors scroll past the primary CTA without engaging; let’s test moving the CTA to a sticky position in the header to lift click-through to /contact by 15-25%”.

The hypothesis should specify:

• What you’re changing (the variant)
• What outcome you expect (the metric you’ll measure)
• Why you expect it (data evidence)
• How large you expect the effect to be (so you know if it reached significance)

2. Pick the Primary Metric (One, Not Five)

Every A/B test should have ONE primary metric you’re measuring. Common choices:

• Conversion rate to purchase (for ecommerce)
• Conversion rate to form submission (for service businesses)
• Click-through to next step (when the change is mid-funnel)
• Add-to-cart rate (when testing product page changes)

Secondary metrics (revenue per visitor, average order value, bounce rate) are useful to watch but the primary metric is what defines win or loss. If you change the primary metric mid-test, you’ve invalidated the test.

3. Calculate Required Sample Size in Advance

Use a sample size calculator. VWO’s calculator and Evan Miller’s tool are both reliable. Inputs:

• Baseline conversion rate (your current rate on this page)
• Minimum detectable effect (how big a lift you want to be able to detect; usually 10-20%)
• Statistical significance threshold (usually 95%)
• Statistical power (usually 80%)

The output is the number of sessions per variant you need to detect the effect at the chosen confidence level. Write this number down. You will not call the test until each variant has reached this sample size.

4. Set Up the Test in Your Testing Tool

Tool options:

• VWO — strong all-rounder, fairly intuitive for non-developers
• Optimizely — enterprise-grade, more expensive, deep features
• Convert.com — privacy-focused alternative, simpler interface
• Shopify native split testing (limited but free for Shopify stores)
• Custom server-side testing (most rigorous but requires engineering work)

Configure the test with:

• 50/50 traffic split between A (control) and B (variant)
• Primary conversion event configured correctly (test fires when the actual conversion happens)
• Exclusions for known internal traffic, bot traffic, and segments that would confuse the result

5. QA the Test Before Launching

Look at the variant on multiple devices, multiple browsers, in incognito to confirm:

• The variant displays correctly
• The conversion event fires when expected
• There’s no broken JavaScript or layout shift
• Tracking is firing for the right users in the right segment

A test running on a broken variant produces wrong data for weeks.

6. Launch and Do Not Peek

This is the discipline part. Once the test is live, do not look at the interim results and make decisions. Statistical significance calculations assume you check the result once, at the end, after the sample size is reached. Peeking early and acting on early leads (“the variant is winning, ship it now”) is one of the most common mistakes in A/B testing and produces results that don’t replicate.

If you cannot resist peeking, set yourself a single midpoint check at 50% sample size for sanity (e.g. is the test still running, is tracking still working). Don’t act on the result.

7. Analyse the Result

Once sample size is reached, look at:

• Statistical significance (p-value). Usually expressed as confidence level. 95%+ is the standard threshold.
• Effect size. The actual lift percentage. A 0.5% lift at 95% confidence on a small sample is technically significant but practically noise.
• Secondary metrics. Did the variant lift the primary metric while hurting average order value or revenue per visitor? Sometimes a “winning” variant is a net loss.
• Segment performance. Did the variant work on mobile but lose on desktop? Useful for follow-up tests.

8. Decision and Documentation

Three possible outcomes:

• Variant wins. Implement permanently. Update your CRO log with what was tested and what you learned.
• Control wins. Keep the original. Update your log with what didn’t work and why your hypothesis was wrong.
• Inconclusive. The test didn’t reach significance, or the result was within the margin of error. Either run again with a longer window, scrap the hypothesis, or test a bigger change.

Documenting every test — including the losses — is what compounds your CRO knowledge over time.

What to Test First (For Most Businesses)

If you’re not sure where to start, the highest-leverage first tests for most businesses:

• Primary CTA copy and placement on the homepage and top landing pages
• Form length and field requirements on contact, quote, or signup forms
• Headline clarity on top landing pages (does the headline match the search intent that brought visitors here?)
• Social proof placement (testimonials, review ratings, trust badges, customer count)
• Pricing presentation on plan/quote pages
• Product page above-the-fold content on ecommerce stores (image gallery, title, price, CTA)

What NOT to test first: button colours, microcopy on low-traffic pages, broad layout changes that mix dozens of variables (use a separate redesign process for those).

Guardrail Metrics (What You DON’T Want to Harm)

Every A/B test should have one primary metric you’re trying to lift AND a set of guardrail metrics you’re protecting. A variant that lifts the primary metric but tanks a guardrail is a net loss, regardless of statistical significance.

Common guardrail metrics:

• Revenue per visitor (you might lift CVR by discounting, but RPV could fall)
• Average order value (a simpler product page might convert more visitors at lower order value)
• Customer support tickets (a removed form field might lift submissions but produce poorly-qualified leads that increase support load)
• Refund or return rate (a pushier checkout might lift purchases short-term but generate more refunds)
• Page load time (an A/B testing script added carelessly can slow the page)

Set guardrail thresholds before the test starts. If the variant lifts primary metric by 10% but breaches a guardrail by more than 5%, treat the test as inconclusive even if statistically significant.

A/A Testing: Validate Your Tool Before You Trust It

An A/A test serves the same variant to both groups and measures whether your testing tool reports a (false) “winner”. If it does, your tool setup is broken. The most common causes: incorrect audience randomisation, the conversion event firing inconsistently between variants, or the tool reporting results at different sample sizes for each side.

Run an A/A test for one week before launching your first real A/B test. The expected result is no statistically significant difference. If you get one, you have a configuration problem to fix before any real results can be trusted.

How Often A/B Tests Actually Win

Setting realistic expectations: Nielsen Norman Group’s research finds that only about one in seven A/B tests produces a winning variant. The other six either lose, are inconclusive, or produce wins that don’t replicate post-launch.

This is normal and not a sign your program is broken. The compounding value comes from the documented learnings across all those tests, not from picking only the wins. A test that fails still tells you something true about your audience.

Peeking Inflates False Positives

Stopping a test early because the variant looks like it’s winning is the single most damaging mistake in A/B testing. Convert.com’s research shows that peeking and stopping early can inflate the false positive rate from the standard 5% to as much as 30%.

In practice this means: if you peek and stop early, you’re three to six times more likely to declare a “winner” that doesn’t actually beat the control. The lift you ship evaporates within weeks of going permanent and you’ve burned a test cycle. The discipline of pre-deciding sample size and respecting it is what makes A/B testing trustworthy.

Segment Drilling Traps

After a test ends, the temptation is to slice the data by every available segment — device, source, country, day of week, returning vs new — looking for “where the variant really shined”. This is called the multiple comparisons problem and it inflates false positives the same way peeking does.

Each segment you check is another statistical test. If you check ten segments at 95% confidence each, you’d expect roughly one to look “significant” purely by chance. Segment findings should be treated as hypotheses for follow-up tests targeted specifically at that segment, not as actionable conclusions from the original test.

Holdout Groups (Different to A/B Testing)

For personalisation campaigns (different experiences for different audience segments), you don’t usually run an A/B test the way you would for a single change. Instead you set up a holdout group: a small percentage of your audience continues to see the default experience while everyone else gets the personalised one. The performance of the personalised experience is measured against the holdout group over a longer time window (often months).

This is the right mechanism for personalisation programs run via tools like Dynamic Yield, Mutiny or Optimizely Personalize. It’s not the same as a standard A/B test and the analytical method differs accordingly.

When A/B Testing Isn’t the Right Tool

Three situations where A/B testing is the wrong choice:

• Multi-element overhauls. A full page redesign mixes dozens of variables. A/B testing it tells you the new page is better or worse overall but not why. Use redesign processes (research, prototyping, usability testing) instead, then A/B test refinements after launch.
• Anything that doesn’t have a measurable outcome. Brand perception, customer satisfaction, long-term retention — these can be researched but not A/B tested in the traditional sense.
• One-off events. Black Friday banner copy, a single product launch announcement, a one-time campaign. The time window doesn’t allow for proper testing. Make the best call you can with pre-event research.

Triangulation: Combine Quant With Qual

The most reliable CRO programs combine quantitative A/B testing with qualitative research (session recordings, user interviews, surveys). The quantitative data tells you what is happening. The qualitative data tells you why. Each input alone is incomplete; together they produce sharper hypotheses and more reliable results.

The practical pattern: a session recording surfaces a friction point → an A/B test validates whether removing the friction lifts conversion → a follow-up survey confirms what users felt about the change. Each input checks the others.

Common A/B Testing Mistakes

The patterns I see most often on teams running A/B testing without a structured process:

• Peeking and stopping early. Variant looks like it’s winning after three days, you ship it, the lift evaporates. Pre-decided sample size protects you from this.
• Not running for a full business cycle. Conversion behaviour varies by day of week, time of month, and time of year. A two-week minimum is the floor for most businesses.
• Testing too small a change. “We tested two button colours and got no significant result.” The effect was probably too small to detect at your traffic volume. Test bigger changes.
• Multiple tests running on the same page simultaneously. Cross-test contamination. Run one test per page at a time.
• Changing the primary metric mid-test. Picks the metric that looks best for the result you want. Invalidates the test.
• Ignoring secondary metrics. A variant that lifts conversion rate but tanks average order value is a net loss. Always look at the revenue or lead-quality picture.
• Reading too much into segments. “The variant worked best for mobile users on weekends in Sydney” might be a real insight or might be random sample slicing. Treat segment findings as hypotheses for follow-up tests, not as conclusions.

Want Help Setting Up Your First A/B Test?

If you’ve decided you want to start A/B testing but aren’t sure where to begin or what your first test should be, we offer a free first-pass CRO audit including a recommended first-test hypothesis based on your data.

Request your free CRO audit →

Frequently Asked Questions

What is an A/B test?

An A/B test is a controlled experiment where two versions of a page or element (A = control, B = variant) are served to randomly assigned segments of your traffic simultaneously, and the conversion rates of each are compared to determine if the variant produced a statistically significant improvement. The randomisation and simultaneous serving are what make A/B testing more rigorous than before/after comparisons.

How long should an A/B test run?

Until it reaches the sample size calculated in advance, AND a minimum of two weeks (often four) to cover at least one full business cycle. Stopping early because the variant looks like it’s winning is one of the most common A/B testing mistakes and produces lifts that evaporate after launch.

How much traffic do I need for A/B testing?

For a typical test detecting a 10% relative lift on a baseline 3% conversion rate at 95% confidence, you need roughly 10,000-15,000 sessions per variant (20,000-30,000 total). Pages with lower traffic should focus on qualitative CRO (session recordings, heatmaps, user testing) rather than formal A/B testing.

What is statistical significance in A/B testing?

Statistical significance is the probability that the difference between your variants is real rather than random chance. The standard threshold is 95% confidence (p-value < 0.05). Significance doesn’t mean the lift is large or meaningful — it means you can be reasonably confident the lift isn’t noise. Always consider both significance AND effect size.

Should my first A/B test be on button colours?

Usually no. Button colour changes occasionally produce measurable lifts but they rarely have outsized impact compared to higher-leverage changes (CTA copy and placement, headline clarity, form length, social proof, page speed). Save button colour experiments until after you’ve optimised the higher-impact elements.

What is the difference between A/B testing and multivariate testing?

A/B testing compares two variants of a single change (e.g. headline A vs headline B). Multivariate testing tests multiple changes in combination (e.g. two headlines × two CTAs × two images = 8 variants). Multivariate requires significantly more traffic to reach significance because the sample splits across more variants. For most businesses, A/B testing produces faster, more reliable results.

Can I do A/B testing on Shopify?

Yes. Shopify has native split testing in the admin for theme content, and Shopify Plus stores have more advanced testing tools. For deeper experimentation across the funnel, third-party tools like VWO, Convert.com or Optimizely work on Shopify with proper installation. Note that some advanced changes (checkout, for example) are restricted on standard Shopify plans.