A/B Testing When Your Website Gets Little Traffic

Short answer: On a site with a few thousand visitors a month, a classic A/B test on a small change usually can't reach a reliable answer in any reasonable time. Instead, watch five real people try to use your site, review session recordings, and make bold, clearly better changes that you check with a careful before-and-after comparison. Save true A/B tests for big differences on your highest-traffic page.

Why A/B tests need so much traffic

An A/B test splits visitors between two versions and waits until the difference in results is unlikely to be luck. How many visitors that takes depends on three things:

Your current conversion rate. Rare actions (an enquiry from 2% of visitors) need more traffic than common ones.
The smallest improvement you care about. Testing tools call this the minimum detectable effect. Detecting a small lift takes far more visitors than detecting a big one.
How sure you want to be. Most tools default to 95% statistical significance.

Optimizely's own documentation gives a worked example: with a 15% baseline conversion rate, 95% significance and a 10% minimum lift, its sample size calculator suggests about 8,000 visitors per variation. It then divides total sample size by weekly unique visitors to estimate how many weeks a test will run.

Apply that same arithmetic to a hypothetical small business site with 1,500 visitors a month, roughly 350 a week. Two variations at 8,000 visitors each is 16,000 visitors, or about 46 weeks. And that's with a 15% baseline, which is generous; a contact form converting at a few percent would need more. By the time the test finishes, the season, your prices and your ads have all changed.

That's why "just A/B test it" is often bad advice for small businesses. And a test stopped early, before it reaches its sample size, can easily crown a "winner" that is really just noise.

A note on tools

Google Optimize, which many small businesses used for free, is gone. Google's help page says Optimize and Optimize 360 were no longer available from September 30, 2023, and that Google is working on Analytics integrations with third-party testing providers, naming AB Tasty, Optimizely and VWO. Free and paid alternatives exist, but the tool doesn't fix the traffic problem.

What to do instead: a decision table

Your situation	Best method	What you learn	Time needed
You don't know why people leave without enquiring	Usability test with 5 people	Where people get confused or stuck	A few days
You suspect a specific page or form is the problem	Session recordings and click maps on that page	What people actually do, scroll and skip	2–4 weeks of recordings
You have a clearly better version (shorter form, clearer pricing, real photos)	Make the change, then compare before and after	Whether the big change moved the needle	4–8 weeks either side
You have a high-traffic page and a bold idea	A real A/B test, with a large expected difference	A trustworthy answer on one big question	Use a sample size calculator first
You want to compare two headlines or ad angles	Test them in ads, not on the site	Which message gets clicks	1–2 weeks of ad spend
You're about to rebuild anyway	Test the prototype with 5 people before building	Problems found when they're cheap to fix	A week

Why five people is enough to start

Jakob Nielsen's Nielsen Norman Group article on testing with five users argues that elaborate usability studies waste resources, and that small, repeated tests work best. It suggests running three studies of five users rather than one big one, fixing what you find between rounds. Five people trying to "get a quote for a new water heater" on your site will show you most of the obvious blockers.

Before-and-after, done carefully

Comparing a month before a change with a month after is less rigorous than an A/B test, but it can be honest if you:

change one significant thing at a time,
compare the same days of the week and similar seasons,
watch for other changes (ad spend, a new review, a holiday),
judge on enquiries or sales, not page views,
write down what you expect before you look at the numbers.

If the difference is small or muddled, treat it as "no clear effect," not as a win.

A worked plan for one month

A hypothetical HVAC company gets about 1,500 visits a month and a handful of quote requests. The owner suspects the quote form is the problem.

Week 1: Watch. Turn on session recordings for the quote page. Ask five people (friends of friends who haven't seen the site) to "find out what a furnace tune-up costs and ask for a visit," on their phones, while talking out loud. Note every hesitation.

Week 2: List and rank. Group the problems. Typical finds: the form asks for too many fields, the "Submit" button sits below a large image, prices are buried in a PDF.

Week 3: Make one bold change. Rebuild the form with fewer fields, move it higher on the page, and put a starting price on the page. Record the change date.

Week 4 onward: Compare. After four to eight weeks, compare quote requests per visit with the same length of time before the change. Run another round of five-person tests on the new version to check the fixes worked, as the NN/g article recommends.

If you do run an A/B test, don't trip over SEO

Google's guidance on website testing says not to cloak (showing Googlebot different content from users), to use rel="canonical" on variant URLs pointing to the original, to use 302 rather than 301 redirects for test URLs, and to remove test elements once the experiment ends.

FAQ

Can I A/B test with a few hundred visitors a month?

You can run the software, but you're unlikely to get a reliable answer unless the difference between versions is enormous. Put the effort into user testing and bigger improvements instead.

Is a 90% significance level good enough for a small site?

It shortens tests but raises the chance of acting on a false winner. If you lower the bar, also aim to test bigger changes and accept that some results won't hold up.

Do session recordings slow down my site?

Recording scripts add code. Run them on the pages you're investigating, for a limited period, rather than everywhere forever.

If you'd like five-person tests and a before-and-after plan set up around your busiest page, send us the page you're worried about.