A/B Tests on a Business Website: Who Actually Enters the Result, and What Counts as Success?

Before commissioning an A/B test, write down the decision it will inform. For example, a service business might want to compare two enquiry journeys by completed, usable enquiries rather than by clicks on the first button. Showing different pages is only the delivery mechanism. The experiment also needs an agreed population, outcome and measurement window.

Otherwise a dashboard can look precise while answering a question nobody intended to ask. This guide concerns specifying an experiment's analysis contract, not choosing which customers receive a gradual feature rollout or promising that a redesign will improve sales.

Separate receiving a variant from entering the analysis

A visitor can be assigned a variant before reaching the part of the site that differs. If the test changes a later step in an enquiry form, an early page load is not necessarily evidence that the visitor encountered that change.

PostHog provides a concrete example. Its experiment-creation documentation says the default exposure event is $feature_flag_called. It also supports a custom event to narrow exposure to people who reached the relevant part of the product. The choice affects the population being analysed.

Ask the developer to demonstrate the intended entry point. What does the event prove, and where is it emitted? A flag being evaluated is not automatically proof that a person read a page. Conversely, selecting an event that only happens after the desired outcome could exclude the unsuccessful journeys you meant to measure. Have a competent analyst review the design rather than selecting whichever denominator gives the most attractive result.

Define the business outcome before inspecting results

Write a plain-language primary question and translate it into an event definition. In the fictional enquiry example, decide whether success means submitting the form, passing agreed validation or a later reviewed qualification. Those are different outcomes. Tracking a submit-button click should not silently become evidence of a usable enquiry.

PostHog distinguishes primary metrics, which describe the main goal, from secondary metrics, which provide context and help monitor side effects. Its documentation recommends defining metrics upfront to avoid biasing the analysis. Merely assigning the label “primary” does not repair an unsuitable measurement.

Agree relevant checks alongside the main outcome, such as whether the changed journey produces more invalid submissions or technical failures. Keep the set purposeful. Do not search an expanding list of unrelated figures for something favourable and rename that as the original success criterion.

Give the outcome a deliberate time boundary

An outcome needs a defined relationship to exposure. PostHog's metrics documentation says only events after exposure enter experiment calculations. It also describes conversion windows that limit which later events count.

There is an important operational distinction: with an explicit conversion window, PostHog anchors it to the first exposure, and later exposures do not restart or extend it. The documented window can also extend beyond the experiment's end date for people exposed near the end.

Ask your analyst to choose a window appropriate to the customer journey, and record that choice in the brief. Ending new assignments is therefore not necessarily the same as having a mature final outcome for every participant. The app should show the applicable analysis dates and any still-developing observation period instead of simply marking the dashboard final when the test is switched off.

Preserve exclusions and later changes

PostHog's creation guide describes handling people exposed to multiple variants: the default is exclusion, with an alternative of using the first variant. It also documents excluding internal users through the project's test-account filters. These are analysis choices, not an instruction that every business must use one universal rule.

Decide the rule with the analyst and keep it explainable. Staff testing the site should not accidentally become ordinary customer evidence. An identity change or unexpected variant history should produce a traceable treatment, not an invisible adjustment to the totals.

The same guide says metrics and inclusion criteria can be refined after saving, without changing what users see or what data is collected. That flexibility is useful, but a later analysis is not automatically the originally planned analysis. Ask for a retained configuration history and a clear label when results were recomputed under changed criteria.

Test the measurement before trusting the comparison

Use synthetic journeys in an isolated setup to demonstrate:

• Someone is assigned a variant but never reaches the agreed exposure point.
• Someone reaches that point but does not complete the outcome.
• An outcome occurs before exposure or outside the agreed window.
• A participant encounters more than one variant.
• An internal test account is treated according to the chosen exclusion rule.
• Inclusion criteria change and the displayed result identifies the new analysis version.

The expected records should come from the agreed measurement design. Passing these checks demonstrates implementation behaviour, not sufficient sample size, statistical validity or a winning variant. Have the analyst define when evidence is adequate and allow an inconclusive answer.

For a website or web-app project, bring the decision, exposure point, outcome, time window and review owner together. That is a more useful starting point than asking for a testing tool and a green winner badge.

Official sources and scope

PostHog: Creating an experiment and Experiment metrics. Product-specific descriptions above refer to those documents. Configuration history, acceptance journeys and business review arrangements are proposed requirements, not claims that a tool automatically supplies a valid study. Privacy, consent and production-data collection need their own appropriate review before launch.