A/B testing services: an experiment planned in advance, with results you can read with confidence.

We wrote this A/B testing guide for marketing, product and ecommerce teams who want to know, by measuring it, whether a change they make on their site or app actually works. The rules we describe are based on documentation from Google Search Central, Google Analytics and Firebase, articles published by Microsoft’s Experimentation Platform team (ExP), Optimizely’s help center, and the statistics handbook of the US National Institute of Standards and Technology (NIST); the sources are listed by number at the bottom of the page. We verified product names and features against the source as of September 2026. Our own field observations are marked separately under the heading “Expert note”.

Our work on a test runs in three layers: design, which writes down the hypothesis, metrics and sample size before the test; implementation, which ships the variant without flicker and in line with search engine rules; and the decision, which reads the result with the p-value, the confidence interval and a sample ratio check.

What is A/B testing (split testing), and what does it prove?

A/B testing is a controlled experiment that shows two or more versions of a page or flow to randomly split visitors over the same period, and measures which version produces the target action at a higher rate. Google defines A/B testing as testing two or more variations of a change; its example is changing a button’s font and checking whether clicks go up.¹ An experiment that tests more than one type of change at the same time, looking both at the effect of each change and at possible interactions between them, is called a multivariate test.¹ The method, also known as split testing, draws its power from random assignment: because everything other than the change is similar on average between the two groups, the observed difference can be attributed to the change.

An A/B test proves the average difference between two versions in the period and audience measured; on its own, it doesn’t prove why the change worked. We distinguish test types like this:

How do you build a good test hypothesis and metric set?

A good hypothesis writes down a single change, its expected effect, and the metrics that will confirm or refute that effect, all before the test. Microsoft’s Experimentation Platform team recommends keeping the hypothesis clear and simple, and breaking a complex change into simple changes that each carry their own hypothesis.² According to the same article, a hypothesis must be provable or falsifiable with the chosen metrics; a change claimed to strengthen the brand is hard either to prove or to refute if there’s no metric measuring the brand’s impact.²

We split metrics into three groups. The primary metric is the single metric that makes the decision, usually a key event such as a purchase or a lead. Guardrail metrics show that the winner isn’t causing harm elsewhere: returns, error rate, page load time. Data quality metrics check that the measurement itself works the same way in both variants. Microsoft recommends computing a core set made up of user satisfaction, guardrail, feature and engagement, and data quality metrics in every experiment on a product, whatever feature it targets.² A hypothesis usually starts from an observation in heatmaps and session recordings; we cover that side on our Microsoft Clarity page.

How long should an A/B test run, and why is the sample fixed in advance?

Sample size and duration are fixed before the test because in a fixed horizon test, the result is valid only when the planned sample is complete and it’s read once. Optimizely writes that in this method the sample must be calculated before the test, and that looking at incomplete data and acting on it increases false positives and invalidates the experiment.³ The same document recommends running every test for at least one business cycle, meaning seven days, to cover behavior such as the difference between weekdays and weekends.³ According to Optimizely’s comparison of methods, sequential (Stats Engine) and Bayesian methods don’t carry this constraint: results can be monitored throughout the test, and the sample doesn’t need to be calculated in advance.¹³ Firebase gives two weeks as the recommended minimum for a typical Remote Config experiment; experiment data is processed for at most 90 days from the start.⁴

We calculate the visitors needed per variant from the baseline conversion rate, the smallest difference to be detected, the significance level and statistical power; we explain the formula, and the routes to take when traffic isn’t enough, on our CRO consulting page.

Making sure a visitor sees the same version throughout the test is also part of the design: in Firebase Remote Config experiments, assignment is based on the variant weights and on a hash generated from the experiment ID and the user’s Firebase installation ID; a user who enters an experiment stays in the same variant for as long as it runs.⁴ On the web, assignment usually relies on cookies; Microsoft writes that cookie-based IDs aren’t stable, that a user who clears their cookies may land in the other variant on their next visit, and recommends using these IDs only in short experiments.²

Should the variant be applied in the browser or on the server?

We apply copy, image and layout changes on the client side, and changes that touch business logic, such as pricing, checkout and search, on the server side. The main risk on the client side is the visitor briefly seeing the original page, in other words flicker. According to Optimizely, when the test script loads asynchronously the page doesn’t slow down, but the original page may appear first and the variant shortly after; this flicker makes the result less reliable and can hurt the visitor’s experience.⁵ That’s why Optimizely Web Experimentation uses a synchronously loaded snippet to prevent flicker.⁵ That has a cost too: the browser normally shows a blank page until synchronous resources in the head section have loaded.⁵

How are results read in GA4 now that Google Optimize has shut down?

Since Google Optimize shut down, web tests run in a third-party tool integrated with GA4; results can be read in GA4 through variant audiences and the exp_variant_string parameter of the experience_impression event. Optimize and Optimize 360 have been unavailable since September 30, 2023; Google announced that it was working on integrations with AB Tasty, Optimizely and VWO, and had opened up its APIs so anyone can connect their own testing tool to Google Analytics.⁶ When an experiment starts, GA4 creates an audience for each variant and deletes these audiences when the experiment ends; variant data, however, stays accessible in Explore and BigQuery through the exp_variant_string parameter.⁷

When an experiment ends, the data can be exported to BigQuery; statistical inference can be done in the testing tool or with a separate query in BigQuery.⁷ Firebase A/B Testing continues for apps; experiments started on or after October 23, 2023 don’t rely on Optimize infrastructure and use frequentist inference.⁶ Firebase A/B Testing has no separate BigQuery table; experiment and variant membership is written to every event as a user property in the Analytics event tables, and experiment data can be extracted in BigQuery using these properties.⁴

Does A/B testing affect Google Search rankings?

An A/B test that tries out small changes in line with the rules usually has little or no effect on Google Search rankings; Google writes that small changes, such as a button’s size or color or the call-to-action text, often have little or no impact on the search result snippet or ranking.¹ The risk lies in how the test is set up; the practices Google recommends during testing are:¹

• No cloaking: Googlebot isn’t shown one set of URLs while people see another; whether done with server logic or robots.txt, this violates spam policies and can lead to the site being demoted in rankings or removed from results.
• rel=”canonical”: in split URL tests, every variant address points to the original address as canonical; in this case, Google recommends a canonical link rather than noindex.
• 302 redirect: when redirecting from the original address to a variant, a temporary 302 is used, not a permanent 301; a JavaScript redirect is also acceptable.
• Cookies: Googlebot generally doesn’t support cookies; if the test is managed with cookies, Googlebot sees the version shown to browsers that don’t accept cookies.
• Duration: the test runs only as long as necessary; when it ends, the winning version is built into the site, and variant addresses, test scripts and markup are removed as soon as possible.

Google states that it may interpret an experiment that runs unnecessarily long, especially if one variant is shown to a large share of users, as an attempt to deceive search engines, and take action.¹

What does an A/A test validate?

An A/A test compares two identical versions to validate that the testing tool, random assignment and measurement are working correctly; since there’s no real change, the expected result is “no difference”. Optimizely also calls the A/A test a calibration test, and defines it as a data reliability and quality assurance step that validates the experiment configuration.⁸ According to the same document, most A/A results should come out statistically inconclusive; but significance is a probability, not a certainty, and a chance difference can appear even between identical versions.⁸

The size of that probability is set by the significance level. In Firebase’s definition, the p-value is the probability that a difference at least as large as the one observed would arise by chance alone when there’s actually no difference, and a p-value below 0.05 is considered significant.⁴ So in A/A tests run at the 0.05 level and read once at the planned sample, about 5% are expected to produce a “significant” result even when the setup is correct; we don’t treat a single significant A/A result as proof of an error, but as a signal to investigate.

How do you read significance and the confidence interval?

We use a two-proportion z-test to see whether the difference between two conversion rates can be explained by chance, and report the likely size of the difference with a confidence interval. The NIST/SEMATECH statistics handbook writes that when samples are large enough, the equality of two proportions can be tested with a z statistic built on the normal approximation to the binomial distribution:⁹

The handbook points to Fisher’s exact probability test as a good option for small samples.⁹ Firebase A/B Testing, too, uses a z-test for proportions on binary metrics such as conversion and a t-test with unequal variances on continuous metrics such as revenue, and calculates p-values for a one-tailed test in the direction of the variant being greater than the baseline.⁴ Optimizely’s fixed horizon method, on the other hand, tests the relative difference and calculates the p-value two-tailed.³ That’s why the two tools can show different p-values for the same data.

Firebase shows a 95% confidence interval for the difference; if the interval includes zero, no significant difference was detected between the variant and the baseline.⁴ The calculator on this page, too, gives the relative difference, the two-proportion z-test p-value and the 95% confidence interval from the visits and conversions of two variants. If several variants are compared with the same baseline, every comparison is a new chance for a false positive; Optimizely writes that this multiple comparisons problem calls for controlling false positive discoveries.³

How do you catch a sample ratio mismatch (SRM)?

A sample ratio mismatch (SRM) is a statistically significant deviation of the number of visitors assigned to each variant from the ratio set in the test; if there is one, its cause must be found before the result is read. At Microsoft, every A/B test has to pass this check before its effect is analyzed; according to an analysis cited in an article the company published in 2020, about 6% of A/B tests had an SRM.¹⁰ According to Microsoft, looking at the ratio alone isn’t enough, because the ratio doesn’t carry the sample size, and a statistical test such as chi-square is needed; the threshold it uses to reduce false alarms is p < 0.0005.¹⁰

Microsoft classifies the causes of SRM by experiment stage as follows:¹⁰

In an MSN test Microsoft describes, a variant that increased the number of cards in a carousel from 12 to 16 appeared to reduce engagement and triggered an SRM warning; a bot detection algorithm had removed users who engaged heavily with the variant from the analysis, and once the problem was fixed, the result reversed.¹⁰ The missing users are rarely a random group; more often, they’re the ones most affected by the change.¹⁰

How does the winner go live, and how do our A/B testing services start?

A winner goes live when three conditions are met together: a significant difference in the primary metric, guardrail metrics that haven’t deteriorated, and an acceptable possible loss at the lower end of the confidence interval. Firebase, too, because it identifies the leading variant based only on the primary goal, recommends looking at secondary metrics, the expected gain and downside risk such as the lower end of the confidence interval before rolling out.⁴ Before deciding, Microsoft checks whether the number of users in each variant matches the expected share of traffic, and whether the change left the expected trace in the metrics; for example, page size and load time changing accordingly for a variant that adds content to the page.¹² We don’t leave the winner in the testing tool; we build it into the site’s code and remove variant addresses and test scripts.¹

The first 30 days go in this order:

1. Week 1 · Measurement and tooling: metric events, the GA4 connection and the BigQuery export are audited; an A/A test starts if needed.
2. Week 2 · Hypothesis queue: hypothesis records are written, and sample size and duration are calculated; hypotheses that traffic can’t support are set aside.
3. Week 3 · Implementation: the first variant is built; flicker, speed and Google Search rules are checked before launch.
4. Week 4 · Launch and monitoring: the test goes live, and an SRM check is run in the first days; the result is read only once the planned sample and duration are complete.

What to look for in an A/B testing service

A testing tool makes it easy to show a variant; whether the decision is right depends on the design and reading discipline around the tool. Testing needs are usually met in one of three ways:

Before you start working with a testing partner, get written answers to four questions: which metric makes the decision, how is the sample calculated before the test, is an SRM check run, and how are losing tests reported? On top of that:

1. Hypothesis record: for every test, the observation, the single change, primary and guardrail metrics and the decision rule are written down before launch.
2. Checking the dashboard: the result can be confirmed by a second calculation outside the testing tool’s dashboard, such as a BigQuery query.
3. Search engine compliance: canonical links and temporary redirects for split URL tests are agreed before setup.
4. Account ownership: the testing tool, GA4 and BigQuery are opened in your name; the partner is an authorized user.
5. Guarantee language: a proposal saying “every test wins” is telling you inconclusive tests won’t be reported.

What determines A/B testing pricing?

The cost of A/B testing services depends less on how many tests you run than on how each test is built: changing a headline in the browser and splitting a checkout step on the server are different jobs.

We work out the scope together in the first meeting, then send a written proposal; any testing tool license is separate from our service fee.

A/B testing pricing: estimated ranges

Estimated range, excluding VAT; rows marked “/ month” are monthly, the others are one-off. Scope is confirmed in a written proposal after the first meeting. As of October 2026.

If you’d like to talk about which change on your site to test first, pick a meeting day that suits you below.