A/B Testing Statistical Analysis -- Complete Guide to Experiment Design & Evaluation

In this tutorial, you'll learn about A/B Testing Statistical Analysis. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

A/B testing statistically compares two or more variants to determine which performs better on a defined metric, providing a scientific basis for product and marketing decisions rather than intuition.

What You'll Learn

In this tutorial, you will learn how to design A/B tests with proper sample size calculations, analyze results using frequentist and Bayesian methods, implement sequential testing for faster decisions, avoid common statistical pitfalls like peeking and multiple comparisons, and communicate experiment results to stakeholders with confidence intervals.

Why It Matters

Companies running rigorous A/B testing programs make better decisions faster. Amazon runs thousands of experiments annually, with each successful experiment improving conversion by measurable fractions that compound to billions in revenue. Without proper statistical analysis, teams make decisions based on noise, implement changes that actually hurt metrics, and waste engineering resources on pointless iterations. A single statistically invalid test that leads to a bad product decision can cost months of development time.

Real-World Use

Doda Browser ran an A/B test comparing two onboarding flows. The control showed a 14% activation rate. The variant showed 16%. A naive interpretation declared the variant the winner. Statistical analysis revealed the result was not significant (p = 0.23) and the sample size was insufficient to detect a 2% lift. Running the test for two more weeks with the correct sample size showed the variant actually performed 1% worse. The team saved months of work on a bad feature by waiting for statistical significance.

A/B Testing Statistical Framework

flowchart TD
    A[Define Hypothesis] --> B[Choose Primary Metric]
    B --> C[Calculate Sample Size]
    C --> D[Randomize Users]
    D --> E[Run Experiment]
    E --> F[Collect Data]
    F --> G[Check for Peeking]
    G --> H[Run Statistical Test]
    H --> I{Significant?}
    I -->|Yes| J[Estimate Effect Size]
    I -->|No| K[Power Analysis]
    K --> L{Increase Sample?}
    L -->|Yes| D
    L -->|No| M[Conclude No Effect]
    J --> N[Make Decision]
    M --> N

Sample Size Calculation

Calculate the minimum sample size required before starting an experiment:

import math
from scipy import stats

def minimum_sample_size(baseline_rate, minimum_detectable_effect,
                        alpha=0.05, beta=0.20):
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(1 - beta)

    p1 = baseline_rate
    p2 = baseline_rate * (1 + minimum_detectable_effect)
    p_pooled = (p1 + p2) / 2

    n = (
        (z_alpha * math.sqrt(2 * p_pooled * (1 - p_pooled)) +
         z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    ) / ((p2 - p1) ** 2)

    return math.ceil(n)

baseline = 0.08  # 8% conversion rate
mde = 0.10       # 10% relative improvement (detect 8% -> 8.8%)

n_per_variant = minimum_sample_size(baseline, mde)
print(f"Required sample per variant: {n_per_variant}")
print(f"Total sample (2 variants): {n_per_variant * 2}")
print(f"At 10,000 daily visitors, run for: {math.ceil(n_per_variant * 2 / 10000)} days")

Expected output: For an 8% baseline with 10% MDE, typical sample sizes range from 50,000 to 80,000 per variant. At 10,000 daily visitors, the test runs 10-16 days. Running for fewer days risks drawing conclusions from insufficient data.

Frequentist Hypothesis Testing

Analyze experiment results using a two-proportion z-test:

import numpy as np
from scipy import stats

def analyze_ab_test(control_visitors, control_conversions,
                    variant_visitors, variant_conversions):
    control_rate = control_conversions / control_visitors
    variant_rate = variant_conversions / variant_visitors
    relative_lift = (variant_rate - control_rate) / control_rate

    p_pooled = (control_conversions + variant_conversions) / (
        control_visitors + variant_visitors
    )
    se = math.sqrt(p_pooled * (1 - p_pooled) * (
        1 / control_visitors + 1 / variant_visitors
    ))
    z_stat = (variant_rate - control_rate) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z_stat)))

    z_critical = stats.norm.ppf(0.975)
    margin = z_critical * se
    ci_lower = (variant_rate - control_rate) - margin
    ci_upper = (variant_rate - control_rate) + margin

    return {
        "control_rate": round(control_rate * 100, 2),
        "variant_rate": round(variant_rate * 100, 2),
        "absolute_difference_pp": round((variant_rate - control_rate) * 100, 2),
        "relative_lift_pct": round(relative_lift * 100, 2),
        "z_statistic": round(z_stat, 3),
        "p_value": round(p_value, 4),
        "significant_at_95": p_value < 0.05,
        "confidence_interval_95": (
            round(ci_lower * 100, 2),
            round(ci_upper * 100, 2),
        ),
    }

result = analyze_ab_test(
    control_visitors=52000,
    control_conversions=4160,
    variant_visitors=51800,
    variant_conversions=4455,
)

for key, value in result.items():
    print(f"{key}: {value}")

Expected output: The analysis shows a 1.28 percentage point absolute difference with a 3.5% relative lift. If p < 0.05, the result is statistically significant. The 95% confidence interval shows the range of plausible true effect sizes. A confidence interval crossing zero means the result is not significant.

Bayesian A/B Testing

Bayesian methods provide intuitive probability-based interpretation:

import numpy as np
from scipy.stats import beta

def bayesian_ab_test(control_visitors, control_conversions,
                     variant_visitors, variant_conversions,
                     alpha_prior=1, beta_prior=1):
    control_posterior = beta(alpha_prior + control_conversions,
                              beta_prior + control_visitors - control_conversions)
    variant_posterior = beta(alpha_prior + variant_conversions,
                              beta_prior + variant_visitors - variant_conversions)

    samples_control = control_posterior.rvs(100000)
    samples_variant = variant_posterior.rvs(100000)

    prob_variant_better = np.mean(samples_variant > samples_control)

    expected_loss_control = np.mean(np.maximum(0, samples_variant - samples_control))
    expected_loss_variant = np.mean(np.maximum(0, samples_control - samples_variant))

    lift_samples = (samples_variant - samples_control) / samples_control
    lift_quantiles = np.percentile(lift_samples, [2.5, 25, 50, 75, 97.5])

    return {
        "probability_variant_is_better": round(prob_variant_better * 100, 1),
        "expected_loss_if_choose_control": round(expected_loss_control * 100, 3),
        "expected_loss_if_choose_variant": round(expected_loss_variant * 100, 3),
        "median_lift_pct": round(lift_quantiles[2] * 100, 2),
        "lift_95_credible_interval": (
            round(lift_quantiles[0] * 100, 2),
            round(lift_quantiles[4] * 100, 2),
        ),
    }

bayesian_result = bayesian_ab_test(52000, 4160, 51800, 4455)
for key, value in bayesian_result.items():
    print(f"{key}: {value}")

Expected output: The Bayesian analysis shows a probability that the variant is better (e.g., 97.3%), expected loss for choosing wrong, and a credible interval for the lift. Unlike frequentist p-values, this directly answers the question "how likely is the variant to be better?"

Sequential Testing with Alpha Spending

Avoid the peeking problem by using sequential testing frameworks:

import numpy as np
from scipy.stats import norm

class SequentialABTest:
    def __init__(self, baseline_rate, minimum_effect, alpha=0.05):
        self.baseline_rate = baseline_rate
        self.minimum_effect = minimum_effect
        self.alpha = alpha
        self.control_visitors = 0
        self.control_conversions = 0
        self.variant_visitors = 0
        self.variant_conversions = 0

    def add_observation(self, is_control, converted):
        if is_control:
            self.control_visitors += 1
            if converted:
                self.control_conversions += 1
        else:
            self.variant_visitors += 1
            if converted:
                self.variant_conversions += 1

    def compute_sequential_boundary(self, n):
        return norm.ppf(1 - self.alpha / (2 * np.log(np.e + n / 1000)))

    def check_significance(self):
        n_total = self.control_visitors + self.variant_visitors

        if n_total < 100:
            return False, "Insufficient data"

        boundary = self.compute_sequential_boundary(n_total)

        control_rate = self.control_conversions / max(self.control_visitors, 1)
        variant_rate = self.variant_conversions / max(self.variant_visitors, 1)

        p_pooled = (self.control_conversions + self.variant_conversions) / n_total
        se = np.sqrt(p_pooled * (1 - p_pooled) * (
            1 / max(self.control_visitors, 1) + 1 / max(self.variant_visitors, 1)
        ))

        z_stat = (variant_rate - control_rate) / max(se, 0.0001)

        is_significant = abs(z_stat) > boundary

        return is_significant, {
            "z_statistic": round(z_stat, 3),
            "boundary": round(boundary, 3),
            "total_observations": n_total,
        }

test = SequentialABTest(baseline_rate=0.08, minimum_effect=0.01)

np.random.seed(42)
for i in range(50):
    for _ in range(1000):
        test.add_observation(is_control=True,
                             converted=np.random.random() < 0.08)
        test.add_observation(is_variant=True,
                             converted=np.random.random() < 0.088)

    significant, details = test.check_significance()
    if significant:
        print(f"Stopped at observation batch {i+1}: {details}")
        break
else:
    print(f"Completed monitoring, final: {details}")

Expected output: The sequential test may stop early (after 10-20 batches of 1000 each) if the effect is strong, or run to completion if the effect is small. The alpha-spending boundary adjusts for multiple looks, maintaining statistical validity.

Tool Comparison

Common Errors

1. Peeking at Results and Stopping Early

Checking results daily and stopping as soon as p < 0.05 dramatically inflates false positive rates. A test that peeks 10 times has a 40% chance of seeing a false positive during the monitoring period. Use sequential testing or commit to a fixed sample size before starting.

2. Multiple Comparison Confounding

Testing 10 metrics and declaring victory when one shows significance at p < 0.05 ignores the multiple comparisons problem. Apply Bonferroni correction (divide alpha by number of metrics) or use False Discovery Rate methods.

3. Sample Ratio Mismatch (SRM)

Expecting 50/50 traffic split but observing 48/52 indicates a technical implementation bug. Run a chi-squared test on the sample ratio before analyzing results. SRM often correlates with biased assignment that invalidates the entire experiment.

4. Novelty Effect and Primacy Bias

Users may respond differently to a change because it is new (novelty effect) or because they are accustomed to the old version (primacy bias). Run tests for at least two full weeks to let these effects stabilize.

5. Metric Sensitivity Mismatch

Choosing a metric that is too insensitive (e.g., overall revenue when testing a button color change) or too sensitive (e.g., click-through rate on an element users barely see) leads to either false negatives or misleading positives. Select metrics at the right level of the user journey.

Practice Questions

1. What is statistical power and why does it matter for A/B testing? Statistical power is the probability of detecting a true effect when one exists (typically set at 80%). Low power means your test is likely to miss real improvements. Power depends on sample size, effect size, and significance level.

2. What is the peeking problem and how do you solve it? Peeking means checking results during the experiment and stopping early when significance is reached. It inflates false positive rates. Solutions include fixing a sample size in advance, using sequential testing with alpha-spending boundaries, or applying Bayesian methods with stopping rules.

3. How does Bayesian A/B testing differ from frequentist? Bayesian methods produce a probability that one variant is better than another, directly answering the question stakeholders care about. Frequentist methods produce a p-value that answers "how likely are these data if there is no real difference?" Bayesian also naturally incorporates prior information.

4. What is Sample Ratio Mismatch and how do you detect it? SRM occurs when the actual traffic split differs significantly from the expected split (e.g., 48/52 instead of 50/50). Detect it with a chi-squared goodness-of-fit test during the experiment. SRM often indicates a bug in randomization logic.

5. Challenge: Design and analyze a full A/B test comparing two checkout page designs. Calculate the required sample size for an 8% conversion baseline with 5% MDE. Run the experiment with proper randomization. Analyze results using both frequentist z-test and Bayesian beta-binomial model. Check for SRM, novelty effects, and multiple comparison issues. Write a one-page stakeholder summary with the recommendation.

Mini Project

Build a complete A/B testing analysis framework in Python. Implement sample size calculation, frequentist two-proportion z-test with confidence intervals, Bayesian beta-binomial model with posterior simulation, sequential testing with alpha-spending boundaries, and SRM detection with chi-squared testing. Create a reporting function that generates a one-page experiment summary including all key statistics, visualizations of posterior distributions, and a clear recommendation. Test the framework on synthetic data where the true effect is known, verifying that the framework correctly identifies significant and non-significant results.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.