Free ebook on A/B testing statistics for product and marketing: metrics, confidence intervals, power, pitfalls, and decision-making.
Free ebook content
-
A/B Testing Essentials: Controlled Experiments for Product and Marketing Decisions
+ Exercise: Why is choosing the correct unit of randomization important in an A/B test?
-
Metrics for A/B Testing: Choosing Outcomes That Match Product and Marketing Goals
+ Exercise: In an A/B test, when should a variant be rejected even if it improves the primary metric?
-
Randomization and Experiment Integrity in A/B Testing
+ Exercise: In an A/B test, why is a 50/50 allocation ratio typically preferred when both variants are safe to expose?
-
Statistical Building Blocks for A/B Testing: Distributions, Variability, and Estimators
+ Exercise: In an A/B test, why should you treat an observed metric (like conversion rate or average revenue) as an estimate rather than the true value?
-
Interpreting Lift in A/B Testing: Absolute vs Relative Changes and Business Impact
+ Exercise: Why is it recommended to report both absolute lift and relative lift in an A/B test readout?
-
Confidence Intervals for A/B Testing: Quantifying Uncertainty in Differences
+ Exercise: In A/B testing, which interpretation best matches how a 95% confidence interval should be used for decisions about shipping a change?
-
Hypothesis Tests in A/B Testing: p-values, Error Rates, and Decision Rules
+ Exercise: Which decision best reflects how to combine statistical significance and practical importance when choosing whether to ship an A/B test variant?
-
Sample Size Intuition for A/B Testing: Power, Detectable Effects, and Runtime Expectations
+ Exercise: In A/B test planning, what is the key practical tradeoff when you choose a higher power target (e.g., 90% instead of 80%) while keeping the MDE and confidence level the same?
-
Variance Reduction Basics for A/B Testing: Getting Clearer Results Faster
+ Exercise: Which approach is NOT safe for variance reduction in an A/B test because it can bias the treatment effect estimate?
-
Common Traps in A/B Testing: Peeking, Optional Stopping, and Repeated Looks
+ Exercise: Why does stopping an A/B test the first time the p-value drops below 0.05 during daily checks increase the false-positive rate?
-
Novelty Effects, Seasonality, and Interference: When Results Don’t Generalize
+ Exercise: A/B test results show a large positive lift in the first few days that steadily decays toward zero (and may even turn negative). What is the most appropriate interpretation and next step?
-
Metric Misuse and Multiple Comparisons in A/B Testing: Avoiding False Discoveries
+ Exercise: In an A/B test, the primary metric is flat but a secondary metric shows a large “significant” lift after scanning many metrics. What is the most appropriate interpretation and next step?
-
From Results to Decisions: Communicating A/B Test Findings with Statistical Confidence
+ Exercise: An A/B test shows a positive point estimate that meets the minimum practical lift (MPL), but the 95% interval lower bound is below the MPL while guardrails are within limits. What decision best matches the decision framework described?
About the free ebook
A/B Testing Essentials: Statistics for Product and Marketing
This free ebook explains how to design, evaluate, and communicate controlled experiments for product and marketing decisions. It connects statistical reasoning with practical questions such as whether a new landing page, feature, email, or pricing message truly improves an outcome.
Make experiment results more trustworthy
Learn how to select metrics that reflect real goals, assign users fairly through randomization, and interpret observed differences without confusing noise for meaningful change. The ebook clarifies absolute lift, relative lift, variability, estimators, confidence intervals, p-values, error rates, statistical power, and detectable effects.
Recognize risks before making a decision
A/B test results can be misleading when teams peek at data early, stop tests opportunistically, compare too many metrics, or overlook novelty effects and seasonality. This guide shows why these issues matter and how disciplined experiment practices reduce false discoveries.
Turn statistics into action
Use a practical statistical foundation to assess uncertainty, weigh business impact, and present findings clearly to stakeholders. The focus is not simply on declaring a winner, but on making decisions that account for evidence quality, trade-offs, and whether results are likely to generalize.
What you will be able to evaluate
- Whether a metric is suitable as a primary experiment outcome
- How randomization protects comparisons between variants
- How confidence intervals and hypothesis tests support decisions
- Why sample size, power, and variance affect test runtime and clarity
- How to communicate results with appropriate statistical confidence
Designed for learners working with product analytics, digital marketing, conversion optimization, and applied statistics, this ebook builds a clear framework for more reliable A/B testing.
What is the difference between absolute lift and relative lift in an A/B test?
Absolute lift is the percentage-point difference between variants; relative lift is that difference divided by the control rate.
Why is peeking at A/B test results a problem?
Repeatedly checking results and stopping when they look significant increases the chance of a false positive.
How do confidence intervals help interpret an A/B test?
They show a plausible range for the true difference, helping assess both uncertainty and practical business impact.
This ebook includes:
13 content chapters
Digital certificate of course completion (Free)
Exercises to train your knowledge
100% free, from content to certificate
Ready to get started?
recommended
In the app you will also find...
Over 5,000 free courses
Programming, English, Digital Marketing and much more! Learn whatever you want, for free.
Study plan with AI
Our app's Artificial Intelligence can create a study schedule for the course you choose.
From zero to professional success
Improve your resume with our free Certificate and then use our Artificial Intelligence to find your dream job.
You can also use the QR Code or the links below.
