Learning path · Level 6 · Tutorial 4

Design and Interpret a Simple A/B Test

Write a practical A/B test brief and interpret simulated results without running an experiment or using paid testing software. Distinguish an observed improvement from evidence strong enough to justify a decision.

  • Level: Intermediate
  • Duration: 25 min
  • 8 steps
Start tutorial Back to the path
Design and Interpret a Simple A/B Test

What you will go through

  1. Turn a Conversion Barrier into a Testable Hypothesis Turn simulated usability evidence into a focused A/B test hypothesis without claiming a certain result. 3 min
  2. Make the Comparison Fair Define eligibility and stable, concurrent visitor assignment so that Trailside’s control A and variation B can be compared fairly. 3 min
  3. Choose the Primary Metric and Guardrails Define Trailside's purchase conversion metric and two guardrails before examining simulated results. 3 min
  4. Set the Stopping Plan Before Results Arrive Predefine when Trailside enrolls, observes, analyzes, and pauses so results do not change the rules. 4 min
  5. Calculate Rates, Differences, and Uplift Calculate the simulated purchase conversion rates, percentage-point difference, and relative uplift while keeping descriptive arithmetic separate from causal claims. 4 min
  6. Recognize an Observed Lead Without Declaring a Winner Interpret simulated A/B test results carefully by comparing sample sizes, sensitivity to individual outcomes, and the limits of descriptive rates. 3 min
  7. Check Validity and Choose a Cautious Next Action Diagnose invalid evidence, safety concerns, and valid but inconclusive simulated results before choosing a cautious next action. 3 min
  8. Complete the Test Brief and Decision Note Integrate a predeclared A/B test plan with simulated final results, then write a cautious decision note that separates observed improvement from evidence still needed. 3 min

What you will learn

  • Turn a prioritized conversion barrier into a hypothesis with a control and one focused variation.
  • Define visitor assignment, a primary metric, guardrails, and stopping criteria before examining results.
  • Calculate each variation's conversion rate, the percentage-point difference, and relative uplift from a result table.
  • Identify why small samples, premature stopping, or measurement problems can make an apparent improvement inconclusive.
  • Recommend a cautious next action without claiming statistical significance from conversion rates alone.

Before you start

  • Prioritize Conversion Improvements from Evidence
  • Set Marketing Objectives and Calculate Core KPIs
  • Read GA4 Acquisition, Engagement, and Key-Event Reports

Step 1 of 8

Turn a Conversion Barrier into a Testable Hypothesis

Turn simulated usability evidence into a focused A/B test hypothesis without claiming a certain result.

Compare versions, not time periods

A concurrent comparison

A/B testing compares two versions at the same time: control A is the current experience, and variation B contains one deliberate change. The purpose is a fair comparison, not a before-and-after story that could be affected by a different day, audience, or traffic mix.

One focused difference

Two otherwise identical mobile cart screens differ only in their checkout button label.

Control A and variation B should differ in one focused element, while they are viewed during the same period.

From barrier to focused hypothesis

Evidence suggests a question

Trailside has simulated evidence: 3 of 5 observed mobile shoppers hesitated because the cart button's next step was unclear. This observation is a reason to test a clearer label. It is not proof that a new label will increase purchases.

Build a testable hypothesis

A useful hypothesis has four parts:

  1. Evidence: the observed barrier.
  2. Change: one specific difference in variation B.
  3. Expected outcome: a purchase outcome stated as a possibility, not a certainty.
  4. Reason: why the change could reduce the barrier.

Keep the test focused. If button copy, layout, pricing, and checkout behavior all change together, you cannot tell which change may be related to an outcome.

Example

Worked example

Evidence and hypothesis — simulated case

Observed evidence: 3 of 5 observed mobile shoppers hesitated because the cart button's next step was unclear.

Control A keeps the label Continue. Variation B changes only that label to Proceed to checkout; no other copy, layout, pricing, or checkout behavior changes.

Hypothesis: Because some observed mobile shoppers hesitated at an unclear next step, changing only the cart button label from “Continue” to “Proceed to checkout” may reduce that uncertainty and may increase completed purchases.

Tip

Use calibrated language

Do not treat the hypothesis as a prediction guaranteed to come true. It states what the change is expected to do and why; later evidence is needed to evaluate the expectation.

Practice: state the test clearly

Write the Trailside hypothesis

Using the simulated Trailside case, identify control A and the single element changed in variation B. Then write one hypothesis that includes: the observed evidence, the proposed change, an expected purchase outcome, and a reason. Phrase the outcome as a possibility, not a certainty.

Write at least 180 characters (0/180).

Step 2 of 8

Make the Comparison Fair

Define eligibility and stable, concurrent visitor assignment so that Trailside’s control A and variation B can be compared fairly.

Set assignment rules before outcomes

Predeclare who enters and how they are assigned

Set comparison rules before outcomes appear. For Trailside, eligible visitors are mobile cart visitors. Staff and known bots are excluded using rules fixed before assignment. Apply those same eligibility rules to control A and variation B; do not add or remove visitor groups after seeing outcomes.

Randomly assign each eligible visitor to control A or variation B with a planned 50/50 allocation. This is a target allocation, not a promise that final counts will be exactly equal.

Use one stable, permitted visitor identifier to assign a version consistently. A returning visitor keeps the original version rather than being reassigned on every pageview. The identifier supports consistency, but it does not reliably identify one person across devices.

Run both versions concurrently. Do not deliberately send one version a different traffic source or device population.

One consistent comparison flow

Diagram showing pre-set eligibility, exclusion of staff and bots, random split into two versions, and returning visitors staying in their original branch.

Apply the same entry rules first, then assign each eligible visitor once and preserve that assignment on return visits.

Tip

Fair comparison principle

Fair does not mean that the two groups contain identical people or exactly equal final totals. It means both versions use the same predeclared eligibility rules and receive visitors concurrently through random assignment, without deliberately separating traffic sources or device populations.

Check the comparison design

Connect the rule to its purpose

Match each design choice to its comparison consequence.

Tap an item and then its matching pair.

Apply the fairness rules

Assess an assignment plan

Trailside is considering four assignment proposals:

  1. Show control A on Monday and variation B on Tuesday.
  2. Randomly allocate every pageview, even for people who return.
  3. Send paid-ad mobile traffic to variation B and email mobile traffic to control A.
  4. Randomly assign each eligible mobile cart visitor 50/50 with one stable, permitted identifier.

In 3–5 sentences, identify which proposal is valid. Explain why the other three do not support a fair comparison, and state exactly how a returning visitor should be handled.

Write at least 180 characters (0/180).

Step 3 of 8

Choose the Primary Metric and Guardrails

Define Trailside's purchase conversion metric and two guardrails before examining simulated results.

Define the primary metric before results

One outcome measure

The hypothesis concerns confirmed purchases, so Trailside should predefine one primary metric that represents that outcome—not select whichever number later looks most favorable.

Use this simulated-case definition:

purchase conversion rate = purchasing visitors / eligible assigned visitors × 100

A purchasing visitor is an eligible assigned visitor with at least one confirmed purchase within 7 days of first assignment. Count each visitor once, even if they return or place multiple orders.

Count visitors, not events

Diagram showing each eligible assigned visitor counted once, and counted as purchasing only when at least one confirmed purchase occurs within seven days of first assignment.

The counting unit is the visitor. A return visit or an additional order does not create another visitor count.

What belongs in the metric

Use the same definition for control A and variation B.

Purchase conversion rate counting rules for the simulated Trailside case.
PartIncludeDo not substitute
NumeratorEligible assigned visitors with at least one confirmed purchase within 7 daysButton clicks, sessions, or total orders
DenominatorAll eligible assigned visitorsOnly visitors who clicked or reached checkout
Counting ruleEach visitor onceOne count per visit or per order

Use guardrails to detect possible harm

Add harm checks

An observed improvement in the primary metric could conceal a harmful side effect. Guardrails are predefined checks for those effects.

For Trailside, measure both guardrails over the same 7-day window, using affected eligible assigned visitors divided by all eligible assigned visitors. Apply the same event definitions and counting rules to control A and variation B.

Trailside guardrails

These alert thresholds are simulated-case rules. They indicate when to investigate; they do not prove that smaller changes are harmless.

Compare each rate with control A using the predefined alert threshold.
GuardrailAffected visitorsAlert threshold versus control A
checkout error rateEligible assigned visitors with a checkout errorIncrease of at least +1.0 pp
checkout-related support-contact rateEligible assigned visitors with a checkout-related support contactIncrease of at least +0.5 pp

Practice: complete the measurement plan

Classify the metric and checks

Complete the simulated Trailside metric specification. Use the exact counting roles taught in this step.

yaml
purchase_conversion_rate:
  numerator: ""
  denominator: ""
  observation_window: " from first assignment"

cart_button_click: ""
checkout_error: ""
checkout_related_support_contact: ""

Step 4 of 8

Set the Stopping Plan Before Results Arrive

Predefine when Trailside enrolls, observes, analyzes, and pauses so results do not change the rules.

Set a fixed enrollment and observation plan

Predeclare the timeline

A stopping plan is written before results are examined. It states when Trailside stops enrolling visitors, when it can analyze outcomes, and what exceptions justify action during the test.

For this simulated case:

  • Enroll eligible mobile cart visitors for 14 consecutive days.
  • Use a predeclared operational exposure requirement of 1,000 eligible visitors per version.
  • Let every enrolled visitor complete the existing 7-day follow-up window.
  • Analyze only after that final follow-up ends.

Writing these rules first prevents a temporary result from deciding when the test ends.

Enrollment is not analysis

A timeline shows 14 enrollment days followed by a separate 7-day follow-up period for the final enrolled visitors, then analysis.

Trailside enrolls for 14 consecutive days. The analysis date comes after the final enrolled visitor has had 7 days of follow-up.

Example

Example

Example stopping-plan wording for the simulated Trailside case:

“Enroll eligible visitors for 14 consecutive days using the planned 50/50 allocation. Require 1,000 eligible visitors per version as an operational exposure requirement. After enrollment ends, wait until every enrolled visitor has had 7 days of follow-up before final analysis. Do not end early for a numerical lead or extend enrollment after the deadline to seek more favorable results.”

Handle exceptions without rewriting the plan

What the exposure rule does—and does not—mean

The 14-day period covers two weekly cycles in this simulated case. That can reduce the chance that a single weekday pattern dominates the comparison, but business cycles vary. It is a case choice, not a universal duration.

The requirement of 1,000 eligible visitors per version is also an operational case assumption. It does not, by itself, show that the comparison has enough evidence for a reliable decision. Real planning depends on factors such as the baseline rate and the effect worth detecting.

Checks allowed during the test

During enrollment and follow-up, Trailside may perform planned measurement and safety checks. A guardrail alert should be investigated using the predefined thresholds:

  • checkout error rate: an increase versus control A of at least +1.0 pp
  • checkout-related support-contact rate: an increase versus control A of at least +0.5 pp

A confirmed checkout-blocking defect requires an immediate safety pause. This safety action is separate from any primary-metric result.

Warning

Avoid result-driven stopping

Do not declare an observed improvement to be a winner because it appears before the planned end. Likewise, if either version is below 1,000 eligible visitors when day 14 ends, do not extend the test opportunistically.

Finish the planned follow-up, record that exposure was insufficient, and treat the result cautiously. Investigate a guardrail alert; pause immediately only for a confirmed checkout-blocking defect.

Practice: classify stopping situations

Apply the stopping plan

Match each simulated Trailside situation to the correct stopping-plan response.

Tap an item and then its matching pair.

Step 5 of 8

Calculate Rates, Differences, and Uplift

Calculate the simulated purchase conversion rates, percentage-point difference, and relative uplift while keeping descriptive arithmetic separate from causal claims.

Read the simulated result table

Start with the assigned-visitor counts

The Simulated final results below come from Trailside’s completed test. This step uses only the purchase counts to produce three descriptive results: each purchase conversion rate, their percentage-point difference, and relative uplift. Guardrail counts are included for the later interpretation step.

Simulated final results

All figures are simulated. Control A appears first, followed by variation B.

Counts are eligible assigned visitors observed through the case’s 7-day window.
VersionEligible visitorsPurchasing visitorsCheckout errorsSupport contacts
control A1,000402010
variation B1,000502211

Use each version’s own denominator

A visual comparison of 40 purchases out of 1,000 eligible visitors for control A and 50 purchases out of 1,000 eligible visitors for variation B.

A rate is calculated within each assigned version, not from a combined denominator.

Calculate rates separately

Calculate each purchase conversion rate

Use the predefined formula:

conversion rate = purchasing visitors / eligible assigned visitors × 100

Count purchasing visitors once, and use the eligible assigned visitors for that same version as the denominator. Do not substitute sessions, button clicks, or total orders.

For control A:

40 / 1,000 × 100 = 4.0%

For variation B:

50 / 1,000 × 100 = 5.0%

Tip

Keep calculation and conclusion separate

The arithmetic answers a descriptive question: what proportion of each assigned group purchased within the specified window? It does not by itself show that the button label reliably caused the difference.

Example

A clear rate summary

Report the two rates with their version names: control A is 4.0%; variation B is 5.0%. Naming both the numerator-based rate and its version prevents a comparison from becoming ambiguous.

Use the correct comparison language

Difference versus relative uplift

Subtract percentage-valued rates in percentage points (pp):

5.0% - 4.0% = +1.0 pp

Then calculate the change relative to the control rate:

relative uplift = (rate B - rate A) / rate A × 100

For Trailside:

(5.0% - 4.0%) / 4.0% × 100 = +25%

The same observed change can therefore be described as +1.0 pp or +25% relative uplift. These are different units.

Warning

Label the result precisely

Do not call a move from 4.0% to 5.0% a “1% relative improvement.” The absolute change is +1.0 percentage point; the relative uplift uses 4.0% as its baseline and is +25%. For now, call this an observed improvement, not proof of a reliable effect.

Practice the distinction

Check the labels

Which summary correctly describes the simulated purchase result?

Calculate the full result

Summarize the descriptive result

Using these simulated purchase counts—control A: 40 purchasing visitors out of 1,000 eligible assigned visitors; variation B: 50 out of 1,000—write one compact result statement. Include both conversion rates, the percentage-point difference, and relative uplift. Use the term “observed improvement”; do not describe the uplift as a 1% relative improvement.

Write at least 55 characters (0/55).

Step 6 of 8

Recognize an Observed Lead Without Declaring a Winner

Interpret simulated A/B test results carefully by comparing sample sizes, sensitivity to individual outcomes, and the limits of descriptive rates.

See the difference behind the rates

An observed lead is not a decision

An observed improvement means that variation B’s rate was higher in the visitors observed for this test. It does not, by itself, show that the button label caused a reliable effect.

Even with concurrent random assignment, each version receives a different mix of visitor outcomes. Some differences between observed rates can arise from chance variation. The same percentage pattern can therefore be more fragile in a small sample than in a larger one.

Why sample size changes sensitivity

A side-by-side diagram contrasts a large visitor sample with a small visitor sample; a single purchase dot has a visibly larger effect in the small sample.

The same rate pattern can be based on very different numbers of visitors and purchases. A single outcome changes a small denominator more sharply.

Compare the simulated datasets

Same percentages, different counts

These are separate simulated datasets. They produce the same descriptive purchase rates, difference, and relative uplift—but not the same amount of information.

Simulated final results

The main case dataset. Control A appears before variation B.

Purchase conversion rate = purchasing visitors / eligible assigned visitors × 100
VersionEligible visitorsPurchasing visitorsConversion rate
control A1,000404.0%
variation B1,000505.0%

Small-sample illustration

A separate illustration, not an additional segment of the main dataset.

It has the same displayed conversion rates as the simulated final results.
VersionEligible visitorsPurchasing visitorsConversion rate
control A10044.0%
variation B10055.0%

Warning

One outcome matters more in a small sample

In both simulated datasets, 5.0% - 4.0% = +1.0 pp and relative uplift = (rate B - rate A) / rate A × 100 = +25%.

But one purchase equals 1.0 percentage point out of 100 visitors and only 0.1 percentage point out of 1,000 visitors. If variation B had one fewer purchase in the small-sample illustration, its rate would fall from 5.0% to 4.0%.

Test your judgment

Check the interpretation

Which statement best interprets the two simulated datasets?

Apply the distinction

State a calibrated conclusion

Write three short sentences about the two simulated datasets: compare their per-version sample sizes; explain how one purchase changes the rate in the 100-visitor illustration; and classify whether the displayed rate difference alone justifies a claim of a reliable effect.

Write at least 150 characters (0/150).

Step 7 of 8

Check Validity and Choose a Cautious Next Action

Diagnose invalid evidence, safety concerns, and valid but inconclusive simulated results before choosing a cautious next action.

Validate the comparison before interpreting it

Validate the evidence first

Before interpreting an observed improvement, verify that both versions produced comparable evidence. A comparison can be invalid when one version has duplicate purchase recording, missing purchase events, mixed counting units, or a changed definition.

For Trailside, both versions must count each eligible assigned visitor once under the same confirmed-purchase rule and the same 7-day window. If variation B records a confirmed purchase twice while control A records it once, the reported conversion rates do not describe a fair comparison.

Also investigate incomplete follow-up or an unexpected assignment imbalance that suggests an allocation, traffic, or eligibility problem. Do not automatically explain either issue as an effect of the button label.

Comparable measurement

Two parallel measurement paths show how duplicated or missing purchase events can make the two versions incomparable.

Comparable measurement requires the same visitor-based definitions and event recording in control A and variation B.

Warning

Keep the predeclared rules

Do not repair an inconvenient result after seeing it. Do not switch the primary metric, remove selected visitors, or extend the same test only because the result currently looks promising. Investigate a flaw transparently; if needed, plan a separate valid test.

Read guardrails without overclaiming

Review harm checks separately

The simulated final results should be reviewed against the predeclared increases versus control A. Guardrails are harm checks; they do not turn the purchase result into proof of a reliable effect.

Guardrail comparison

Simulated final guardrail review

Both rates use eligible assigned visitors and the same 7-day window.
Guardrailcontrol Avariation BIncrease vs control AAlert thresholdStatus
checkout error rate2.0%2.2%+0.2 pp+1.0 ppBelow alert
checkout-related support-contact rate1.0%1.1%+0.1 pp+0.5 ppBelow alert

Example

Below an alert is not a safety guarantee

The simulated increases are below the predeclared alert thresholds. That is useful evidence, but it does not prove that variation B cannot cause harm. A confirmed checkout-blocking defect is different: pause exposure immediately and investigate, regardless of these percentages.

Match diagnostic evidence to an action

Choose the response that fits the problem

Match each simulated diagnostic situation with its most appropriate immediate response.

Diagnostic response

Match each situation to the appropriate action.

Tap an item and then its matching pair.

Make an evidence-calibrated decision

Explain three different next actions

Write three short bullets, one for each simulated situation:

A. An audit finds that variation B excludes confirmed orders made with one payment method, while control A includes them.

B. A confirmed checkout-blocking defect appears in variation B.

C. Measurement is comparable, enrollment and follow-up are complete, and the observed purchase rates are 4.0% for control A and 5.0% for variation B. Both guardrails remain below their alert thresholds, but no formal uncertainty assessment is supplied.

For each, state the next action and explain why it differs from the other two.

Write at least 180 characters (0/180).

Step 8 of 8

Complete the Test Brief and Decision Note

Integrate a predeclared A/B test plan with simulated final results, then write a cautious decision note that separates observed improvement from evidence still needed.

1. Assemble the test brief

Build one connected brief and note

Your final deliverable has two parts: a test brief written before results, then a decision note written from the simulated final results. Keep the plan fixed when you interpret the data.

Use these brief headings exactly:

  • Evidence and hypothesis
  • Control and variation
  • Assignment
  • Metrics and guardrails
  • Stopping plan

Use these decision-note headings exactly:

  • Observed result
  • Limitations
  • Next action

The simulated evidence is that 3 of 5 observed mobile shoppers hesitated because the cart button’s next step was unclear. This motivates a hypothesis; it does not prove that a new label will increase purchases.

One focused change

Two matching mobile cart screens differ only in their bottom action-button wording.

For Trailside, control A uses “Continue” and variation B uses “Proceed to checkout.” No other copy, layout, pricing, or checkout behavior changes.

Fill the brief with the predeclared design

Write the hypothesis as a connection: because the simulated observation suggests that the next step is unclear, changing only the cart-button label may increase confirmed purchases. Name control A and variation B precisely; do not add a second change to the variation.

For Assignment, specify eligible mobile cart visitors. Exclude staff and known bots using rules set before assignment. Randomly allocate eligible visitors 50/50 using one stable, permitted visitor identifier for assignment and counting. Returning visitors keep their original version. The identifier does not reliably identify a person across devices, and both versions run concurrently for comparable traffic.

2. Apply the plan to the simulated record

Use fixed definitions and stopping rules

Use the following predeclared definitions in the brief. The primary metric is purchase conversion rate = purchasing visitors / eligible assigned visitors × 100. Count eligible assigned visitors with at least one confirmed purchase within 7 days of first assignment. Count each visitor once in each relevant numerator or denominator.

The guardrails use the same eligible assigned visitors and 7-day window: checkout error rate and checkout-related support-contact rate. Their alert thresholds are increases versus control A of at least +1.0 pp and +0.5 pp, respectively.

The stopping plan is 14 consecutive enrollment days, an operational exposure requirement of 1,000 eligible visitors per version, then final analysis only after every enrolled visitor has had 7 days of follow-up. If exposure is insufficient at the deadline, do not extend opportunistically. A confirmed checkout-blocking defect requires an immediate safety pause.

Simulated final results

Enrollment and follow-up are complete. This is the main simulated dataset; it is not a live experiment.

Counts are eligible assigned visitors, each counted once for the relevant measure.
VersionEligible visitorsPurchasing visitorsCheckout errorsSupport contacts
control A1,000402010
variation B1,000502211

Record results without changing the plan

Purchase conversion rate: control A is 40 / 1,000 × 100 = 4.0%; variation B is 50 / 1,000 × 100 = 5.0%. The observed difference is 5.0% - 4.0% = +1.0 pp. Relative uplift = (rate B - rate A) / rate A × 100 = +25%.

For guardrails, checkout error rate is 2.0% for control A and 2.2% for variation B: an increase of +0.2 pp. Support-contact rate is 1.0% and 1.1%: an increase of +0.1 pp. Neither reaches its alert threshold.

These are descriptive simulated results. No formal uncertainty assessment is supplied, so the observed improvement is not evidence enough to declare a winner or statistical significance.

3. Review what the evidence can support

Decision-note review

Before you write the decision note, make these distinctions:

  • Report the observed purchase comparison separately from the guardrail checks.
  • Below-threshold guardrails mean that no predefined alert was crossed; they do not prove that harm is impossible.
  • Completed enrollment and follow-up make the planned observation complete, but they do not supply the missing uncertainty assessment.
  • Do not revise eligibility, the primary metric, thresholds, or stopping rules after seeing this observed improvement.
  • An inconclusive interpretation is not proof of no effect, equality, or failure of the button-label idea.
  • A cautious next action can retain control while obtaining an appropriate uncertainty assessment or planning a separate future test.

Summary

Final checklist

Use your response to show that you can connect design, arithmetic, guardrails, and uncertainty without overstating what simulated rates establish.

  • The brief fixes the hypothesis, one focused variation, assignment, metrics, guardrails, and stopping plan before results.
  • The simulated primary-metric comparison is an observed improvement, expressed both in percentage points and relative uplift.
  • The simulated guardrail changes remain below their predeclared alert thresholds.
  • No formal uncertainty assessment is supplied; therefore a reliable winner, significance, and proven equivalence cannot be claimed.
  • A defensible next action is cautious rather than result-driven.

4. Submit your final application

Final application: write the brief and decision note

Submit one concise response with all eight required headings, in this order:

Evidence and hypothesis
Control and variation
Assignment
Metrics and guardrails
Stopping plan
Observed result
Limitations
Next action

In the brief, specify the focused hypothesis, the unchanged elements, consistent assignment and eligibility, the primary-metric counting rule, both guardrails and thresholds, and the stopping plan. In the decision note, report the calculated purchase comparison in both required forms, assess both guardrails against their thresholds, identify what evidence is missing, and recommend a cautious next action. Do not claim a winner, statistical significance, or proven equivalence.

Write at least 700 characters (0/700).

Step complete

Congratulations! You completed: Design and Interpret a Simple A/B Test

You completed the simulated A/B test brief and an evidence-calibrated decision note.

Download the App now to have access to + 5000 free courses, exercises, certificates and lots of content without paying anything!

  • 100% free online courses from start to finish

    Thousands of online courses in video, ebooks and audiobooks.

  • More than 60 thousand free exercises

    To test your knowledge during online courses

  • Valid free Digital Certificate with QR Code

    Generated directly from your cell phone's photo gallery and sent to your email

Cursa app on the ebook screen, the video course screen and the course exercises screen, plus the course completion certificate