Mini Case Study: Choosing a Variant Using Expected Loss and a Business Cost Model

Scenario Setup: Two Variants, One Decision, Real Costs

Imagine you run an e-commerce checkout flow and you are choosing between two variants: Variant A (current) and Variant B (new). The experiment goal is not “maximize conversion rate” in isolation; it is “maximize profit per visitor” while accounting for operational costs and business risks. The decision you must make is binary: ship A or ship B. The uncertainty is continuous: each variant’s true conversion rate is unknown, and the true average order value and downstream costs may differ slightly too. In this mini case study, we will use a business cost model to translate uncertain outcomes into dollars, then choose the variant that minimizes expected loss (equivalently, maximizes expected utility).

We assume you already have posterior samples (or an analytic posterior) for each variant’s conversion rate from your experiment data. The focus here is how to connect those posteriors to a decision using a cost model that reflects your business reality.

Define the Business Cost Model (Profit per Visitor)

To choose a variant, we need a payoff function: a way to compute profit (or loss) per visitor given a set of parameters. A simple but practical model for profit per visitor is:

profit_per_visitor = p_convert * margin_per_order - cost_per_visitor

Where p_convert is the conversion probability for that variant, margin_per_order is the expected contribution margin per order (revenue minus variable costs), and cost_per_visitor includes per-visitor costs that might differ by variant (e.g., extra support load, payment fees, fraud checks, infrastructure).

In many real decisions, the model needs a few more terms. Suppose Variant B introduces a new payment step that slightly increases conversion but also increases fraud risk and customer support contacts. Then we might model:

Continue in our app.

You can listen to the audiobook with the screen off, receive a free certificate for this course, and also have access to 5,000 other free online courses.

Or continue reading below...

Download the app

profit_per_visitor = p_convert * margin_per_order - cost_base - p_convert * cost_per_order - p_fraud * cost_fraud

Here cost_base is a per-visitor cost (e.g., extra infrastructure), cost_per_order is a per-order cost (e.g., payment processing fee differences), and p_fraud is the probability of a fraud event per visitor (or per order) multiplied by its expected cost. If you do not have a posterior for p_fraud, you can treat it as a scenario parameter (best case / base case / worst case) or assign a distribution based on historical rates.

Keep the model “decision-complete,” not “perfect”

The model should include the costs that could plausibly change the decision. You do not need to model every accounting line item; you need to model the terms that differ between A and B or that interact with conversion in a way that changes the ranking. A good test is: “If I set this term to zero, could the chosen variant change?” If yes, include it.

Translate “Expected Loss” into a Concrete Decision Rule

Expected loss is the average regret you would feel if you choose a variant and reality turns out differently. In a two-variant setting, a common loss definition is “opportunity loss”: the gap between the profit you could have earned by choosing the better variant (in hindsight) and the profit you actually earn with your chosen variant.

For each possible state of the world (i.e., each draw from the posterior), define:

profit_A = f(theta_A, other_params_A, shared_params, scenario_params, draw_noise_if_any)
profit_B = f(theta_B, other_params_B, shared_params, scenario_params, draw_noise_if_any)

If you choose A, your loss in that draw is:

loss_if_choose_A = max(profit_A, profit_B) - profit_A

If you choose B, your loss is:

loss_if_choose_B = max(profit_A, profit_B) - profit_B

Expected loss is the average of these losses across posterior draws. Choose the variant with lower expected loss. This rule automatically balances upside against downside, and it naturally incorporates business costs and risk terms.

Step-by-Step Workflow Using Posterior Samples

This workflow assumes you have posterior samples for conversion rates pA and pB. If you also have uncertainty about margin, fraud, or other terms, you can sample those too. The key is to compute profit for each draw and then compute expected loss.

Step 1: Gather inputs and decide what is uncertain

Start with a table of parameters. Separate them into (a) variant-specific uncertain parameters, (b) shared uncertain parameters, and (c) fixed scenario parameters.

• Variant-specific uncertain: p_convert_A, p_convert_B (posterior samples from the experiment).
• Shared uncertain: margin_per_order (if it varies), baseline fraud rate (if you want to model uncertainty).
• Fixed scenario: per-visitor infrastructure cost difference, per-order payment fee difference, expected cost per fraud event.

Example fixed business assumptions (illustrative):

• Contribution margin per order: $40 (or a distribution if uncertain).
• Base per-visitor cost: $0.20 for A, $0.28 for B (B is heavier).
• Per-order payment fee: $1.20 for A, $1.35 for B.
• Fraud cost per fraud event: $120.
• Fraud probability per order: A = 0.20%, B = 0.35% (could be scenario-based).

Step 2: Define the profit function

Write a function that maps a draw of uncertain parameters into profit per visitor. For the example above, we can model fraud as “per order,” so fraud probability per visitor is p_convert * p_fraud_given_order. Then:

profit = p_convert * (margin_per_order - payment_fee_per_order) - base_cost_per_visitor - p_convert * p_fraud_given_order * fraud_cost

This is intentionally simple: it captures the main trade-off (conversion vs. costs) and a risk term (fraud) that scales with orders.

Step 3: Compute profit per draw for A and B

For each posterior draw i, compute profit_A[i] and profit_B[i]. If margin is uncertain, sample margin[i] too. If fraud is uncertain, sample that as well. The output is two arrays of profits aligned by draw index.

Step 4: Compute opportunity loss per draw

For each draw, compute the best achievable profit and the loss for each choice:

best[i] = max(profit_A[i], profit_B[i])
loss_A[i] = best[i] - profit_A[i]
loss_B[i] = best[i] - profit_B[i]

These losses are always non-negative. When A is better in a draw, loss_A is zero and loss_B is the profit gap; when B is better, the reverse happens.

Step 5: Average losses and choose the variant

Compute expected loss:

EL_A = mean(loss_A)
EL_B = mean(loss_B)

Choose A if EL_A < EL_B, otherwise choose B. Also compute the expected profit difference for context:

expected_profit_A = mean(profit_A)
expected_profit_B = mean(profit_B)
expected_uplift = expected_profit_B - expected_profit_A

Expected loss is the decision criterion; expected uplift is a helpful summary but can hide tail risk. Two variants can have similar expected profit but very different expected loss if one has a small chance of being much worse.

Worked Mini Example with Interpretable Numbers

Suppose your posterior samples imply that B likely improves conversion, but not by much. For intuition, imagine the posterior draws roughly behave like:

• p_convert_A centered around 3.0%.
• p_convert_B centered around 3.2%.
• There is overlap: in some draws A is higher, in others B is higher.

Using the cost model above, B’s higher base cost and higher payment fee reduce profit, and the higher fraud rate adds a risk penalty that grows with conversion. When you compute profit per visitor, you might see outcomes like:

• In many draws, B wins by a small margin because conversion is slightly higher.
• In a meaningful minority of draws, A wins because the conversion gain is not enough to pay for B’s extra costs.

Expected loss captures this asymmetry. If B’s downside (when it is worse) is larger than its upside (when it is better), B can have higher expected profit but still higher expected loss depending on the distribution of gaps. This is common when the new variant introduces a cost that is “always on” (base cost) while the benefit is uncertain (conversion lift).

Adding a Business Constraint: Minimum Acceptable Profit Impact

Many organizations require that a change clears a minimum bar, such as “at least +$X per 1,000 visitors in expected profit” or “no more than Y% chance of losing money.” You can incorporate this without reverting to arbitrary significance thresholds by defining loss to include constraint violations.

Constraint example: penalize negative profit impact beyond a tolerance

Let delta = profit_B - profit_A. Suppose the business can tolerate small negative swings (noise, short-term volatility) up to -$0.002 per visitor (i.e., -$2 per 1,000 visitors), but beyond that it becomes painful (support load, executive scrutiny, budget impact). Define a piecewise loss for choosing B:

loss_choose_B = 0 if delta >= 0
loss_choose_B = -delta if -0.002 <= delta < 0
loss_choose_B = 5 * (-delta) if delta < -0.002

This says: if B is worse, you pay the regret; if it is worse beyond tolerance, you pay a larger penalty (a risk-aversion multiplier). You can define an analogous loss for choosing A if you want symmetry, or you can treat A as the “safe” default and only penalize B’s downside more heavily. The multiplier should reflect a real business preference: for example, outages, brand damage, or operational overload can make downside much more costly than upside is valuable.

Constraint example: include a fixed “switching cost”

Shipping B may require engineering time, QA, documentation, and training. Model that as a one-time cost amortized over an expected horizon of traffic. If the one-time cost is $30,000 and you expect the variant to run for 3 million visitors before the next redesign, that is $0.01 per visitor. Add it to B’s base cost:

base_cost_B = base_cost_B + switching_cost / expected_visitors_horizon

This often changes decisions when lifts are small. It also forces clarity about the time horizon: a variant that is slightly better per visitor may not be worth shipping if it will be replaced soon.

Practical Implementation Sketch (Pseudo-Code)

The following pseudo-code shows the mechanics. It assumes arrays of posterior draws pA and pB of equal length N. You can generate these draws from your Bayesian model and then apply this decision layer.

# Inputs (arrays of length N): pA, pB
# Fixed parameters: margin, baseA, baseB, feeA, feeB, fraudCost, fraudRateA, fraudRateB

for i in 1..N:
  profitA[i] = pA[i] * (margin - feeA) - baseA - pA[i] * fraudRateA * fraudCost
  profitB[i] = pB[i] * (margin - feeB) - baseB - pB[i] * fraudRateB * fraudCost

  best = max(profitA[i], profitB[i])
  lossA[i] = best - profitA[i]
  lossB[i] = best - profitB[i]

EL_A = mean(lossA)
EL_B = mean(lossB)

choose = 'A' if EL_A < EL_B else 'B'

# Helpful diagnostics
prob_B_better = mean(profitB > profitA)
expected_uplift = mean(profitB - profitA)
p5_uplift = quantile(profitB - profitA, 0.05)
p95_uplift = quantile(profitB - profitA, 0.95)

Notice that we compute prob_B_better on profit, not conversion. This is a common real-world correction: a variant can “win” on conversion but lose on profit once costs are included.

Interpreting Outputs for Stakeholders

When you present the decision, show outputs that map directly to business questions. A compact set that works well is:

• Expected profit per visitor for A and B.
• Expected uplift (B minus A) in dollars per 1,000 visitors.
• Probability B is more profitable than A.
• Expected loss for choosing A vs choosing B (your decision metric).
• A downside summary such as the 5th percentile of uplift (a “bad but plausible” scenario).

Expected loss is especially useful when stakeholders ask, “What’s the risk if we pick the wrong one?” You can answer in the same unit as the business: dollars per visitor (or per week of traffic). If you multiply by expected traffic over the decision horizon, you get an expected regret in dollars over that horizon, which is often easier to act on than abstract probabilities.

Common Modeling Choices and Pitfalls

Mixing per-visitor and per-order costs incorrectly

Be explicit about which costs scale with visitors and which scale with orders. Payment fees, fulfillment costs, and fraud often scale with orders, so they should be multiplied by p_convert. Infrastructure costs might be per visitor (page load, API calls) or per order (post-purchase processing). If you treat a per-order cost as per-visitor, you will over-penalize high-conversion variants.

Ignoring interaction effects that matter operationally

Some costs are nonlinear. For example, customer support might be fine until you exceed staffing capacity, after which wait times and refunds spike. If you have such a threshold, approximate it with a piecewise penalty in the loss function (similar to the tolerance example). The goal is not perfect realism; it is capturing the decision-relevant kink.

Using a single “average order value” when variants change basket composition

If Variant B changes what people buy (e.g., upsells), then margin per order may differ by variant. In that case, treat margin_A and margin_B separately, and if you have uncertainty, sample them. Even a small change in margin can dominate a small change in conversion.

Forgetting the time horizon

Expected loss and expected uplift should be evaluated over the period you expect the decision to stand. If you will revisit in two weeks, the downside exposure is limited; if you will lock in for a quarter, the exposure is larger. This matters especially when you include switching costs or when downside risk is severe.

Extending the Case: Multiple Variants and “Pick the Best”

If you have more than two variants, the same approach works: compute profit for each variant per draw, identify the best profit in that draw, and compute each variant’s opportunity loss relative to that best. Expected loss then ranks all variants by decision quality under uncertainty.

# For K variants, profits is an N x K matrix
best[i] = max_k profits[i, k]
loss[i, k] = best[i] - profits[i, k]
EL[k] = mean_i loss[i, k]
choose k with smallest EL[k]

This is often more stable than choosing the variant with the highest probability of being best, because expected loss accounts for “how bad it is when it’s wrong,” not just “how often it’s wrong.”