BUS 2100 Module 3 Inference and a Rollout Decision Example

Significant or Not? Inference and a Checkout Rollout Decision

Student Name

Bachelor of Science in Business Administration, Thomas Edison State University

BUS-2100: Quantitative Skills for Business

Mentor Name

Month Day, Year

Significant or Not? Inference and a Checkout Rollout Decision

The Decision

A composite online kitchenware retailer redesigned its checkout page and tested it against the current page for two weeks, sending 2,400 visitors to each version at random. The current page converted 168 visitors to buyers, 7.0 percent, and the new page 204, 8.5 percent. The marketing director wants to roll out the new page now; the finance manager wants more evidence. Wisniewski and Shafti (2020) treat this kind of question in two steps: first, interpret what the sample says about the population, then choose a course of action under the uncertainty that remains.

Step 1: Testing the Difference

The null hypothesis is that the two pages convert at the same rate. Pooling both groups gives an overall rate of 372 / 4,800 = 7.75 percent. The standard error of the difference under the null hypothesis is the square root of 0.0775 x 0.9225 x (1/2,400 + 1/2,400), about 0.0077. The observed difference of 1.5 percentage points divided by that standard error gives z = 1.94.

The two-sided p-value is 0.052, just above the conventional 0.05 threshold, and the one-sided p-value, appropriate only if the company had committed in advance to caring solely about improvement, is 0.026. A 95 percent interval around the 1.5-point difference, built with the unpooled standard error of about 0.0077, runs from -0.01 to 3.0 percentage points, so it barely includes zero.

Interpreting the Result

It would be a mistake to read p = 0.052 as proof that the new page does not work, or a one-sided p = 0.026 as proof that it does. Wasserstein and Lazar (2016), writing for the American Statistical Association, warn that a p-value tells neither how likely a hypothesis is to be correct nor how large an effect is, and that decisions should not rest on whether a p-value crosses a threshold. The more useful reading is the interval: the data are consistent with an improvement anywhere from about zero to three points, with 1.5 points the best estimate. Kohavi et al. (2020), drawing on large-scale online experiments, add that tests should run for a planned duration with an adequate sample, since stopping when results look favorable inflates false positives.

Step 2: The Payoff Table

The site has about 120,000 visitors a month, and each purchase contributes about $19.20 in profit, so each percentage point of conversion is worth about $23,040 a month. Management considered three actions: roll out now at a cost of $20,000; extend the test four weeks at an added $5,000, delaying any gain by a month but avoiding a harmful rollout; or keep the current page. Four states of the world are possible, with probabilities the team judged from the test evidence.

Applying the Decision Criteria

Each criterion reflects a different attitude to risk. Maximax, the optimist's rule, picks the best possible outcome and chooses to roll out now. Maximin, the pessimist's rule, picks the action with the best worst case and chooses to keep the current page, whose worst outcome is zero. Minimax regret measures, for each state, how far each action falls short of the best action in that state; rolling out has a maximum regret of $158,240 if the page harms sales, keeping the page has a maximum regret of $394,720 if the lift is strong, and extending the test has a maximum regret of only $39,560, so it is chosen. Expected value, which weights outcomes by probability, favors rolling out at $208,096 against $194,424 for extending the test and zero for keeping the page.

Recommendation

The company should extend the test for four more weeks with a fixed end date, then roll out if the combined data confirm the lift. Extending costs about $13,700 in expected value compared with rolling out now, but it reduces the worst case from a loss of about $158,000 to $5,000 and has the lowest maximum regret. Given a result on the edge of conventional significance and a confidence interval that includes zero, buying more evidence at that price is sound. If leadership is less averse to risk and trusts the probability estimates, rolling out now is defensible on expected value.

Conclusion

The test suggests the new page probably helps, but not conclusively. Reading the confidence interval rather than a threshold, and comparing actions with several decision criteria, shows that a short extension of the test is a low-cost way to reduce the risk of a wrong decision.

References

Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108

Wisniewski, M., & Shafti, F. (2020). Quantitative analysis for decision makers (7th ed.). Pearson.