Bring these inputs
A funnel problem, baseline conversion, traffic estimate, event QA and business constraints.
Use approved, minimized data. Replace personal details with anonymous IDs.
Work through the lesson
- Identify the earliest relevant friction point using analytics and customer evidence. Fix broken functionality without calling the repair an A/B test.
- Write a hypothesis linking one change to a shopper behavior. Pick a primary metric and guardrails such as order value, errors, margin and returns.
- Estimate required sample size with a suitable calculator or analyst using baseline rate, minimum detectable effect, significance and power. AI can explain inputs; it cannot create sufficient traffic.
- Randomly assign eligible users with stable allocation. Verify tracking, exclusions and sample-ratio balance. Run through the planned duration and business cycles; do not stop when a result first looks positive.
- Report counts, effect size, uncertainty and guardrails. If traffic is too low, use interviews, usability sessions and sequential improvements, clearly labeled as non-causal observations.
Try this prompt
Replace the bracketed fields with your own evidence.
Draft an A/B test brief for [problem/change]. Baseline [rate], traffic [eligible users], primary outcome [metric]. Ask for missing sample-size assumptions. Specify unit of randomization, allocation, tracking QA, guardrails, duration and stopping rule. Explain what an inconclusive result means. Do not invent a statistically significant winner.
Worked example
Illustrative results: 100 purchases / 5,000 users versus 115 / 5,000 gives 2.0% versus 2.3%. A 15% relative increase alone does not establish significance or justify rollout. Assess uncertainty and guardrails.
Before you use the result
- The test was planned before launch.
- AI-generated personas are not counted as user research.
- An inconclusive result is reported honestly.
Complete the exercise
Create the deliverable described at the top of this lesson. Save the input evidence, your edited output, unresolved questions and the name of the reviewer in your workbook.