Free Sample Size Calculator

Smaller effects are harder to distinguish from random noise. If you want to detect a 1% relative lift instead of a 20% lift, you need far more data to be confident the difference is real and not just statistical fluctuation. The sample size grows roughly as the inverse square of the MDE.

The industry standard is 95% (alpha = 0.05), which means there is a 5% chance of a false positive. Use 90% for exploratory tests where speed matters more than precision. Use 99% for high-stakes decisions like pricing changes where a wrong call is costly.

No. Peeking at results and stopping early inflates your false positive rate — a phenomenon called "peeking bias." If you want to evaluate results before the full sample is collected, use a sequential testing method designed for early stopping.

An underpowered test is likely to miss real effects (false negatives) or declare winners that aren't actually better (false positives due to noise). The result might look statistically significant by chance, but it won't replicate. Always calculate your sample size before starting a test.

Run it until you reach the required sample size, but also for at least one full business cycle (typically one or two weeks) to account for day-of-week effects. Tests shorter than a week can be biased by traffic patterns. Tests longer than 4-6 weeks risk cookie expiration and external factors contaminating results.