A result is statistically significant when the p-value falls below a predetermined threshold (alpha, typically 0.05), indicating that the observed effect is unlikely to have occurred by random chance alone. Statistical significance is a binary decision criterion used in hypothesis testing — it does not measure the size or practical importance of the effect.
Statistical significance is a decision rule: before looking at data, you set a threshold (usually 5%). If your p-value is below this threshold, you declare the result 'significant' - meaning unlikely to be just random noise. It's not about importance; it's about confidence that something real is happening.
Showing a random 20 of 50 problems.
Example 1
hard
An experiment runs at α=0.05 but is repeated 4 independent times against the same H0. What is the family-wise probability of at least one false rejection when H0 is true?
Example 2
easy
Is statistical significance the same as proving causation?
Example 3
easy
A scientist sets α=0.05 but reports significance after seeing the data is close. Why is this problematic?
Example 4
hard
A team plans an A/B test with target α=0.05 and 80% power to detect a 1% lift. Suppose during the test they peek and stop early when significance is hit. What is the danger?
Example 5
easy
If α=0.05 and the null is true, what fraction of experiments will falsely be called significant?
Example 6
easy
Statistical significance is a ____ decision (yes/no), not a continuous measure.
Example 7
easy
A result is statistically significant when the p-value falls below what?
Example 8
medium
Using α=0.01, a test gives p =0.03. Is the result significant at this level?
Example 9
medium
A 95% confidence interval for an effect is [0.5, 2.0] and excludes 0. Is the effect statistically significant at α=0.05?
Example 10
medium
Two independent studies report p=0.05 each. Is the combined evidence stronger?
Example 11
medium
Two results: A has p =0.04 with a large effect, B has p =0.04 with a tiny effect. Are they equally important?
Example 12
challenge
A trial sets α=0.05. A result gives z=1.9 (two-sided p ≈0.057). The team lowers the bar to α=0.10 after seeing this to claim significance. Identify the methodological error.
Example 13
medium
Why is using α=0.05 for a high-stakes medical decision potentially inappropriate?
Example 14
easy
With p =0.03 and α=0.05, is the result statistically significant?
Example 15
medium
A drug trial fails to show statistical significance (p=0.20, n=20). Why might it still be wrong to conclude 'the drug doesn't work'?
Example 16
medium
A drug lowers blood pressure by 0.2 mmHg with p =0.001 in a huge trial. Is it statistically significant? Is it practically important?
Example 17
hard
A treatment improves average test scores by 12 points, but the p-value is 0.08. At α=0.05 is the result statistically significant, and could the effect still be practically important?
Example 18
medium
Fill in: at α=0.10, it becomes ____ to declare significance than at α=0.05.
Example 19
easy
At α=0.05, is a result with p=0.049 statistically significant?
Example 20
medium
A result has p =0.001 at α=0.05. Is it statistically significant?