Statistical Significance Examples: 44 Problems with Answers

Worked examples of Statistical Significance to try before you open each solution, then practice problems with answers.

Worked Examples

Example 1

medium
A test gives p=0.07 at α=0.10. Is the result statistically significant? Would it be at α=0.05?
Show solution

Answer

Yes at α=0.10; No at α=0.05

First step

1.
At α=0.10: 0.07≤0.10, reject.

See the full worked solution + why-it-works coaching

SetupKey insightWhy it worksCommon pitfallConnection

Free: full solutions on 3 topics a day. No card.

Every topic, every day: the Family plan

Example 2

hard
An experiment reports p=0.045 for a treatment effect of 0.2 standard deviations. Is the result statistically significant and important?

Example 3

hard
An experiment runs at α=0.05 but is repeated 4 independent times against the same H0. What is the family-wise probability of at least one false rejection when H0 is true?

Example 4

challenge
A two-sided test of H0:μ=0 gives a 95% CI of [0.4, 0.8] for μ. Is the result statistically significant at α=0.05? What about at α=0.01 if we know the corresponding 99% CI is [0.3, 0.9]?

Example 5

hard
A drug trial finds a statistically significant reduction in blood pressure (p=0.02). The mean reduction was 2 mmHg. Is this result practically significant?

Example 6

hard
Explain the difference between a Type I error and a Type II error in hypothesis testing.

Practice Problems

Try each one, then open the solution to compare your method.

Example 1

easy
A result is statistically significant when the p-value falls below what?

Example 2

easy
What is the most common value of the significance level α?

Example 3

easy
Is statistical significance a measure of how large an effect is?

Example 4

easy
With p =0.03 and α=0.05, is the result statistically significant?

Example 5

easy
With p =0.08 and α=0.05, is the result statistically significant?

Example 6

easy
Statistical significance is a ____ decision (yes/no), not a continuous measure.

Example 7

easy
When should the significance level α be chosen?

Example 8

easy
If α=0.05 and the null is true, what fraction of experiments will falsely be called significant?

Example 9

medium
A drug lowers blood pressure by 0.2 mmHg with p =0.001 in a huge trial. Is it statistically significant? Is it practically important?

Example 10

medium
Using a stricter α=0.01 instead of 0.05, does it become easier or harder to declare significance?

Example 11

medium
Why is using α=0.05 for a high-stakes medical decision potentially inappropriate?

Example 12

medium
A 95% confidence interval for an effect is [0.5, 2.0] and excludes 0. Is the effect statistically significant at α=0.05?

Example 13

medium
Researchers run 40 subgroup analyses and highlight the one with p =0.04 as 'significant.' Why is this misleading?

Example 14

medium
A study reports 'not significant (p =0.30).' Does this prove there is no effect?

Example 15

medium
Two results: A has p =0.04 with a large effect, B has p =0.04 with a tiny effect. Are they equally important?

Example 16

medium
A result has p =0.001 at α=0.05. Is it statistically significant?

Example 17

medium
Using α=0.01, a test gives p =0.03. Is the result significant at this level?

Example 18

challenge
A trial sets α=0.05. A result gives z=1.9 (two-sided p ≈0.057). The team lowers the bar to α=0.10 after seeing this to claim significance. Identify the methodological error.

Example 19

challenge
Out of 100 independent tests with all nulls true and α=0.05, about how many significant results occur, and what does this teach about a single 'significant' finding among many?

Example 20

challenge
Explain why a statistically significant result with a 95% CI of [0.1%, 0.3%] improvement might still lead a company to NOT adopt a change.

Example 21

easy
At α=0.05, is a result with p=0.049 statistically significant?

Example 22

easy
At α=0.01, is a result with p=0.03 statistically significant?

Example 23

easy
True or false: at α=0.05, if H0 is actually true, about 5% of experiments will be 'statistically significant.'

Example 24

easy
A scientist sets α=0.05 but reports significance after seeing the data is close. Why is this problematic?

Example 25

medium
A study reports a 95% CI for an effect of [0.01, 0.04]. Is the effect statistically significant at α=0.05?

Example 26

medium
Why might a huge clinical trial detect a 'statistically significant' but irrelevant 0.05 mmHg drop in blood pressure?

Example 27

medium
A team reports many subgroup analyses at α=0.05 each and highlights the one with p=0.01. Why is this misleading?

Example 28

medium
A 99% CI for a difference is [−2, 5]. Is the difference statistically significant at α=0.01?

Example 29

medium
A drug trial fails to show statistical significance (p=0.20, n=20). Why might it still be wrong to conclude 'the drug doesn't work'?

Example 30

medium
Two independent studies report p=0.05 each. Is the combined evidence stronger?

Example 31

hard
A team prefers α=0.005 for a new drug-approval pipeline. Why is this stricter threshold sensible?

Example 32

hard
A journal publishes only statistically significant results. What bias does this create in the literature?

Example 33

hard
A start-up reports 'the new feature led to a statistically significant lift of 0.1%.' Should the engineering team ship it?

Example 34

medium
True or false: 'statistically significant' is a continuous measure of how strong the evidence is.

Example 35

medium
A study reports p=0.04 and a 95% CI of [0.01, 0.40] for an effect. What does the CI add over the p-value?

Example 36

hard
A team plans an A/B test with target α=0.05 and 80% power to detect a 1% lift. Suppose during the test they peek and stop early when significance is hit. What is the danger?

Example 37

hard
A study with 10,000 participants finds a statistically significant difference in test scores between two teaching methods (p=0.001), but the difference is only 0.5 points out of 100. Discuss.

Example 38

hard
A treatment improves average test scores by 12 points, but the p-value is 0.08. At α=0.05 is the result statistically significant, and could the effect still be practically important?

What to focus on

Core idea: Fix a threshold α before seeing the data and call a result significant when p<α. That rule wrongly rejects a true null only α of the time, but it says nothing about how large or important the effect is.

Common stuck point: Students often know a procedure related to statistical significance but skip the recognition step: Must you give a yes-or-no verdict by comparing a p-value with a threshold chosen before the data came in? That leads to a calculation or graph that looks reasonable but answers a different question.

Ask yourself: Must you give a yes-or-no verdict by comparing a p-value with a threshold chosen before the data came in?

Concept recap

A result is statistically significant when the p-value falls below a predetermined threshold (alpha, typically 0.05), indicating that the observed effect is unlikely to have occurred by random chance alone. Statistical significance is a binary decision criterion used in hypothesis testing — it does not measure the size or practical importance of the effect.

A smoke alarm with a sensitivity dial works like a significance test. You set the dial (α) before cooking; at 0.05, the alarm sounds on about 5% of smoke-free days. A beep means "probably not just chance", not "big fire": burnt toast and a house fire both set it off. And silence does not prove there is no smoke.

Read the full lesson on Statistical Significance

Statistical Significance is covered with related ideas in the guide Experimental Design: Confounding Variables, Control Groups, Random Assignment and Blinding.

Background knowledge

These ideas may be useful before you work through the harder examples.

p valuehypothesis testing