Not sure which lesson to do first? Find your level and get a plan →

Statistics · Grade 9-12

Statistical Significance

A result is statistically significant when the p-value falls below a preset threshold α (typically 0.


Quick Answer

A result is statistically significant when the p-value falls below a preset threshold α (typically 0.05), meaning the observed effect is unlikely to be just random chance. Use it at the decision step of a hypothesis test: choose α before seeing the data, compute p, and report "significant, reject H0" if p<α or "not significant" otherwise. With α=0.05, a true null gets wrongly rejected 5% of the time. Significance says chance is a poor explanation; it does not say the effect is large.

Why This Matters

A fixed cutoff lets a lab, a drug regulator or a website team make the same call on the same evidence, and it caps how often they raise a false alarm when nothing is there. Misused, it misleads: with a big enough sample a trivial difference becomes "significant", and running 20 tests on pure noise yields about one "significant" result by luck.

Intuitive Explanation

Before collecting data, a researcher picks a significance level, usually α=0.05. After the test, the rule is mechanical: p=0.03 is below 0.05, so the result is significant and H0 is rejected; p=0.08 is not, so H0 stands. The same p=0.03 would not be significant at α=0.01, which is why the level must be fixed in advance: choosing it after seeing p lets you get whichever verdict you wanted.

The level is also an error rate. When the null is true, a test's p-value is equally likely to fall anywhere between 0 and 1, so it lands below 0.05 in 5% of studies; that is the false-alarm rate you accept. Test many true nulls and some will come out significant by luck alone, which is why testing until something works is cheating. In the other direction, p=0.08 does not show the effect is zero; the study may simply be too small to detect it.

Statistically significant sounds like "important", but the two answer different questions. Significance asks whether chance can explain the result; importance asks whether the effect is big enough to matter. Suppose a tutoring app is tested on two groups of 50,000 students whose scores have standard deviation 15: a gain of just 0.2 points out of 100 gives p≈0.035, significant and useless. To judge importance, look at the size of the effect and its confidence interval, not at how small p is.

Think of it like this. A smoke alarm with a sensitivity dial works like a significance test. You set the dial (α) before cooking; at 0.05, the alarm sounds on about 5% of smoke-free days. A beep means "probably not just chance", not "big fire": burnt toast and a house fire both set it off. And silence does not prove there is no smoke.

Core idea. Fix a threshold α before seeing the data and call a result significant when p<α. That rule wrongly rejects a true null only α of the time, but it says nothing about how large or important the effect is.


When to Use

Use statistical significance at the decision step: you have a p-value and a preset significance level α, and the question asks whether to reject H0 or whether a result is "significant". Reject when p<α; otherwise say the evidence is insufficient, not that the null is true. If the question asks how surprising the data are, report the p-value itself; if it asks whether an effect is big enough to matter, use the effect size or a confidence interval.

Tip. Ask: Must you give a yes-or-no verdict by comparing a p-value with a threshold chosen before the data came in?

How to Recognize It

Before calling a result significant, find the threshold, when it was chosen, and what the result actually measured.

  1. Is a significance level such as 0.05 or 0.01 stated or implied?

    Yes means the result can be judged significant or not. With no threshold, report the P-Value itself as a measure of the strength of evidence.

  2. Was the threshold set before looking at the data, and was only one test run?

    Yes keeps the false-alarm rate at α. Many tests, or a cutoff picked afterward, inflate it, so a significant-looking result may be chance.

  3. Does the question ask whether the effect is big enough to matter in practice?

    That is a different question. Significance only rules out chance as a good explanation; size is judged from the estimate and its Confidence Interval.

Statistical Significance vs P-Value vs Hypothesis Testing vs Confidence Interval

These are parts or companions of one test, so they get used as if they meant the same thing. Ask whether you need the final verdict, the probability it rests on, the whole procedure, or a range showing how big the effect is.

Statistical Significance

Meaning
A result is statistically significant when its p-value is below a threshold α fixed before the data were seen.
Key test
A yes-or-no verdict that says nothing about size.
Formula
p<α
Example
A fertilizer trial with p=0.03 is significant at α=0.05 but not at α=0.01.

P-Value

Meaning
The p-value is the probability of results at least this extreme if the null hypothesis were true.
Key test
A number on a scale, not a verdict.
Example
p=0.03: results at least this extreme would occur about 3 times in 100 if the fertilizer did nothing.

Hypothesis Testing

Meaning
Hypothesis testing is the whole procedure, from stating the null and alternative to the final decision.
Key test
The full process, of which significance is the last step.
Example
State H0: no growth difference, run the trial, compute p, then decide.

Confidence Interval

Meaning
A confidence interval gives a range of plausible values for the size of an effect.
Key test
Shows how big the effect might be, which significance cannot.
Formula
estimate±margin of error
Example
Fertilized plants grew 0.2 to 3.8 cm more, with 95% confidence.

Formula & Notation

Worked Examples

Example 1 — Significant at 0.05, not at 0.01

Easy

A reading program claims to raise scores above the district mean of 75. A random sample of 64 students in the program averages 78, and scores have standard deviation σ=12. Test H0:μ=75 against Ha:μ>75. Is the result significant at α=0.05? At α=0.01?
Show the steps and the answer
  1. Find the standard error: 12/64=1.5.

    Under H0, means of 64 students vary around 75 with this spread.

  2. Compute z=78−751.5=2 and p=P(Z≥2)≈0.023.

    Ha is one-sided (scores go up), so only the upper tail counts.

  3. Compare with α=0.05: 0.023<0.05, so the result is significant and H0 is rejected.

    Chance alone would give a mean this high in only about 2.3% of samples.

  4. Compare with α=0.01: 0.023>0.01, so the result is not significant at that level.

    A stricter level demands stronger evidence, so the same data can pass one cutoff and fail another.

Answer: p≈0.023: significant at α=0.05, not significant at α=0.01.

Takeaway: Significance is a comparison of p with a chosen α; report the p-value as well, so readers can apply their own standard.

Example 2 — Significant but tiny

Standard

A typing app is tested on 40,000 students. Students without the app type 60.8 words per minute on average, with σ=10. The app users average 61.0 words per minute. Test H0:μ=60.8 against Ha:μ>60.8. Is the gain statistically significant at α=0.05? Is it important?
Show the steps and the answer
  1. Find the standard error: 10/40,000=10/200=0.05.

    With 40,000 students the sample mean is pinned down very precisely.

  2. Compute z=61.0−60.80.05=4, so p=P(Z≥4)≈0.00003.

    A 0.2-word gain is 4 standard errors above the no-effect value, far beyond what chance usually produces.

  3. Decide: p<0.05, so the gain is statistically significant.

    Chance is a poor explanation for a gain this many standard errors from zero.

  4. Judge the size: 0.2 words per minute is 0.2/10=0.02 standard deviations.

    That is one extra word every five minutes, too small for any student to notice, so it is not practically important.

Answer: Statistically significant (z=4, p<0.001), but practically unimportant: a gain of only 0.2 words per minute.

Takeaway: Large samples make tiny effects significant. Always ask how big the effect is, in real units, before calling it important.

Example 3 — Find the mistake: twenty tests, one winner

Application

Theo tests whether eating each of 20 different snacks before a quiz raises scores, running a separate test for each at α=0.05. Gummy bears give p=0.04; the other 19 snacks are not significant. Theo writes: "Gummy bears significantly improve quiz scores (p=0.04)." Find the mistake and correct it.
Show the steps and the answer
  1. Count the false alarms expected if no snack does anything: 20×0.05=1.

    Each test of a useless snack has a 5% chance of a significant result, so one "winner" out of 20 is exactly what luck predicts.

  2. Find the chance of at least one false alarm, treating the tests as independent: 1−0.9520≈0.64.

    Theo's study had about a 64% chance of producing a "significant" snack even if all 20 are useless.

  3. Apply a correction for 20 tests: the Bonferroni cutoff is 0.05/20=0.0025, and 0.04>0.0025.

    Dividing α by the number of tests keeps the overall false-alarm rate near 5%.

  4. Plan the fix: test gummy bears alone in a new study, with α set in advance.

    A fresh test of one pre-chosen hypothesis restores the 5% false-alarm rate.

Answer: Not significant once all 20 tests are counted (corrected cutoff 0.0025); gummy bears are a lead to retest, not a finding.

Takeaway: Running many tests and reporting the one that passes inflates false alarms; adjust α for the number of tests or confirm the result in a new study.

Where You'll See It

  • A/B testing a website button

    A company changes a button color and finds a statistically significant rise in clicks (p < 0.05), meaning the bump is unlikely to be random — but a 0.1% increase may be significant yet too tiny to matter for the business.

  • New medicine vs placebo

    A drug shows a statistically significant improvement over a placebo, telling doctors the effect is probably real; whether it's worth prescribing depends separately on how big the improvement is.

  • Huge sample making everything 'significant'

    A survey of 100,000 people can find a statistically significant 0.2-point difference in test scores — significant by the p-value rule, yet far too small to be practically meaningful.

Common Mistakes

  1. The slip: Statistically significant means the effect is large or practically important.

    Why it's tempting: The everyday meaning of 'significant' is 'big and meaningful.'

    Instead: Statistical significance only means the effect is unlikely to be chance; with a large enough sample, even a trivially small effect can be significant, so check the effect size too.

  2. The slip: A non-significant result (p > 0.05) proves there is no effect.

    Why it's tempting: Not crossing the threshold feels like a clean 'nothing here' verdict.

    Instead: Non-significant means insufficient evidence to declare an effect at that threshold; the effect may exist but be undetectable with the current sample size.

  3. The slip: You can keep testing different things and report whichever comes out significant.

    Why it's tempting: Each individual significant result looks like a legitimate finding.

    Instead: Testing many things until one passes (p-hacking) inflates the chance of a false positive; the significance threshold must be set in advance and account for multiple comparisons.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. A study reports p=0.02. Is the result statistically significant at α=0.05? At α=0.01?

    Hint: Significant means the p-value is below the chosen alpha.

  2. A two-sided test gives z=1.8, and P(Z≥1.8)≈0.036. Is the result significant at α=0.05?

    Hint: A two-sided p-value counts both tails.

  3. A 95% confidence interval for the difference in mean scores (new method minus old) is −1.2 to 4.8 points. Is the difference significant at α=0.05 with a two-sided test?

    Hint: Does the interval include the value that means no difference?

  4. A researcher tests 20 unrelated foods for a link to acne, each at α=0.05, when in fact none has any effect. How many significant results should she expect by chance, and what is the chance of at least one?

    Hint: Multiply for the expected count. For at least one, start with the chance that none is significant.

  5. A school board asks: "How big is the score gain from the new reading program?" A report says only "statistically significant, p=0.01." Should the answer be a significance verdict or a confidence interval?

    Hint: Does a p-value say how large an effect is?

  6. A trial of 40,000 patients finds a new drug lowers blood pressure 0.5 mmHg more than placebo, with p<0.001. A student writes: "It is highly significant, so the drug has a big effect." Correct the reasoning.

    Hint: Separate whether the effect is real from how big it is.

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

Why is 0.05 the usual threshold?

Mostly tradition. The statistician Ronald Fisher suggested in the 1920s that 1 in 20 was a convenient line for flagging results worth a second look, and the habit stuck. Nothing special happens at 0.05. Particle physicists demand about 3 in 10 million, a five-sigma result, before announcing a discovery, and some researchers have proposed 0.005 as the bar for new claims in medicine and psychology.

Does a significant result mean the effect is definitely real?

No. If most ideas being tested are wrong, a good share of significant findings will be false alarms. Suppose only 10 of 100 tested treatments really work, tests use α=0.05\alpha = 0.05, and each has 80% power. You expect 8 true hits and about 4.5 false ones from the 90 useless treatments, so more than a third of the significant results are wrong.

How does a significance test connect to a confidence interval?

For a mean, they agree when built at matching levels: a two-sided test at α=0.05\alpha = 0.05 rejects a claimed value exactly when that value lies outside the 95% confidence interval. So if a drug's 95% interval for the drop in blood pressure runs from 1 to 9 mmHg, the drop is significant at the 5% level, since 0 is outside, and the interval also shows the effect could be as small as 1.

Learning Path

Statistical Significance

You are here

You need P-Value (how surprising the data are under the null) and Hypothesis Testing (the two claims being decided between). Significance adds the yes-or-no rule p<α and its built-in false-alarm rate. With it you can run complete hypothesis tests and judge them critically: pick α to suit the stakes, report effect sizes with confidence intervals, and adjust the cutoff when many tests are run at once.

See Also