Statistical Significance Statistics Example 3

Follow the full solution, then compare it with the other examples linked below.

Example 3

hard
A study with 10,000 participants finds a statistically significant difference in test scores between two teaching methods (p=0.001p = 0.001), but the difference is only 0.5 points out of 100. Discuss.

Solution

  1. 1
    Step 1: With n=10,000n = 10,000, even tiny differences become statistically significant because the standard error is very small.
  2. 2
    Step 2: A 0.5-point difference is negligible in practice — no teacher would change methods for half a point.

Answer

Statistically significant due to large sample size, but practically insignificant — the effect size is trivial.
Large sample sizes can make even meaningless differences statistically significant. Always consider effect size alongside p-values.

About Statistical Significance

A result is statistically significant when the p-value falls below a predetermined threshold (alpha, typically 0.05), indicating that the observed effect is unlikely to have occurred by random chance alone. Statistical significance is a binary decision criterion used in hypothesis testing — it does not measure the size or practical importance of the effect.

Learn more about Statistical Significance →

More Statistical Significance Examples

Example 1 hard

A drug trial finds a statistically significant reduction in blood pressure ([formula]). The mean red

Example 2 hard

Explain the difference between a Type I error and a Type II error in hypothesis testing.

Example 4 hard

A treatment improves average test scores by 12 points, but the p-value is 0.08. At [formula] is the

← All Statistical Significance Examples Statistical Significance Concept