Not sure which lesson to do first? Find your level and get a plan →

Statistics · Grade 9-12

Hypothesis Testing

Hypothesis testing is the formal procedure for using sample data to decide between two competing claims about a population: you state a null (no effect) and an alternative, compute a test statistic, and judge whether the evidence is strong enough to reject the null.


Quick Answer

Hypothesis testing is the formal procedure for using sample data to decide between two competing claims about a population: you state a null (no effect) and an alternative, compute a test statistic, and judge whether the evidence is strong enough to reject the null. Use it when someone makes a specific claim, such as a coin is fair, a drug does nothing, or a machine fills bottles to 500 ml, and you want to know whether the data contradict it. If chance alone would rarely produce a result this far from the claim, you reject the claim.

Why This Matters

Every sample differs a little from its population, so every comparison shows some gap. A test separates gaps that chance produces routinely from gaps it almost never produces. Without it, 54 out of 100 tasters preferring a new recipe looks like a win; with it, you can see that tasters with no real preference would produce a split that lopsided about one time in four, so it proves little.

Intuitive Explanation

A friend claims her coin is fair. You flip it 100 times and get 65 heads. If the coin really were fair, the number of heads would center on 50 with standard deviation 100×0.5×0.5=5, so 65 is 3 standard deviations out. A fair coin lands that far from 50 (65 or more heads, or 35 or fewer) only about 1 time in 300. Either something very unusual happened or the coin is not fair, and the sensible bet is the second.

That is the whole logic of a test. Assume the boring claim, the null hypothesis H0 (here p=0.5). Work out how the statistic would vary if H0 were true. Measure how far the data fall from that prediction with a test statistic such as z, turn the distance into a probability (the p-value), and reject H0 when that probability is below a cutoff α chosen in advance. The alternative Ha (here p≠0.5) is the claim you accept only when the evidence forces you to.

Now suppose you had seen 55 heads. A fair coin gives a result at least that far from 50 more than a third of the time, so you do not reject H0. That is not proof the coin is fair: a coin that lands heads 55% of the time would produce 55 heads just as easily. "Fail to reject" means the evidence fell short, not that the null won. A test also answers only "is this claim believable?"; to say how biased a coin is, you need a confidence interval.

Think of it like this. A criminal trial runs on the same logic. The defendant (the null) is presumed innocent, the prosecution's evidence (the data) must show guilt beyond reasonable doubt (a small p-value), and the verdict is "guilty" or "not guilty", never "innocent". One difference: a test fixes in advance how often it will convict an innocent null, the rate α.

Core idea. Assume nothing is going on, work out how the statistic would vary by chance alone, and see whether the observed result would be rare. A rare result counts as evidence against the null; an ordinary one leaves the null standing but unproven.


When to Use

Run a hypothesis test when a problem names a specific claimed value or "no difference" and asks whether the data give evidence against it: is the coin biased, does the new method raise scores, is the machine underfilling. You need a sample statistic, the null value, and the statistic's spread under the null. If the question asks how large an effect is or wants a range of plausible values, build a confidence interval; if there is no claim to test, simply describe the data.

Tip. Ask: Is there a specific claim about a population, such as no effect or a stated value, that the sample data might contradict?

How to Recognize It

Before setting up a test, find the exact claim being challenged and what kind of result would count against it.

  1. Can you write the claim as one value or as no difference, such as p=0.5 or equal mean scores?

    Yes, that is the null hypothesis. If nothing is being claimed and you only need an estimate, build a Confidence Interval instead.

  2. Is there a suspected direction or difference that the researcher hopes to find evidence for?

    Yes, that is the alternative hypothesis, and its wording decides whether the test is one-sided or two-sided.

  3. Did the data come from a random sample or a randomized experiment?

    Yes leaves chance as the only innocent explanation the test must rule out. With a biased sample, no test can rescue the conclusion, because the problem is Sampling Bias.

  4. Are you asked only how surprising the result would be if the claim were true?

    Then the job is the P-Value alone; a full test also states the hypotheses and ends with a decision.

Hypothesis Testing vs P-Value vs Statistical Significance vs Confidence Interval

These show up in the same inference problems, often in one paragraph. Ask whether you need the whole claim-and-decide procedure, the surprise number inside it, the yes-or-no verdict at its end, or a range estimate with no claim to test.

Hypothesis Testing

Meaning
Hypothesis testing is the full procedure of stating a null and an alternative, computing a test statistic, and deciding about the null.
Key test
Starts with a claim to challenge and ends with a decision.
Formula
t=xˉ−μ0s/n
Example
Testing H0:μ=500 g against Ha:μ≠500 g with 36 boxes.

P-Value

Meaning
The p-value is the probability, assuming the null is true, of a result at least as extreme as the one observed.
Key test
One number between 0 and 1 that measures surprise.
Formula
P(∣T∣≥∣tobs∣∣H0)
Example
With t≈−2.67 and 35 degrees of freedom, p≈0.01.

Statistical Significance

Meaning
A result is statistically significant when its p-value falls below a threshold α chosen in advance.
Key test
The yes-or-no verdict at the end.
Formula
p<α
Example
Since 0.01<0.05, the box result is significant at the 5% level.

Confidence Interval

Meaning
A confidence interval is a range of plausible parameter values built from sample data at a stated confidence level.
Key test
Estimates a value and needs no claim.
Formula
496±2.03×1.5
Example
Mean box weight between about 493 g and 499 g, with 95% confidence.

Formula & Notation

Worked Examples

Example 1 — Is the machine underfilling?

Easy

A machine is supposed to fill bottles with 500 ml on average, and fill amounts have standard deviation σ=6 ml. A random sample of 36 bottles has mean 497 ml. At α=0.05, is there evidence that the machine underfills?
Show the steps and the answer
  1. State the hypotheses: H0:μ=500, Ha:μ<500.

    The null is the machine working as designed; the alternative is the suspicion being tested, that the mean is too low.

  2. Find the standard error: 6/36=1 ml.

    If H0 is true, means of 36 bottles vary around 500 ml with this spread.

  3. Compute the test statistic: z=497−5001=−3.

    The sample mean sits 3 standard errors below what the null predicts.

  4. Find the p-value: P(Z≤−3)≈0.0013.

    Only the low side counts, because Ha says the mean is below 500.

  5. Decide: 0.0013<0.05, so reject H0.

    A sample mean this low would turn up only about 1 time in 750 if the machine were filling correctly.

Answer: Reject H0: there is strong evidence (z=−3, p≈0.001) that the machine fills less than 500 ml on average.

Takeaway: State H0 and Ha, measure the gap in standard errors, convert it to a p-value, then compare with α.

Example 2 — Not rejecting is not proving

Standard

In a blind taste test, 56 of 100 people prefer a new soda recipe over the old one. Test H0:p=0.5 against Ha:p>0.5 at α=0.05. Then decide whether the result shows that people have no preference.
Show the steps and the answer
  1. Find the standard error under H0: 0.5×0.5/100=0.05.

    The test asks how p^ would vary if people were choosing at random.

  2. Compute z=0.56−0.500.05=1.2 and the p-value P(Z≥1.2)≈0.115.

    By the normal approximation, 56 or more out of 100 would happen about 11.5% of the time with no real preference.

  3. Decide: 0.115>0.05, so do not reject H0.

    The evidence is not strong enough to say that most people prefer the new recipe.

  4. Check what is still plausible with a 95% interval: 0.56±1.960.56×0.44/100=0.56±0.097, from 0.46 to 0.66.

    True preference rates as high as 60% or 65% still fit the data, so "no preference" has not been shown.

Answer: Do not reject H0 (p≈0.115), but this does not show that people have no preference: anything from about 46% to 66% preferring the new recipe fits the data.

Takeaway: A test can find evidence against the null but never proves it; a small sample can leave a real effect undetected.

Example 3 — Find the mistake: hypotheses swapped

Application

A school claims its students sleep 7 hours a night on average. Riley thinks they sleep less. In a random sample of 49 students the mean is 6.7 hours; assume σ=1.4 hours. Riley writes: "H0:μ<7, Ha:μ=7. My sample mean is below 7, so H0 is supported." Find the mistakes and run the test correctly at α=0.05.
Show the steps and the answer
  1. Swap the hypotheses: H0:μ=7, Ha:μ<7.

    The null is the claim being challenged and must name one exact value, so the test can work out what samples would look like if it were true.

  2. Compute SE=1.4/49=0.2 and z=6.7−70.2=−1.5.

    Being below 7 is not enough on its own; the question is whether 0.3 hours below is more than chance would produce.

  3. Find the p-value: P(Z≤−1.5)≈0.067.

    A sample mean this low would occur about 6.7% of the time even if the true mean were 7 hours.

  4. Decide: 0.067>0.05, so do not reject H0.

    The evidence falls short of the preset cutoff, so the school's claim stands, though it has not been proved.

Answer: H0:μ=7, Ha:μ<7; z=−1.5, p≈0.067, so do not reject H0: the sample does not give convincing evidence that students average less than 7 hours.

Takeaway: Put the equality, the status-quo claim, in H0 and your suspicion in Ha; then let the p-value, not the direction of the sample mean, decide.

Where You'll See It

  • Testing a 'lucky' coin

    You suspect a coin is biased. The null hypothesis is 'fair coin (p = 0.5)'; you flip it 100 times and check whether the number of heads is too extreme to be explained by chance under that assumption.

  • Does a new study app raise test scores?

    Null: the app makes no difference. You compare scores of students who used it vs those who didn't, and test whether the gap is large enough to be unlikely if the app truly did nothing.

  • Taste test for a soda recipe

    Null: people can't tell the new recipe from the old one (they'd be right 50% by guessing). You run a blind test and reject the null only if correct identifications are far above chance.

Common Mistakes

  1. The slip: Failing to reject the null hypothesis proves the null is true.

    Why it's tempting: Not finding evidence of an effect feels like finding evidence there is no effect.

    Instead: Failing to reject means there isn't enough evidence against the null — 'absence of evidence is not evidence of absence'; you never prove the null true.

  2. The slip: You set up the alternative hypothesis as 'no effect' and try to support it.

    Why it's tempting: Students often write the hypothesis they hope is true as the starting assumption.

    Instead: The null is the 'no effect / status quo' claim you assume and try to disprove; the alternative is the new effect you're gathering evidence for.

  3. The slip: A p-value of 0.0013 means there is only a 0.13% chance that the null hypothesis is true.

    Why it's tempting: A small p-value is what leads you to reject H0, so it reads like the chance that H0 is right.

    Instead: The p-value is worked out assuming H0 is true: it is the chance of a sample mean of 497 ml or lower if the machine really averages 500 ml. It measures how surprising the data would be under H0, not the probability that H0 itself is true.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. A cereal company says its boxes hold 500 g on average. A consumer group suspects the boxes are underfilled. Write the null and alternative hypotheses.

    Hint: The null is the claim of no difference; the alternative is what the group suspects.

  2. Bulbs are claimed to last 1000 hours on average, with σ=100 hours. A random sample of 25 bulbs averages 960 hours. Compute the z test statistic for H0:μ=1000.

    Hint: Find the standard error first, then see how many standard errors 960 is from 1000.

  3. A test of H0:μ=1000 against Ha:μ<1000 hours of bulb life gives a p-value of 0.023. At α=0.05, state the decision and the conclusion in context.

    Hint: Compare the p-value with alpha, then say what that means for bulb life.

  4. A coin lands heads 60 times in 100 flips. Test H0:p=0.5 against Ha:p≠0.5 by computing z. The two-sided p-value for that z is about 0.046. What do you conclude at α=0.05?

    Hint: Use SE=0.5⋅0.5/100, the spread when the null is true.

  5. Which task needs a hypothesis test: (a) "If a coin is fair, what is the chance of 3 heads in 3 flips?" or (b) "A coin gave 3 heads in 3 flips; is that evidence it is biased?"

    Hint: Which one uses observed data to judge a claim?

  6. A student tests whether a new teaching method changes average scores, gets p=0.31, and writes: "This proves the new method has no effect." Correct the conclusion.

    Hint: What is the difference between failing to reject a claim and proving it?

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

What are Type I and Type II errors?

A Type I error rejects a null that is actually true, like convicting an innocent person; its chance is α\alpha, the threshold you choose. A Type II error fails to reject a null that is false, like letting a guilty person go free. Lowering α\alpha makes Type I errors rarer but Type II errors more common, unless you also collect more data, which reduces both.

When should a test be one-sided instead of two-sided?

Only when the question, fixed before seeing any data, cares about one direction, such as whether a new battery lasts longer than the old one, with a shorter life counting the same as no improvement. A one-sided test puts all of α\alpha in one tail, so the same data give half the p-value. Picking the side after seeing which way the data point doubles the real false-alarm rate.

What is the power of a test?

Power is the chance that a test correctly rejects a false null, that is, detects an effect that really exists. It rises with a bigger sample, a bigger true effect and a larger α\alpha, and falls when the data are noisy. Researchers often plan studies to have about 80% power for the smallest effect they care about, since an underpowered study can easily miss a real effect.

Learning Path

You need Sampling Distribution (how a statistic varies when the null is true), Standard Error (the yardstick in a test statistic) and Basic Probability. Hypothesis testing assembles them into a decision: null versus alternative, test statistic, p-value, verdict. Next, P-Value looks closely at the probability at the center of the test, and Statistical Significance at the rule that turns it into a decision.

See Also