A p-value is the probability of observing a test statistic at least as extreme as yours, computed under the assumption that H0 is true.
p-value=P(∣Z∣≥∣zobs∣∣H0 true)
Quick Answer
A p-value is the probability of observing a test statistic at least as extreme as yours, computed under the assumption that H0 is true. Use it to measure how surprising your data would be if nothing unusual were going on. A small p-value, below the significance level α you chose in advance, means data like yours would rarely occur under H0, so you reject it. For a two-sided test, count both tails: P(∣Z∣≥∣zobs∣).
Why This Matters
Most scientific claims you read, from "this drug lowers blood pressure" to "this teaching method helps," are backed by a p-value. Reading it correctly separates "these data would be surprising if the drug did nothing" from the common misreading "there's only a 3% chance the drug does nothing." That misreading inflates confidence in weak results and lies behind many headlines that later fail to replicate.
Intuitive Explanation
A friend's coin lands heads 62 times in 100 flips. Is it biased? Start by assuming it is fair. Then the number of heads has mean 50 and standard deviation 100×0.5×0.5=5, so 62 heads is z=562−50=2.4 standard deviations above the mean. The p-value asks: if the coin is fair, how often would a result at least this extreme happen? Counting both directions (62 or more heads, or 38 or fewer), the normal approximation gives P(∣Z∣≥2.4)≈0.016.
So a fair coin would be this lopsided only about 1.6% of the time. Either a rare fluke happened or the coin isn't fair; with α=0.05 fixed in advance, 0.016<0.05 and you reject "fair." Had the coin shown 54 heads, z=0.8 and p≈0.42: a fair coin does that all the time, so there would be no evidence of bias.
The p-value is computed by assuming H0 is true, so it cannot be the probability that H0 is true. "1.6% of fair coins would do this" is not "there is a 1.6% chance this coin is fair"; the second would also depend on how common biased coins are in the first place. The p-value also differs from α: α is the cutoff you set before collecting data, while the p-value comes from the data and is compared with it.
Think of it like this.A p-value is a surprise meter that assumes nothing special is going on. A friend says she is an average free-throw shooter, then sinks 20 shots in a row; you ask how often an average shooter would do that. The rarer that is under her story, the less you believe the story. The meter rates the data, not the story's probability.
Core idea.The p-value is the probability of getting data at least this extreme assuming the null hypothesis is true.
When to Use
Use a p-value after computing a test statistic (z, t or χ2) when you need to say how surprising the data are under H0, then compare it with α. Count both tails for a two-sided Ha (=) and one tail for > or <. If the question asks how likely H0 itself is, a p-value can't answer it. If it asks for the long-run rate of rejecting a true null, that is α, the Type I error rate.
Tip.Ask: Am I computing the probability of data this extreme assuming the null is true (not the probability the null is true)?
How to Recognize It
Before reading a p-value, find the test statistic it comes from, the direction of the alternative hypothesis, and the significance level that was fixed before the data arrived.
1.
Has a test statistic, such as z=2.1, already been computed from this sample?
Yes means the p-value is the next step: the tail area beyond that statistic under H0. A number chosen before any data, such as 0.05, is the significance level α, not a p-value.
2.
Is the alternative hypothesis two-sided, with a = sign?
Yes means count both tails: for z=2.1 that is 2×0.0179≈0.036. For > or <, use only the tail in the direction of Ha, here about 0.018.
3.
Is the question about how often a test would wrongly reject a true H0 over many repetitions?
Yes is the Type I error rate, which equals α and is fixed by the design. A p-value describes one sample; the error rate describes the long run of the procedure.
4.
Is the question about how often an interval method captures the true value in the long run?
Yes is a confidence level, such as 95%, which belongs to estimation. A p-value answers a different question: how unusual one sample would be if H0 were true.
P-Value compared with Significance level α, Confidence level, and Type I error rate
All four rows are probabilities near 0.05 or 95%. Ask when each is set and what it describes: one sample's surprise (p-value), the cutoff chosen in advance (α), the long-run false-alarm rate (Type I error), or an interval method's capture rate (confidence level).
P-Value compared with Significance level α, Confidence level, and Type I error rate
Type
Meaning
Key test
Formula
Example
P-Value
Use this when you have already computed a test statistic and need to measure how surprising the data is under H0 before deciding to reject.
Am I computing the probability of data this extreme assuming the null is true (not the probability the null is true)?
p-value=P(∣Z∣≥∣zobs∣∣H0 true)
A coin lands heads 92 times in 100 flips. Test H0: the coin is fair (p=0.5) at α=0.05.
Significance level α
A threshold you fix in advance (like 0.05); the p-value is then compared to it.
Use α when setting the decision cutoff before seeing data, not measuring the data's rarity.
reject if p <α
Choosing α=0.05 before the experiment
Confidence level
The long-run capture rate of an interval estimate, not a test of a specific claim.
Use when estimating a parameter with a range, not testing a hypothesis.
1−α
A 95% CI for the true mean
Type I error rate
The probability of falsely rejecting a true null over many repeated tests, equal to α, not the p-value of one sample.
Use when describing the test's long-run false-positive rate.
α=P(reject H0∣H0 true)
5% of fair-coin studies will wrongly call the coin biased
P-Value
Meaning
Use this when you have already computed a test statistic and need to measure how surprising the data is under H0 before deciding to reject.
Key test
Am I computing the probability of data this extreme assuming the null is true (not the probability the null is true)?
Formula
p-value=P(∣Z∣≥∣zobs∣∣H0 true)
Example
A coin lands heads 92 times in 100 flips. Test H0: the coin is fair (p=0.5) at α=0.05.
Significance level α
Meaning
A threshold you fix in advance (like 0.05); the p-value is then compared to it.
Key test
Use α when setting the decision cutoff before seeing data, not measuring the data's rarity.
Formula
reject if p <α
Example
Choosing α=0.05 before the experiment
Confidence level
Meaning
The long-run capture rate of an interval estimate, not a test of a specific claim.
Key test
Use when estimating a parameter with a range, not testing a hypothesis.
Formula
1−α
Example
A 95% CI for the true mean
Type I error rate
Meaning
The probability of falsely rejecting a true null over many repeated tests, equal to α, not the p-value of one sample.
Key test
Use when describing the test's long-run false-positive rate.
Formula
α=P(reject H0∣H0 true)
Example
5% of fair-coin studies will wrongly call the coin biased
Formula & Notation
p-value=P(∣Z∣≥∣zobs∣∣H0 true)
p-value=P(∣Z∣≥∣zobs∣∣H0) (two-tailed); reject H0 when p-value <α
How to read it:If p-value <α, reject H0. If p-value ≥α, fail to reject H0.
Worked Examples
Example 1 — Biased coin?
Easy
A coin lands heads 92 times in 100 flips. Test H0: the coin is fair (p=0.5) at α=0.05.
Show the steps and the answer
Under H0 the count of heads has mean 100×0.5=50 and standard deviation 100×0.5×0.5=5.
The p-value is computed in a world where the coin is fair, so these are the numbers to use.
Standardize: z=592−50=8.4.
92 heads is 8.4 standard deviations above what a fair coin gives on average.
Find the two-sided p-value: P(∣Z∣≥8.4) is far below 10−15.
Even 4 standard deviations out has a probability under 0.0001; 8.4 is off every table.
Compare with α: p≈0<0.05, so reject H0.
A fair coin essentially never gives 92 heads in 100 flips, so the claim of fairness can't stand.
Answer:p-value ≈0<0.05, so reject H0
Takeaway:The p-value measures how rare the data would be if H0 were true; when it is essentially 0, the claim falls.
Example 2 — The probability the null is true
Standard
A study reports p =0.04. A student says 'so there's a 4% chance the drug doesn't work.' Is that the p-value's meaning?
Show the steps and the answer
Write what the study computed: P(data this extreme∣drug does nothing)=0.04.
A p-value is calculated by assuming H0, so H0 sits on the condition side of the bar.
Write what the student claimed: P(drug does nothing∣data)=0.04.
That flips the condition, and a flipped conditional probability can have a very different value.
Show the gap with counts: of 1,000 drugs tested, suppose 900 do nothing and 100 work. With α=0.05, about 45 useless drugs pass; if 80 working drugs pass, 45 of the 125 passing drugs, 36%, are useless.
Even among drugs that passed, far more than 4% are useless, because useless drugs were common to begin with.
Restate it correctly: if the drug did nothing, results this extreme would occur about 4% of the time.
That is a statement about the data under H0, which is all a p-value can give.
Answer:No — p is P(data∣H0), not P(H0∣data)
Takeaway:P(data∣H0) is not P(H0∣data); a p-value never gives the chance that the null is true.
Example 3 — Find the mistake: one tail for a two-sided test
Application
Testing H0:μ=50 against Ha:μ=50 at α=0.05, Kai gets z=1.8. He writes: "P(Z>1.8)≈0.036<0.05, so reject H0." Find the mistake and correct it.
Show the steps and the answer
Read the alternative: Ha:μ=50 is two-sided.
A result as far below 50 as this one is above it would be equally strong evidence, so both tails count.
Add both tails: p=P(∣Z∣≥1.8)=2×0.036=0.072.
Kai used only the upper tail, which halved the p-value.
Compare with α: 0.072>0.05.
The data are not rare enough under H0 at the 5% level.
Cross-check with the critical value: 1.8<1.96.
The two-sided cutoff at α=0.05 is 1.96 and 1.8 falls short, so both methods agree.
Answer:p≈0.072>0.05: fail to reject H0.
Takeaway:For a = alternative, double the one-tail area before comparing with α.
Where You'll See It
Reading a drug trial headline
A trial reports that patients on a new drug had lower blood pressure than those on a placebo, with p=0.003. That means: if the drug did nothing, a gap this large would appear in about 3 trials out of 1,000. It says nothing about how big or useful the drop is; that needs the effect size or a confidence interval.
Physics and the five-sigma rule
Physicists announced the Higgs boson only when the signal reached 5 standard deviations, a one-sided p-value of about 2.9×10−7. Because they search through so many possible signals, they demand a far smaller p-value than 0.05 before claiming a discovery.
Website A/B tests
A company compares two sign-up pages and gets p=0.30 for the difference in sign-up rates. A gap that size would be common if the pages performed equally well, so the team keeps collecting data instead of declaring a winner.
Common Mistakes
The slip:Reading the p-value as the probability the null hypothesis is true
Instead:it is the probability of the data given the null, not of the null given the data.
The slip:Concluding H0 is true when p is large
Instead:a large p-value means insufficient evidence to reject, never proof the null holds.
The slip:Comparing the p-value to the wrong tail or forgetting two-sided
Instead:use ∣Z∣≥∣zobs∣ for a two-sided test, doubling the one-tail area.
Mini Practice
Try these on your own. Tap Reveal when you want to check.
1.
A test gives a p-value of 0.03. Do you reject H0 at α=0.05? At α=0.01?
Hint:Reject only when the p-value is at or below α.
2.
A two-sided z-test gives zobs=2.00. The area to the right of 2.00 under the standard normal curve is 0.0228. Find the p-value.
Hint:Two-sided means extreme results in both tails count.
3.
A coin lands heads 9 times in 10 flips. For Ha:p>0.5, find the p-value: the probability of 9 or more heads if the coin is fair.
Hint:Count the ways to get exactly 9 heads and exactly 10 heads out of 210 outcomes.
4.
Same coin, 9 heads in 10 flips, but now the test is two-sided (Ha:p=0.5). Find the p-value.
Hint:At least as extreme in the other direction means 1 or fewer heads.
5.
Before collecting any data, a researcher decides to reject H0 only if the result would happen less than 1% of the time when H0 is true. Is 0.01 a p-value or the significance level?
Hint:Which number is chosen before the data, and which is computed from it?
6.
A study gets p=0.40. Leo writes: "The p-value is large, so the null hypothesis is true." Correct his conclusion.
Hint:A large p-value means the data are ordinary under H0. Is that the same as proof?
Want the full set?
50 practice questions for this concept — free to try, every one with a complete worked
solution showing the why, not just the answer.
Is a p-value of 0.06 really that different from 0.04?
Hardly. The two results carry almost the same strength of evidence; one just happens to fall on the other side of a line drawn by convention. That is why many statisticians recommend reporting the p-value itself, along with the size of the effect, rather than only significant or not significant. A cutoff is useful for making a yes-or-no decision, not for describing how strong the evidence is.
If the null hypothesis is true, what values does the p-value usually take?
Any value from 0 to 1, all equally likely. Under a true H0, with a continuous test statistic, the p-value is uniformly distributed, so it falls below 0.05 exactly 5% of the time and below 0.5 half the time. That is why α is the Type I error rate. It also means a single p-value of 0.30 tells you little: true nulls produce values like that all the time.
Can a p-value be exactly 0?
Not really. Software may print 0.000, but that only means the value is smaller than the digits shown. As long as the observed result is possible under H0, its probability is above zero, even if it is tiny. Report it as p<0.001 rather than p=0, which would claim the data are impossible if the null is true.
Why does the definition say at least as extreme instead of exactly this result?
Because the probability of any one exact result is small, even under H0. Getting exactly 50 heads in 100 fair flips has probability only about 0.08, yet 50 heads is the least surprising result possible. Asking for results at least as far from the expected value as yours measures how unusual your result is compared with all the others, which is what a p-value is meant to capture.
You need Hypothesis Testing (null, alternative, test statistic) and Probability first. The p-value turns a test statistic into one number between 0 and 1, the chance of data at least this extreme if H0 holds, to compare against α. Next, Type I and Type II Errors explain what can go wrong when you act on that comparison, and why α is the false-alarm rate.