Not sure which lesson to do first? Find your level and get a plan →

Statistics · Grade 6-8

Sampling Bias

Sampling bias occurs when a sample is collected in a way that systematically makes some members of the population more likely to be included than others, producing results that do not accurately represent the full population and leading to misleading conclusions.


Quick Answer

Sampling bias occurs when a sample is collected in a way that systematically makes some members of the population more likely to be included than others, producing results that do not accurately represent the full population and leading to misleading conclusions. Look for it whenever you read how a sample was gathered: asking whoever is nearby (convenience), letting people choose to respond (voluntary response), or using a list that leaves people out (undercoverage). A biased method leans the same way each time it is used, so a bigger sample does not fix it; only changing how people are chosen does.

Why This Matters

Spotting sampling bias lets you judge a claim before believing it: a poll on a gaming website saying "80% of teens game daily" tells you about that website's visitors, not about teens. Without the idea, people trust huge online polls and reviews that only certain people bother to answer, and they blame bad luck for errors that the method itself built in.

Intuitive Explanation

Suppose a town asks, "Do you support building a skate park?" and collects answers outside the skate shop. Forty-five of 50 people say yes, which is 90%. The people there are not a fair slice of the town: skaters are far more likely to be standing outside a skate shop than anyone else. The result describes skate shop visitors. That tilt, built into how people were chosen, is sampling bias.

Three patterns cause most of it. Convenience sampling asks whoever is easy to reach, such as friends, people in one place, or the first in line. Voluntary response lets people decide whether to answer, so those with strong feelings dominate, which is why online reviews are full of 5-star and 1-star ratings. Undercoverage leaves part of the population off the list entirely, like an online poll about internet service that cannot reach people without internet. For each one, ask who was more likely to be included, and which way that pushes the answer.

Bias is different from bad luck. Four random samples of 100 students might show 46%, 53%, 50% and 48% wanting a later start: they scatter on both sides of the truth, and that is sampling variability. An online poll repeated three times showing 79%, 82% and 80% agrees with itself but is high every time, because the same kind of people answer it. Variability shrinks with bigger samples; bias does not.

Think of it like this. Picture a bathroom scale that always reads 3 kg heavy. That is sampling bias: weigh yourself ten times and the readings agree with each other, but every one is wrong in the same direction. Weighing more often never fixes it; you have to fix the scale, which here means fixing how people are chosen.

Core idea. Bias lives in the method of choosing, not in the size of the sample: if some members of the population are more likely to be included, the result leans toward their answers every time, and only a fairer selection method fixes it.


When to Use

Check for sampling bias whenever a problem tells you how people were chosen for a survey or study: "asked shoppers at the mall", "an online poll", "surveyed members of the club". Ask who was more likely to be included or left out, and which way that tilts the answer. If the sample was drawn at random and two results simply differ a little, that is sampling variability, not bias; if the question is how to choose fairly, use random sampling.

Tip. Ask: Would the way these people were chosen push the result in the same direction every time the survey was repeated?

How to Recognize It

Before blaming an odd survey result on bias, look at who could have ended up in the sample and who could not.

  1. Were some members of the population unable to be picked, such as people without internet for an online poll?

    Yes means undercoverage, one kind of Sampling Bias. If everyone had a fair chance, an odd result is more likely the luck of the draw, which is Sampling Variability.

  2. Did people decide for themselves whether to answer, as in a call-in or website poll?

    Yes means voluntary response bias, because people with strong feelings answer more often. If a chance process chose who was asked, the method is Random Sampling.

  3. Did the person collecting data pick whoever was easiest to reach?

    Yes means convenience sampling, which leans toward people like those nearby, such as one class or one store's customers. A no rules out this kind of bias, but check the other questions too.

  4. Is the trouble in how the survey question was worded rather than in who was asked?

    Then it is response bias: a question starting 'Don't you agree...' pushes answers one way even in a perfect sample. Sampling Bias is only about who gets into the sample.

Sampling Bias vs Sampling Variability vs Random Sampling vs Confounding Variables

Each of these explains why a result might be wrong. The question that separates them is whether the error comes from how people were picked, from plain luck, or from a hidden third factor in a comparison.

Sampling Bias

Meaning
Sampling bias is a selection method that keeps including some kinds of members more than others, so results lean one way.
Key test
The error has a direction set by how people were picked; more data keeps it.
Formula
Bias=E[θ^]−θ
Example
An online poll about internet speed misses people with no internet, who may have the most complaints.

Sampling Variability

Meaning
Sampling variability is the chance difference that shows up between fair random samples of one population.
Key test
Errors go either way and shrink as samples get bigger.
Example
Three fair samples of 50 students report 40%, 46% and 52% liking a new lunch menu.

Random Sampling

Meaning
Random sampling lets chance choose the sample, so every member has the same chance of being picked.
Key test
The cure for a biased selection method, not a kind of error.
Example
Picking 50 names from the full school list with a random number generator.

Confounding Variables

Meaning
A confounding variable is a hidden third factor linked to both things being compared, which can create a false connection.
Key test
The error comes from a hidden cause in a comparison, not from who was surveyed.
Example
Towns with more libraries have more crime, because bigger towns have more of both.

Formula & Notation

How to read it: Bias is measured as Bias=E[θ^]−θ, where θ^ is the sample estimate and θ is the true population parameter.

Worked Examples

Example 1 — The skate shop survey

Easy

A town of 12,000 wants to know whether residents support a new skate park. Volunteers ask 50 people outside the town's skate shop, and 45 say yes. Find the sample result, name the bias, say which way it pushes, and suggest a fix.
Show the steps and the answer
  1. Sample result: 4550=90% say yes.

    This is what the 50 people said; the question is whether it speaks for 12,000 residents.

  2. Name the bias: a convenience sample with undercoverage, since people who never visit a skate shop had almost no chance of being asked.

    The location decides who is included, so skaters and their friends are heavily over-represented.

  3. Direction: the 90% is very likely too high.

    People who skate are the most likely to want a skate park, and they are exactly the ones the method favored.

  4. Fix: draw, say, 200 names at random from the town's list of residents and ask them.

    A random draw from a full list gives every resident the same chance, removing the tilt.

Answer: The sample shows 90%, but it is a biased convenience sample that over-represents skaters, so support in the town is probably much lower; a random sample of residents would fix it.

Takeaway: Ask who the method made more likely to answer, and the direction of the bias usually follows.

Example 2 — Bias or bad luck?

Standard

Four random samples of 100 students each find 46%, 53%, 50% and 48% wanting a later school start. An online poll on the student council's page, run three times, finds 79%, 82% and 80%. Is either method biased?
Show the steps and the answer
  1. Random samples: the results range from 46% to 53%, scattered on both sides of about 49%.

    Honest random samples differ by chance, and chance pushes up as often as down.

  2. Online poll: 79%, 82%, 80%, all about 30 points above the random samples.

    Repeating the method gives the same high answer, so the gap is not chance but a tilt built into the method.

  3. Find the cause: students who care strongly about start times are the ones who visit the page and vote.

    This is voluntary response: people choose themselves, and those with strong opinions choose most often.

Answer: The random samples show ordinary sampling variability, not bias. The online poll is biased: its results agree with each other but sit far above the truth.

Takeaway: Variability scatters results both ways and shrinks with sample size; bias pushes results the same way every time.

Example 3 — Find the mistake: a huge sample is safe

Application

In 1936 the Literary Digest mailed about 10 million ballots, drawn largely from telephone books and car registration lists, and received about 2.4 million back. It predicted Alf Landon would beat Franklin Roosevelt with about 57% of the vote. Jay writes: "With 2.4 million responses, the poll can't have been biased; bias is only a problem for small samples." Find the mistake and correct it.
Show the steps and the answer
  1. Check how people were chosen: in 1936, phone and car owners were mostly wealthier households.

    This is undercoverage: poorer voters, who favored Roosevelt, had a much smaller chance of getting a ballot.

  2. Check who answered: 2.410=24% of ballots came back.

    With a 24% return, voluntary response adds a second tilt toward people motivated to reply.

  3. Compare with the result: Roosevelt won with about 61% of the vote, and Landon got about 37%.

    The poll overstated Landon's share by about 20 points (57% predicted, about 37% actual), far more than chance could cause in 2.4 million responses.

  4. Correct Jay's rule: size reduces chance error, not bias.

    A huge biased sample gives a very precise estimate of the wrong group's opinion.

Answer: Jay is wrong. The poll was badly biased by who was mailed a ballot and who chose to return it, and its size did not help: it predicted a Landon win, but Roosevelt won with about 61%.

Takeaway: To judge a sample, look first at how it was chosen; no number of responses can make up for a biased method.

Where You'll See It

  • Surveying only your friends

    Asking just your friends about favorite music over-represents people with tastes like yours, so the result is biased toward your group and not the whole school.

  • Online poll on a gaming site

    A poll asking 'do you love video games?' posted on a gaming website mostly reaches gamers, so it systematically excludes non-gamers — undercoverage that pushes the result toward 'yes.'

  • Restaurant review skew

    People who had an extreme experience (very good or very bad) are far more likely to write reviews, so the reviews are a biased, voluntary-response sample of all customers.

Common Mistakes

  1. The slip: Sampling bias is just bad luck in who you happened to pick.

    Why it's tempting: A weird sample looks like random misfortune.

    Instead: Sampling bias is SYSTEMATIC: the method consistently favors certain people. Random bad luck is sampling variability, which is different and unavoidable even with good methods.

  2. The slip: A large sample can't be biased.

    Why it's tempting: Lots of responses feels like it must average out.

    Instead: A famously huge sample can still be badly biased (e.g., the 1936 Literary Digest poll of 2 million people). Size never removes bias built into the selection method.

  3. The slip: Letting everyone choose to respond removes bias because it's inclusive.

    Why it's tempting: Open participation seems like the fairest option.

    Instead: Open, voluntary responses cause voluntary-response bias: people with strong feelings answer more. Reducing bias requires the researcher to randomly select who is asked.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. To learn how students get to school, Dev surveys the 25 students at the bus stop at 7:30 a.m., and 23 of them ride the bus. What percent of his sample rides the bus, and why is this likely too high for the whole school?

    Hint: Who would never be standing at the bus stop?

  2. A radio show asks listeners to call in about a new skate park. Of 120 callers, 102 oppose it. Name the type of bias and explain why the 85% may overstate opposition.

    Hint: Who decided to be in this sample, and what kind of people bother to call?

  3. A school of 800 students has 200 who play a sport, which is 25%. A survey of 60 students taken in the gym after practice finds 45 play a sport. Compare the sample percent with the true percent.

    Hint: Turn 45 out of 60 into a percent, then subtract.

  4. A phone survey calls 500 numbers from a list of landline phones to estimate what percent of town residents use streaming apps. Which group is undercovered, and which way will the estimate likely lean?

    Hint: Who in the town has no landline, and how might they differ?

  5. Two random samples of 50 students from the same school find that 34% and 40% like the cafeteria pizza. Is this 6-point difference a sign of sampling bias, or what random sampling normally produces?

    Hint: Bias pushes results one way because of how people were chosen. Was either sample chosen unfairly?

  6. A student writes: "Our survey cannot be biased, because we asked 1,000 people at the mall about how often they shop." Correct the mistake.

    Hint: Does asking more people from the same place change who is left out?

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

What went wrong with the famous 1936 Literary Digest poll?

The magazine mailed about 10 million ballots, drawing names largely from telephone books, car registrations and its own subscriber lists, and about 2.4 million came back. It predicted Alf Landon would beat Franklin Roosevelt, but Roosevelt won about 61% of the vote. During the Great Depression, people with phones and cars were richer than average, and the people who bothered to mail ballots back differed too. A much smaller poll by George Gallup picked the right winner.

If I pick people at random but half of them never reply, is my sample still fair?

Not necessarily. The people who do reply may differ from those who ignore you, for example by having more free time or stronger opinions. This is called nonresponse bias, and it can tilt results even when the first picks were random. Good surveys follow up with people who did not answer and report the response rate, so readers can judge how much to trust the results.

Is a biased sample always somebody's fault on purpose?

Almost never. Most biased samples come from shortcuts that seem harmless, such as surveying shoppers at a mall on a weekday morning, who are more likely to be retired or to work night shifts, or asking only the students who stay after school. The person may honestly want a fair answer. That is why statisticians judge the method, not the intentions: if some members were more likely to be picked, the sample is biased.

Learning Path

Sampling Bias

You are here

You need Data Collection (surveys and how data are gathered) and Population vs Sample (the whole group and the measured part). Sampling bias explains how a sample can misrepresent its population through the way it was chosen, and why a bigger sample does not cure it. Next, Random Sampling gives the fix, and Data Collection returns with ways to design surveys that avoid these traps.

See Also