Not sure which lesson to do first? Find your level and get a plan →

Math · Statistics & Probability · Grade 6-8

Sampling Bias

Sampling bias occurs when the way a sample is chosen systematically tilts it away from the population — some groups show up too much or too little.


Quick Answer

Sampling bias occurs when the way a sample is chosen systematically tilts it away from the population — some groups show up too much or too little. Use the idea whenever a survey, poll or study reports results about a large group from a smaller one: ask how the people were picked and who that method could not reach. An online poll misses people without internet; a survey at a gym oversamples people who exercise. A biased method gives a wrong answer that a bigger sample only repeats more confidently.

Why This Matters

A biased sample can make a precise-looking result completely wrong, and no amount of careful arithmetic afterwards can repair it. Spotting bias lets you read a poll, a product's star rating or a study and ask who was missing before you trust the number, and it shows you how to design a fair survey of your own.

Intuitive Explanation

Suppose you want the average height of students at your school, and you measure the basketball team because they are easy to find after practice. Every player is measured perfectly, yet the answer comes out too high. Nothing went wrong with the tape measure; the way people were chosen guaranteed tall ones. That is sampling bias: the error is built into who gets picked.

Bias comes in a few common forms. Convenience sampling uses whoever is easy to reach, such as shoppers at one mall at noon on a Tuesday, a group unlikely to include many students or people at work. Voluntary response lets people decide whether to answer, so those with strong feelings reply far more than everyone else. Undercoverage happens when the list you choose from leaves groups out, the way an online poll leaves out people without internet.

Sampling bias is different from random sampling error. Draw 100 names truly at random and you might get 52% support when the true figure is 50%; the next random sample might give 48%. Those misses scatter on both sides and shrink as the sample grows. Bias pushes the same way every time, so a bigger sample from the same flawed method just delivers the wrong answer with more confidence.

Think of it like this. A biased sample is like tasting soup only from the top of a pot nobody has stirred. The salt has sunk to the bottom, so every spoonful from the top tastes bland, and ten more spoonfuls from the top do not help. Only changing where the spoon goes, by stirring or tasting from all through the pot, fixes it.

Core idea. Sampling bias is when the selection method systematically over- or under-represents certain groups in the population.


When to Use

Check for sampling bias whenever results about a population come from a sample and you can see how people were chosen: a mall survey, an online poll, a call-in show, volunteers who chose to respond. Ask which groups that method makes more or less likely to be included. If every member had an equal chance and the result is a little off, that is random sampling error; if the question is whether a sample's mix matches the population, compare proportions using representativeness.

Tip. Ask: Did the selection method systematically include or exclude certain groups?

How to Recognize It

Before you trust a survey result, find out who could possibly have ended up in the sample, who could not, and whether people chose themselves.

  1. Did people decide for themselves whether to answer, as in an online poll or a call-in show?

    If yes, expect voluntary response bias: people with strong feelings answer far more often than the calm majority. If people were picked by a random draw from a full list, this problem goes away.

  2. Could everyone in the population be reached by the method, or were some groups left out from the start?

    If a group was left out, such as students who skip the cafeteria in a lunch-line survey, the sample is biased toward everyone else, however many people are asked.

  3. If the survey were repeated the same way, would the result miss in the same direction again?

    If yes, the error is bias, a tilt built into the method. If repeats land sometimes high and sometimes low around the truth, that is random sampling error, which a larger sample shrinks.

  4. Are you given the population's group shares to compare with the sample's?

    Then you can check representativeness directly: a sample that is 80% grade 8 when the school is one-third grade 8 is off. Bias is about the method that caused such a mismatch.

Sampling Bias compared with Random sampling error, Representativeness, and Noise

Sampling bias, random sampling error, noise and representativeness each explain why a sample can miss the truth, so a flawed method gets excused as bad luck. Ask whether the gap comes from how people were picked, from chance in who was picked, or from wobble in what was measured.

Sampling Bias

Meaning
Use this when you must judge whether the method of choosing a sample systematically skews who is included.
Key test
Did the selection method systematically include or exclude certain groups?
Example
A daytime landline poll on "do people support a new park?" reaches mostly retirees. Is this biased?

Random sampling error

Meaning
Is unbiased chance variation that shrinks with sample size, not a systematic tilt.
Key test
Use when error is random noise, not a directional skew.
Example
A fair sample landing slightly off by luck

Representativeness

Meaning
Is the goal — a sample mirroring the population — that bias undermines.
Key test
Use when describing whether a sample matches the population.
Example
A sample matching the population's age mix

Noise

Meaning
Is random fluctuation in measured values, not a flaw in who is chosen.
Key test
Use when variation is random per observation.
Example
Day-to-day measurement wiggle

Worked Examples

Example 1 — Phone survey timing

Easy

A daytime landline poll on "do people support a new park?" reaches mostly retirees. Is this biased?
Show the steps and the answer
  1. Describe the method: calls to landline phones during working hours.

    Bias is a property of how people were selected, so start with the method, not the answers.

  2. Ask who that method reaches easily: people at home in the daytime who still have a landline, mostly retirees.

    Workers are out during the day, and many younger adults have only mobile phones.

  3. Ask whether the missed groups could feel differently, for example parents of young children who would use a playground in the park.

    Bias matters when the over-included group's views can differ from everyone else's.

  4. Conclude: the method itself over-represents retirees, so the result is tilted in one direction.

    Calling more people the same way would reach more retirees, not fix the tilt.

Answer: Yes — it's sampling bias toward retirees

Takeaway: When a method reaches some groups much more easily than others, the sample is biased before a single answer is recorded.

Example 2 — Random error, not bias

Standard

A truly random sample of residents lands at 52% support when the true value is 50%. Is that sampling bias?
Show the steps and the answer
  1. Check the method: every resident had an equal chance of being chosen.

    Equal chances mean no group was favoured, so the method has no built-in tilt.

  2. Find the miss: 52%−50%=2 percentage points.

    A random sample almost never matches the population exactly, and small misses like this are expected.

  3. Picture repeating it: other random samples might land at 49%, 51% or 53%, scattered on both sides of 50%.

    Random error has no favourite direction, while bias would push every repeat the same way.

  4. Note the cure: a larger random sample would usually land closer to 50%.

    Random error shrinks as the sample grows; bias does not.

Answer: No — that's random sampling error, not bias

Takeaway: A small miss from a fair method is random error. Bias is a miss built into the method, repeating in the same direction.

Example 3 — Find the mistake: 20,000 gamers

Application

A gaming website asks its visitors, "How many hours a week do you play video games?" 20,000 people answer, with an average of 15 hours. Dev writes: "With 20,000 answers this is very accurate, so the average teenager plays about 15 hours a week." Find the mistake and correct it.
Show the steps and the answer
  1. Identify who could answer: only visitors to a gaming website who chose to reply.

    That combines undercoverage, since non-gamers rarely visit, with voluntary response, since keen players are the most likely to answer.

  2. Predict the direction: both effects push the average up.

    Heavy players are over-included and light or non-players are left out, so the tilt is systematic, not random.

  3. Test Dev's argument: 200,000 answers from the same site would come from the same kind of visitor.

    More responses shrink random error but repeat the same tilt, so the result would stay too high.

  4. Fix the method: choose teenagers at random from a full list, such as students across several schools, and ask them.

    When every teenager has an equal chance of being asked, the built-in tilt disappears.

Answer: The 15-hour figure overstates the average teenager's gaming, because the sample came from gamers who chose to reply; a random sample of teenagers is needed, whatever the size.

Takeaway: A large sample cannot cure a biased method. Fix who gets picked, not how many.

Where You'll See It

  • The Literary Digest poll of 1936

    The magazine mailed about 10 million ballots, got roughly 2.4 million back, and predicted Alf Landon would beat Franklin Roosevelt. Roosevelt won with about 61% of the vote. The mailing lists, drawn from phone books and car registrations, leaned wealthy, and Landon supporters were more likely to reply. George Gallup's much smaller, more carefully chosen sample picked the winner.

  • Star ratings for restaurants

    Diners who had a great or a terrible meal are far more likely to post a review than those who had an ordinary one. Because the ratings come from these volunteers, they can make a restaurant look more extreme, better or worse, than the typical diner's experience.

  • Bullet holes on returning bombers

    In World War II, analysts studied damage on bombers that came back from missions. Statistician Abraham Wald pointed out that the sample left out the planes that were shot down. Areas with few holes on returning planes, such as the engines, were where hits were deadly, so those needed extra armour.

Common Mistakes

  1. The slip: Believing a large sample cures bias

    Instead: only fixing the selection method removes a systematic tilt.

  2. The slip: Using convenience samples and generalizing

    Instead: easy-to-reach groups rarely mirror the population.

  3. The slip: Confusing bias with random error

    Instead: bias is a consistent directional skew, not symmetric chance variation.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. To learn how students get to school, Ana surveys the 30 students at a bike-club meeting, and 25 say they bike. Is 2530 a good estimate for the whole school?

    Hint: Who is in the room where she asked?

  2. A gym asks its own members how many days a week they exercise and reports that people in town exercise 4 days a week. Is this biased?

    Hint: Do gym members exercise like everyone else in town?

  3. An online poll finds that 95% of people who answered use the internet every day. Is this a fair estimate for all adults?

    Hint: Who can answer an online poll?

  4. A school is half 6th graders and half 8th graders. A survey of 100 got 80 sixth graders and 20 eighth graders. 70% of 6th graders and 30% of 8th graders want a later start. What does the survey report, and what is a fairer estimate?

    Hint: Weight each grade's percent by its share of the survey, then by its share of the school.

  5. Two random samples of 50 students find 46% and 54% like the new lunch menu. Is the difference caused by sampling bias or by random sampling variation?

    Hint: Is there a reason one method would tilt in one direction?

  6. A radio station gets 10,000 calls about a new law, and 85% oppose it. Max says: "10,000 is huge, so it must be accurate." Correct him.

    Hint: Does a larger number of callers change who chooses to call?

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

What is survivorship bias?

It is sampling bias where the sample holds only the things that made it through some test. In World War II, the statistician Abraham Wald studied damage on bombers returning from missions. Bullet holes clustered on the wings and body, with few near the engines. Wald's point was that planes hit in the engines mostly never came back, so they were missing from the sample, and the armor belonged where the returning planes showed the fewest holes.

Is asking my friends a good way to survey my school?

Usually not. Friends tend to share your classes, hobbies and opinions, so a friend survey overcounts people like you. It is called a convenience sample: easy to collect, but tilted toward whoever is nearby. A fairer approach is to number every student on the school list and let a random number generator choose, so every student, including ones you have never met, has the same chance.

Can the way a question is worded bias a survey?

Yes, even with a perfect random sample. 'Do you agree the cafeteria needs healthier food?' invites a yes, while 'How would you rate the cafeteria food?' does not lean either way. The order of questions matters too: a question about litter can make people rate a park more harshly when it comes first. Strictly this is response bias, not sampling bias, because it comes from how people are asked rather than who is asked.

Learning Path

← Before

Data (Abstract)
Sampling Bias

You are here

You need Data (Abstract) first: the idea of a population and a sample drawn from it. Sampling Bias adds the question of how the sample was chosen and which groups that method tilts toward or away from. Next, Representativeness checks the result directly: do the sample's group proportions match the population's?

See Also