Common Mistakes in Sampling, Design & Inference

76 common pitfalls across 19 concepts, including 19 recurring stuck points

These are the pitfalls students hit across all 19 sampling, design & inference concepts, from Random Sampling, Population vs Sample and Sampling Bias onward. Each entry names the concept it comes from, so you can tell a slip from a genuine gap and go straight to the explanation.

1
Random Sampling
Stuck Point

Students often know a procedure related to random sampling but skip the recognition step: Does a chance process, not a person or the volunteers themselves, decide which members of the population end up in the sample? That leads to a calculation or graph that looks reasonable but answers a different question.

2

'Random' doesn't mean haphazard — it requires a formal process

3

Substituting when someone is unavailable, breaking the randomization

4

Using convenience sampling and calling it random

5

Students often know a procedure related to population vs sample but skip the recognition step: Must you decide whether a number describes everyone in the group you care about or only the members actually measured? That leads to a calculation or graph that looks reasonable but answers a different question.

6

Thinking sample = population

7

Using biased samples to generalize

8

Confusing sample size with accuracy

9
Sampling Bias
Stuck Point

Students often know a procedure related to sampling bias but skip the recognition step: Would the way these people were chosen push the result in the same direction every time the survey was repeated? That leads to a calculation or graph that looks reasonable but answers a different question.

10

Convenience sampling (just asking whoever's nearby)

11

Voluntary response bias

12

Undercoverage

13

Students often know a procedure related to experimental design but skip the recognition step: Are you planning a test where you decide who gets a treatment and keep everything else the same between groups? That leads to a calculation or graph that looks reasonable but answers a different question.

14

No control group

15

Not randomizing

16

Changing multiple variables at once

17

Students often know a procedure related to observational vs experimental studies but skip the recognition step: Did the researchers decide which subjects received the treatment, instead of recording choices the subjects had already made? That leads to a calculation or graph that looks reasonable but answers a different question.

18

Claiming causation from observational data

19

Not recognizing study type

20

Ignoring confounding in observational studies

21

Students often know a procedure related to confounding variables but skip the recognition step: Is there a third factor, linked to both the explanatory variable and the outcome, that could produce the association by itself? That leads to a calculation or graph that looks reasonable but answers a different question.

22

Ignoring possible confounders

23

Assuming correlation means direct causation

24

Not controlling for confounders

25
Random Assignment
Stuck Point

Students often know a procedure related to random assignment but skip the recognition step: Is chance being used to decide which treatment each subject already in the study receives? That leads to a calculation or graph that looks reasonable but answers a different question.

26

Confusing random assignment with random sampling

27

Assuming assignment by convenience still supports strong causal claims

28

Thinking random assignment alone guarantees a perfect experiment

29
Control Group
Stuck Point

Students often know a procedure related to control group but skip the recognition step: Does the study need a group that skips the new treatment, so you can see what would have happened without it? That leads to a calculation or graph that looks reasonable but answers a different question.

30

Comparing the treated group to nothing

31

Using a control group that is not actually comparable to the treatment group

32

Assuming any untreated group automatically serves as a strong control

33
Placebo Effect
Stuck Point

Students often know a procedure related to placebo effect but skip the recognition step: Could the outcome improve just because subjects believe they are being treated, even with no active ingredient? That leads to a calculation or graph that looks reasonable but answers a different question.

34

Assuming reported improvement always comes from the active treatment

35

Ignoring the role of expectation in human studies

36

Treating placebo groups as unnecessary when belief can change outcomes

37
Blinding
Stuck Point

Students often know a procedure related to blinding but skip the recognition step: Could knowing who got which treatment change how subjects respond or how researchers score the results? That leads to a calculation or graph that looks reasonable but answers a different question.

38
Blinding
Mistake

Ignoring observer bias when results depend on judgment or reporting

39
Blinding
Mistake

Assuming random assignment alone removes all experiment bias

40
Blinding
Mistake

Treating single-blind and double-blind designs as interchangeable

41

Students often know a procedure related to sampling variability but skip the recognition step: Could chance alone explain why fair random samples from the same population gave different results? That leads to a calculation or graph that looks reasonable but answers a different question.

42

Treating normal sample-to-sample differences as proof of bias

43

Assuming one good sample reveals the exact population value

44

Confusing sampling variability with measurement mistakes

45

Students often know a procedure related to sampling distribution but skip the recognition step: Is the question about how a statistic, such as a sample mean, would be spread across all possible samples of one size? That leads to a calculation or graph that looks reasonable but answers a different question.

46

Confusing with population distribution

47

Forgetting spread decreases with sample size

48

Not understanding variability of estimates

49

Students often know a procedure related to central limit theorem but skip the recognition step: Do you need a sample mean's distribution to be approximately normal even though the individual values are not? That leads to a calculation or graph that looks reasonable but answers a different question.

50

Applying to small samples

51

Thinking it makes data normal (it doesn't)

52

Confusing CLT with Law of Large Numbers

53
Standard Error
Stuck Point

Students often know a procedure related to standard error but skip the recognition step: Do you need one number for how far a sample mean or proportion typically misses the true value from sample to sample? That leads to a calculation or graph that looks reasonable but answers a different question.

54

Confusing with standard deviation

55

Forgetting n relationship

56

Dividing by n instead of n

57

Students often know a procedure related to confidence interval but skip the recognition step: Is the goal to report a range of plausible values for an unknown population mean or proportion, built from one sample? That leads to a calculation or graph that looks reasonable but answers a different question.

58

Thinking 95% CI means 95% of data falls there

59

Interpreting as probability for one interval

60

Confusing confidence with probability

61
Margin of Error
Stuck Point

Students often know a procedure related to margin of error but skip the recognition step: Do you need just the plus-or-minus amount that says how far a reported estimate could be from the truth by chance? That leads to a calculation or graph that looks reasonable but answers a different question.

62

Ignoring margin of error in close races

63

Thinking larger margin means bad survey

64

Not understanding relationship to sample size

65
Hypothesis Testing
Stuck Point

Students often know a procedure related to hypothesis testing but skip the recognition step: Is there a specific claim about a population, such as no effect or a stated value, that the sample data might contradict? That leads to a calculation or graph that looks reasonable but answers a different question.

66

Confusing null and alternative hypotheses

67

Thinking failure to reject means the null is true

68

Not understanding what the test actually tests

69
P-Value
Stuck Point

Students often know a procedure related to p-value but skip the recognition step: Do you need the chance, assuming the no-effect claim is true, of results at least as extreme as the ones observed? That leads to a calculation or graph that looks reasonable but answers a different question.

70
P-Value
Mistake

Thinking p-value is the probability the null is true (it's not)

71
P-Value
Mistake

Treating p=0.049 as meaningful but p=0.051 as nothing

72
P-Value
Mistake

Ignoring effect size and only looking at p-value

73

Students often know a procedure related to statistical significance but skip the recognition step: Must you give a yes-or-no verdict by comparing a p-value with a threshold chosen before the data came in? That leads to a calculation or graph that looks reasonable but answers a different question.

74

Equating statistical significance with practical importance

75

Using α=0.05 blindly without context

76

P-hacking: testing many things until something is 'significant'