- Home
- /
- Statistics
- /
- Mistakes
- /
- Sampling, Design & Inference
Common Mistakes in Sampling, Design & Inference
76 common pitfalls across 19 concepts, including 19 recurring stuck points
These are the pitfalls students hit across all 19 sampling, design & inference concepts, from Random Sampling, Population vs Sample and Sampling Bias onward. Each entry names the concept it comes from, so you can tell a slip from a genuine gap and go straight to the explanation.
Students often know a procedure related to random sampling but skip the recognition step: Does a chance process, not a person or the volunteers themselves, decide which members of the population end up in the sample? That leads to a calculation or graph that looks reasonable but answers a different question.
'Random' doesn't mean haphazard — it requires a formal process
Substituting when someone is unavailable, breaking the randomization
Using convenience sampling and calling it random
Students often know a procedure related to population vs sample but skip the recognition step: Must you decide whether a number describes everyone in the group you care about or only the members actually measured? That leads to a calculation or graph that looks reasonable but answers a different question.
Thinking sample = population
Using biased samples to generalize
Confusing sample size with accuracy
Students often know a procedure related to sampling bias but skip the recognition step: Would the way these people were chosen push the result in the same direction every time the survey was repeated? That leads to a calculation or graph that looks reasonable but answers a different question.
Convenience sampling (just asking whoever's nearby)
Voluntary response bias
Undercoverage
Students often know a procedure related to experimental design but skip the recognition step: Are you planning a test where you decide who gets a treatment and keep everything else the same between groups? That leads to a calculation or graph that looks reasonable but answers a different question.
No control group
Not randomizing
Changing multiple variables at once
Students often know a procedure related to observational vs experimental studies but skip the recognition step: Did the researchers decide which subjects received the treatment, instead of recording choices the subjects had already made? That leads to a calculation or graph that looks reasonable but answers a different question.
Claiming causation from observational data
Not recognizing study type
Ignoring confounding in observational studies
Students often know a procedure related to confounding variables but skip the recognition step: Is there a third factor, linked to both the explanatory variable and the outcome, that could produce the association by itself? That leads to a calculation or graph that looks reasonable but answers a different question.
Ignoring possible confounders
Assuming correlation means direct causation
Not controlling for confounders
Students often know a procedure related to random assignment but skip the recognition step: Is chance being used to decide which treatment each subject already in the study receives? That leads to a calculation or graph that looks reasonable but answers a different question.
Confusing random assignment with random sampling
Assuming assignment by convenience still supports strong causal claims
Thinking random assignment alone guarantees a perfect experiment
Students often know a procedure related to control group but skip the recognition step: Does the study need a group that skips the new treatment, so you can see what would have happened without it? That leads to a calculation or graph that looks reasonable but answers a different question.
Comparing the treated group to nothing
Using a control group that is not actually comparable to the treatment group
Assuming any untreated group automatically serves as a strong control
Students often know a procedure related to placebo effect but skip the recognition step: Could the outcome improve just because subjects believe they are being treated, even with no active ingredient? That leads to a calculation or graph that looks reasonable but answers a different question.
Assuming reported improvement always comes from the active treatment
Ignoring the role of expectation in human studies
Treating placebo groups as unnecessary when belief can change outcomes
Students often know a procedure related to blinding but skip the recognition step: Could knowing who got which treatment change how subjects respond or how researchers score the results? That leads to a calculation or graph that looks reasonable but answers a different question.
Ignoring observer bias when results depend on judgment or reporting
Assuming random assignment alone removes all experiment bias
Treating single-blind and double-blind designs as interchangeable
Students often know a procedure related to sampling variability but skip the recognition step: Could chance alone explain why fair random samples from the same population gave different results? That leads to a calculation or graph that looks reasonable but answers a different question.
Treating normal sample-to-sample differences as proof of bias
Assuming one good sample reveals the exact population value
Confusing sampling variability with measurement mistakes
Students often know a procedure related to sampling distribution but skip the recognition step: Is the question about how a statistic, such as a sample mean, would be spread across all possible samples of one size? That leads to a calculation or graph that looks reasonable but answers a different question.
Confusing with population distribution
Forgetting spread decreases with sample size
Not understanding variability of estimates
Students often know a procedure related to central limit theorem but skip the recognition step: Do you need a sample mean's distribution to be approximately normal even though the individual values are not? That leads to a calculation or graph that looks reasonable but answers a different question.
Applying to small samples
Thinking it makes data normal (it doesn't)
Confusing CLT with Law of Large Numbers
Students often know a procedure related to standard error but skip the recognition step: Do you need one number for how far a sample mean or proportion typically misses the true value from sample to sample? That leads to a calculation or graph that looks reasonable but answers a different question.
Confusing with standard deviation
Forgetting relationship
Dividing by instead of
Students often know a procedure related to confidence interval but skip the recognition step: Is the goal to report a range of plausible values for an unknown population mean or proportion, built from one sample? That leads to a calculation or graph that looks reasonable but answers a different question.
Thinking 95% CI means 95% of data falls there
Interpreting as probability for one interval
Confusing confidence with probability
Students often know a procedure related to margin of error but skip the recognition step: Do you need just the plus-or-minus amount that says how far a reported estimate could be from the truth by chance? That leads to a calculation or graph that looks reasonable but answers a different question.
Ignoring margin of error in close races
Thinking larger margin means bad survey
Not understanding relationship to sample size
Students often know a procedure related to hypothesis testing but skip the recognition step: Is there a specific claim about a population, such as no effect or a stated value, that the sample data might contradict? That leads to a calculation or graph that looks reasonable but answers a different question.
Confusing null and alternative hypotheses
Thinking failure to reject means the null is true
Not understanding what the test actually tests
Students often know a procedure related to p-value but skip the recognition step: Do you need the chance, assuming the no-effect claim is true, of results at least as extreme as the ones observed? That leads to a calculation or graph that looks reasonable but answers a different question.
Thinking p-value is the probability the null is true (it's not)
Treating as meaningful but as nothing
Ignoring effect size and only looking at p-value
Students often know a procedure related to statistical significance but skip the recognition step: Must you give a yes-or-no verdict by comparing a p-value with a threshold chosen before the data came in? That leads to a calculation or graph that looks reasonable but answers a different question.
Equating statistical significance with practical importance
Using blindly without context
P-hacking: testing many things until something is 'significant'