Not sure which lesson to do first? Find your level and get a plan →

Math · Statistics & Probability · Grade 9-12

Overfitting (Intuition)

Overfitting happens when a model fits its training data too closely — capturing random noise as if it were real pattern — so it scores great on data it has seen and poorly on data it hasn't.


Quick Answer

Overfitting happens when a model fits its training data too closely — capturing random noise as if it were real pattern — so it scores great on data it has seen and poorly on data it hasn't. Use the idea whenever you build or judge a model, from a trend line to a machine-learning system: hold back some data the model never sees and compare its performance there with its training performance. A large gap means it memorized details instead of learning the pattern; a simpler model or more data usually helps.

Why This Matters

An overfit model looks excellent until it is used. A stock-picking rule tuned to the last five years, or a medical image classifier that learns which hospital's scanner took each picture, can score near perfectly on familiar data and fail on the next case. Judging models on unseen data is what separates a model that has learned from one that only remembers.

Intuitive Explanation

Take 6 points that roughly follow a straight line, with a little noise. A straight line misses each point by a small amount. A fifth-degree polynomial can pass through all 6 points exactly, with zero error, so on these points it is perfect. But to hit every point it must bend sharply between them, and for a new point those bends can put its prediction far from the true line. The extra wiggles fitted the noise, not the pattern.

This is why models are tested on data they have not seen. Split the data: train on 80% and hold back 20%. A well-fit model scores about the same on both, say 92% and 90%. An overfit model scores far better on training than on the held-back data, say 100% and 65%. The signature is the gap, not the training score, and a perfect training score is itself a warning sign, because real data contains noise no honest model can predict.

Overfitting is not the same as doing badly everywhere. A model that scores 60% on training and 58% on new data has memorized nothing; it is too simple, which is underfitting, and the fix runs the other way, toward a richer model. Overfitting is also a property of the whole model, not of one data point. A single odd value is an outlier, though bending a model to pass through outliers is one way to overfit.

Think of it like this. An overfit model is like a student who prepares for a test by memorizing last year's exam, answers included. On last year's questions they score 100%. On a new exam covering the same topics they struggle, because they learned which answer went with which question, not why. The cure is the same for both: practise on questions you have not seen before.

Core idea. Overfitting is when a model learns the noise in its training data and then stumbles on new data.


When to Use

Suspect overfitting when a model scores much better on training data than on new or held-out data, when it has nearly as many adjustable parts as data points, or when its curve wiggles to pass through every point. Compare training and test performance before choosing a model. If the model does badly on both, it is underfitting; if you only need to know how close one model's predictions are on one data set, that is model fit.

Tip. Ask: Does the model do much better on the data it was trained on than on fresh data?

How to Recognize It

Before calling a model overfit, find out whether it was tested on data it never saw, and compare how many adjustable parts it has with how many data points it learned from.

  1. Has the model's error been measured on held-out data as well as on its training data?

    Yes means you can compare the two numbers directly. No means you cannot diagnose overfitting yet; set some data aside or wait for new cases first. A single error score on one data set only describes model fit.

  2. Is the training error low while the error on new data is much higher?

    Yes is overfitting: the model learned details of these particular points. If both errors are high and close together, the model is too simple, which is underfitting. If both are low, the fit is good.

  3. Does the model have nearly as many adjustable parts as there are data points?

    Yes is a warning sign: a polynomial of degree 9 can pass exactly through 10 points, noise included. A model with far fewer parameters than points has less room to memorize and is less likely to overfit.

  4. Is one strange point the only thing the model gets wrong?

    Yes means you are looking at an outlier, a fact about one data value, not overfitting, which is about how the whole model behaves on new data. Check that point for a recording error before changing the model.

Overfitting (Intuition) compared with Underfitting, Model fit (good), and Outlier

Each row explains why predictions and data disagree. Compare the error on training data with the error on new data: low then high is overfitting, high on both is underfitting, low on both is a good fit, and one stray point is an outlier.

Overfitting (Intuition)

Meaning
Use this when a model performs far better on its training data than on new data.
Key test
Does the model do much better on the data it was trained on than on fresh data?
Example
Model A: 98% on training, 96% on new data. Model B: 100% on training, 62% on new data. Which is overfit?

Underfitting

Meaning
The opposite failure: model too simple, poor on both training and new data.
Key test
Use when the model misses the pattern even on its own training data.
Example
A straight line forced onto clearly curved data

Model fit (good)

Meaning
Predictions match the underlying pattern and generalize to new data.
Key test
Use to describe the healthy middle, not either failure.
Example
Line tracks the trend, ignores random wiggles

Outlier

Meaning
A single unusual point, not a whole-model behavior.
Key test
Use when one data value is far from the rest, not when the model itself learned noise.
Example
One typo of 500 in a list of values near 50

Worked Examples

Example 1 — Spot the overfit model

Easy

Model A: 98% on training, 96% on new data. Model B: 100% on training, 62% on new data. Which is overfit?
Show the steps and the answer
  1. Find A's gap: 98%−96%=2 percentage points.

    A small drop from training to new data is normal, since new data carries noise the model has not seen.

  2. Find B's gap: 100%−62%=38 percentage points.

    B does far worse on cases it could not memorize.

  3. Note B's perfect training score.

    Real data contains noise, so a model that gets every training case right has almost certainly fitted that noise.

  4. Choose A for real use: 96% on new data beats 62%.

    Performance on new data is what counts once a model is put to work.

Answer: Model B is overfit

Takeaway: Overfitting shows up as a large drop from training to new data. Judge models by how they do on data they have not seen.

Example 2 — Too simple, not too complex

Standard

A model scores 60% on training AND 58% on new data. Is it overfit?
Show the steps and the answer
  1. Find the gap: 60%−58%=2 percentage points.

    There is no big drop on new data, so the model has not memorized its training set.

  2. Look at the level: only 60% even on the data it was trained on.

    A model that cannot fit its own training data has missed the pattern, which is underfitting.

  3. Choose the fix: make the model richer, for example by allowing a curve or adding input variables.

    Simplifying, the cure for overfitting, would make an underfit model worse.

Answer: Underfit, not overfit

Takeaway: Low scores on both sets mean underfitting. Overfitting needs high training performance with a big drop on new data.

Example 3 — Find the mistake: the lowest training error

Application

Three models predict house prices. Average training errors: straight line $32,000, quadratic $21,000, 9th-degree polynomial $4,000. Average errors on 50 held-back houses: line $34,000, quadratic $23,000, 9th-degree $61,000. Lena writes: "The 9th-degree model has by far the lowest error, so it is the best model." Find the mistake and correct it.
Show the steps and the answer
  1. Check which error Lena used: $4,000 is the training error.

    Training error rewards memorizing, so the most flexible model nearly always looks best there.

  2. Compare the held-back errors: line $34,000, quadratic $23,000, 9th-degree $61,000.

    These houses were not used to build the models, so they show how each model does on new cases.

  3. Find each gap in thousands: line 34−32=2, quadratic 23−21=2, 9th-degree 61−4=57.

    The 9th-degree model's huge gap is the signature of overfitting.

  4. Pick the quadratic: lowest held-back error and a small gap.

    It captures the real curve in the data without chasing noise.

Answer: The quadratic is the best model, with the lowest held-back error ($23,000); the 9th-degree model is overfit.

Takeaway: Choose models by their error on held-out data. The lowest training error usually belongs to the most overfit model.

Where You'll See It

  • Google Flu Trends

    Google Flu Trends predicted flu levels from search terms and matched past seasons well. In the 2012-13 season it predicted about twice the rate of flu-related doctor visits that the CDC recorded, partly because some of its search terms had matched past flu seasons by coincidence rather than because of flu.

  • Testing a spam filter

    A spam filter trained on one company's email might learn that any message mentioning a certain coworker is safe. That rule fits the training emails perfectly and fails as soon as spammers use the name. Email providers test filters on fresh messages they were not trained on to catch rules like this.

  • Predicting next season's results

    A model that explains every game of last season using dozens of details, such as weather, jersey color and day of the week, can match the past perfectly and still predict next season no better than a simple win-rate average. Sports analysts keep models small and test them on seasons they were not built from.

Common Mistakes

  1. The slip: Picking the model with the lowest training error

    Instead: overfit models always win on training; judge on held-out data.

  2. The slip: Adding more parameters or wiggles to chase a better training score

    Instead: extra complexity buys memorization, not generalization.

  3. The slip: Calling any high-error model overfit

    Instead: overfitting is specifically LOW training error with HIGH new-data error.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. Model A: training error 5%, test error 6%. Model B: training error 0%, test error 30%. Model C: training error 25%, test error 26%. Which model is overfit?

    Hint: Look for a big jump from training error to test error.

  2. A student memorizes the answers to 20 practice questions and gets 20 out of 20 on them, but 9 out of 20 on a test with new questions. Give both percentages. How is this like overfitting?

    Hint: Compare the score on seen questions with the score on new ones.

  3. Curves of degree 1, 3 and 9 are fit to 10 points. Their test errors are 8, 3 and 25. Which degree should you choose?

    Hint: Choose by error on data the model did not train on.

  4. A model has 97% training accuracy and 71% accuracy on new data. Find the gap and say what it suggests.

    Hint: Subtract the new-data accuracy from the training accuracy.

  5. Model P has training error 2 and test error 15. Model Q has training error 14 and test error 15. Which is overfit and which is underfit?

    Hint: Compare training error with test error for each model.

  6. Three models have training errors 0,3,5 and test errors 18,6,7. Lee picks the first model because its training error is 0. Correct his choice.

    Hint: Which error tells you how the model does on new data?

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

If more parameters make the training error smaller, why not keep adding them?

Because each new parameter lets the model bend toward the random noise in this particular sample, and the noise will be different next time. Training error keeps falling as you add parameters, while the error on new data falls at first and then starts to rise. The best model sits near the bottom of that second curve, which you can only see if some data are kept away from training.

Does a small data set make overfitting more likely?

Yes. With 8 data points, a model with 6 parameters has almost nothing to learn from beyond the points themselves, so it fits their noise. With 8,000 points, the same model cannot bend to every point, and the random ups and downs tend to cancel. That is why collecting more data is one of the two standard cures, the other being a simpler model.

What is cross-validation, and how does it help?

Cross-validation tests a model on unseen data without wasting any of it. Split the data into, say, 5 equal parts, train on 4 of them and measure the error on the fifth, then repeat until each part has been the test set once. The average of the 5 test errors estimates how the model will do on new data, and a model that wins on training but loses here is overfitting.

Does overfitting happen outside computers and statistics?

Yes, whenever someone builds a rule from too few cases. A coach who notices the team won the three games when she wore a red scarf, and wears it from then on, has fitted a rule to noise. So has an investor who sees that a stock rose every Tuesday last month. Both rules match the past perfectly and say nothing about what happens next.

Learning Path

Overfitting (Intuition)

You are here

You need Model Fit (Intuition) first: residuals, and how to measure how close a model's predictions are. Overfitting adds a second test, fit on data the model has not seen, and explains why a perfect fit on training data is a warning. Next, Underfitting (Intuition) covers the opposite failure, so you can steer between too simple and too complex.

See Also