Overfitting (Intuition) Math Example 1

Follow the full solution, then compare it with the other examples linked below.

Example 1

medium
A model fits 10 data points with a degree-9 polynomial (perfect fit, R2=1R^2=1). A simpler linear model has R2=0.85R^2=0.85. Explain which model is better for prediction and why.

Solution

  1. 1
    Degree-9 polynomial: 10 parameters for 10 points — fits every point exactly (interpolates), but captures noise not just signal
  2. 2
    Linear model: 2 parameters — fits general trend, ignoring noise
  3. 3
    For prediction: the polynomial will perform poorly on new data (it memorized the training data's random fluctuations)
  4. 4
    The linear model captures the true underlying relationship better, generalizing to new data despite lower in-sample R2R^2

Answer

The linear model (R²=0.85) is better for prediction; the polynomial overfits noise and won't generalize.
Overfitting occurs when a model is too complex relative to the data. A perfect in-sample fit often means the model learned random noise rather than the true signal. Model selection requires balancing fit and complexity using cross-validation or penalty criteria (AIC, BIC).

About Overfitting (Intuition)

Overfitting occurs when a model learns the noise in training data instead of just the underlying pattern, performing well on training data but poorly on new data.

Learn more about Overfitting (Intuition) →

More Overfitting (Intuition) Examples

Example 2 hard

A machine learning model is trained on 1000 observations with 50 predictors. Training error is near

Example 3 easy

A student memorizes all 500 practice problems but performs poorly on the exam, which has new problem

Example 4 hard

Explain the bias-variance tradeoff: how does increasing model complexity affect bias and variance, a

← All Overfitting (Intuition) Examples Overfitting (Intuition) Concept