Overfitting (Intuition) Math Example 2
Follow the full solution, then compare it with the other examples linked below.
Example 2
hardA machine learning model is trained on 1000 observations with 50 predictors. Training error is near zero; test error on 200 held-out observations is very high. Diagnose the problem and suggest two remedies.
Solution
- 1 Diagnosis: overfitting — 50 predictors for 1000 observations is a high model complexity; model memorized training data
- 2 Sign: training error ≈ 0 but high test error — classic overfitting signature
- 3 Remedy 1: regularization (L1/Lasso or L2/Ridge) — penalizes large coefficients, shrinking less important predictors toward zero
- 4 Remedy 2: reduce predictors — use feature selection or domain knowledge to eliminate irrelevant predictors; also collect more training data
Answer
Overfitting: too many predictors for dataset size. Fix with regularization or predictor reduction.
The ratio of predictors to observations matters greatly. As a rule of thumb, each predictor should have at least 10-20 observations. Regularization is the standard remedy in machine learning, adding a penalty for complexity to prevent memorizing noise.
About Overfitting (Intuition)
Overfitting occurs when a model learns the noise in training data instead of just the underlying pattern, performing well on training data but poorly on new data.
Learn more about Overfitting (Intuition) →More Overfitting (Intuition) Examples
Example 1 medium
A model fits 10 data points with a degree-9 polynomial (perfect fit, [formula]). A simpler linear mo
Example 3 easyA student memorizes all 500 practice problems but performs poorly on the exam, which has new problem
Example 4 hardExplain the bias-variance tradeoff: how does increasing model complexity affect bias and variance, a