Underfitting and Overfitting in ML

- Overview

Overfitting and underfitting are two common modeling errors in machine learning (ML) where a model either memorizes training noise or fails to capture basic data patterns. Machine learning (ML) models should learn useful patterns from training data. Underfitting or overfitting occurs when a model learns too little or too much.

Underfitting means the model is too simple to capture all the genuine patterns in the data. Overfitting means the model has learned not only the underlying patterns but also the noise or random outliers present in the training data.

A good model strikes the right balance: it is complex enough to capture genuine patterns but not so complex that it "memorizes" the noise.

1. What is Underfitting? 

Underfitting happens when a machine learning (ML) model is too simple to discover the underlying structure of the data.

• Characteristics: High bias, low variance, and poor performance on both training and test datasets.
• Common Causes: Using models that are too basic (like a straight line for curved data), missing important features, or under-training.

How to Fix:

• Increase model complexity.
• Add more relevant input features.
• Reduce regularization. 

2. What is Overfitting? 

Overfitting occurs when a model learns the training data too well, capturing random noise and minor fluctuations instead of the true trend.

• Characteristics: Low bias, high variance, excellent performance on training data, but poor performance on new, unseen test data. 

• Common Causes: Overly complex models (like high-degree polynomials), too many parameters, or training on a small dataset with lots of noise.

How to Fix:

• Adding regularization (L1/L2).
• Gather more training data.

• Apply early stopping or cross-validation.

- How to Check if the Model is Overfitting or Underfitting? 

When training machine learning (ML) models, data scientists and engineers may face two risks: using an algorithm that is too simple to capture underlying patterns in the data, resulting in underfitting; or using an algorithm that is overly complex, leading to overfitting. Managing overfitting and underfitting is a core challenge in data science workflows and the development of reliable artificial intelligence (AI) systems.

You can check if a machine learning (ML) model is overfitting or underfitting by comparing its performance metrics - such as error or accuracy - between the training data and validation/test data. 

1. Signs of Overfitting: 

Overfitting happens when a model memorizes the training data instead of learning general patterns. It performs great on training data but fails on new data. 

• High Training vs. Test Performance Gap: Training accuracy is very high (or training loss is near zero), but validation/test error is significantly higher.
• Diverging Loss Curves: During training, training loss continues to decrease, but validation loss starts to increase or plateau.
• Brittle Predictions: The model's predictions swing wildly or fail when small, minor changes are made to the input data.

 
2. Signs of Underfitting: 

Underfitting happens when a model is too simple to capture the underlying structure of the data. It performs poorly on both training and validation sets. 

• High Error Across the Board: Both training loss and validation/test loss remain high, or accuracy remains low across both sets.
• Stagnant Improvement: Training and validation metrics stop improving early in the training process and flatline at poor performance levels.

3. How to Test for Fit:

• Holdout Validation: Split your data into a training set and an independent test set to see how the model handles unseen examples.
• Cross-Validation: Use techniques like k-fold cross-validation to ensure your evaluation isn't biased by a single train-test split.
• Learning Curves: Plot training and validation errors over time or across different dataset sizes to visually spot divergence or high baseline error.

- Examples of Overfitting and Underfitting 

Overfitting happens when a machine learning (ML) model learns the training data too well - memorizing its noise and random quirks instead of the true pattern - while underfitting happens when a model is too simple to capture the underlying trend.

1. Overfitting Examples:

• Image Recognition: A model trained mostly on dogs in outdoor parks learns to recognize the green grass behind them rather than the dog itself, so it fails to spot a dog indoors. 
• Medical Diagnosis: An algorithm trained on a tiny set of X-ray scans memorizes tiny imperfections or lighting artifacts in those specific images instead of actual disease symptoms, leading to mistakes on new patient scans.
• Stock Trading: A complex neural network tracks random, historical daily price fluctuations as if they were permanent rules, but loses money when applied to future market conditions.

2. Underfitting Examples:

• Facial Recognition: Using a basic, straight-line linear regression model to identify complex human facial features and pixels results in very low accuracy because the model lacks the capacity to see intricate patterns.
• Curve Fitting: Trying to draw a straight line through data points that curve upward in a U-shape (parabolic data) misses the actual trend entirely, producing high error rates on both training and test sets.

[The Chiba Urban Monorail, Chiba, Japan - Civil Engineering Discoveries]

- How to Avoid Overfitting and Underfitting 

To avoid overfitting (memorizing training noise) and underfitting (failing to learn basic patterns), you must balance model complexity and data quality. 

1. How to Avoid Overfitting (High Variance): 

Overfitting happens when a model is too complex and learns random quirks in the training data instead of real trends.

• Use Regularization: Add penalty terms like L1 or L2 regularization (Lasso or Ridge) to shrink large model coefficients and discourage overly complex weights.
• Apply Early Stopping: Halt training on iterative models or neural networks the moment performance on a validation dataset stops improving.
• Add Dropout: Randomly turn off a percentage of neurons during neural network training to stop layers from co-adapting.

• Get More Data or Augment: Expand your dataset or use data augmentation (like flipping or rotating images) to expose the model to diverse variations.
• Perform Feature Selection: Drop noisy, redundant, or irrelevant input features so the model focuses only on strong signals.
• Use Ensembling: Combine multiple models using methods like bagging or boosting (such as Random Forests) to smooth out individual errors.

2. How to Avoid Underfitting (High Bias): 

Underfitting happens when a model is too simple or poorly trained to uncover the underlying structure of the data.

• Increase Model Complexity: Upgrade from a basic linear model to a more expressive architecture (such as deeper decision trees or neural networks).
• Add Features: Perform feature engineering to create new, informative inputs that better represent the underlying problem.
• Train for More Epochs: Allow optimization algorithms more passes over the training data so they have time to converge on a solution.
• Reduce Regularization: Lower the penalty parameters if your regularization constraint is too strict and is choking the model's capacity to learn.
• Use Transfer Learning: Fine-tune a pre-trained model that already understands generalized features for your specific task.

3. How to Detect Both:

• Cross-Validation: Use K-fold cross-validation to test your model across different subsets of data.
• Performance Comparison: If training accuracy is high but validation/test accuracy is terrible, you have overfitting. If both training and validation accuracy are poor, you have underfitting.

- Achieving the Optimal Model Fit 

An optimal model fit is achieved by successfully navigating the bias-variance tradeoff, balancing model complexity with robust generalization. 

When a model strikes this balance, it learns the true underlying relationships in the training data rather than memorizing random noise or oversimplifying the problem. 

To effectively identify and achieve this sweet spot, data scientists evaluate models across three primary states: 

(A) Underfitting (High Bias): 

1. The Problem: The model is too simple to capture the underlying structure of the data. It makes rigid assumptions about the data distribution. 

2. The Symptoms: Low accuracy and high error rates on both the training data and unseen evaluation data. 

3. The Fixes:

• Increase model complexity (e.g., use a deeper neural network or higher-degree polynomial).
• Engineer and add new domain-specific features.
• Decrease the amount of regularization being applied.
• Train the model for more epochs or training iterations. 

(B) Overfitting (High Variance): 

1. The Problem: The model is too complex relative to the size and noise level of the dataset. It memorizes specific data points and random fluctuations rather than learning general rules. 

2. The Symptoms: High accuracy on training data but poor performance and low accuracy on unseen validation or test data. 

3. The Fixes:

• Simplify the architecture by reducing parameters or pruning.
• Apply regularization techniques (such as L1/L2 regularization or dropout).
• Implement early stopping to halt training before noise is memorized.
• Gather more training data or apply data augmentation. 

(C) Practical Techniques to Find the Sweet Spot: 

Engineers rely on deliberate validation strategy tools rather than guesswork. Using cross-validation ensures the model is consistently tested on different subsets of data. 

Additionally, tracking learning curves - which plot both training error and validation error over time - allows you to see exactly where validation performance peaks before overfitting begins.

- Domain-specific Considerations in Underfitting and Overfitting

Domain knowledge plays a crucial role in addressing underfitting and overfitting, as it enables engineers to customize models based on the specific characteristics of a problem. Real-world datasets often contain noise, imbalances, or inconsistencies.

Here is a breakdown of how domain-specific considerations directly influence underfitting and overfitting across different industries and data types. 

1. Tabular & Financial Data

• The Reality: Data is highly structured but often contains extreme class imbalances (e.g., credit card fraud occurs in less than 0.1% of transactions) or heavy noise (e.g., stock market fluctuations).
• Underfitting Risks: Relying on simple linear models might miss complex, non-linear relationships between financial indicators.
• Overfitting Risks: Over-parameterized models will easily memorize noise or historical anomalies (like a black swan event) rather than learning actual economic trends.
• Domain Fixes: Engineers use domain-driven feature engineering (like computing debt-to-income ratios) and specific metrics like Precision-Recall AUC rather than simple Accuracy to ensure the model doesn't just predict the majority class.

2. Computer Vision & Healthcare:

• The Reality: Medical images (X-rays, MRIs) are expensive to acquire, resulting in small datasets. Furthermore, anomalies (like a rare tumor) are scarce.
• Underfitting Risks: Standard models might fail to capture highly localized, subtle pixel variations that distinguish a benign nodule from a malignant one.
• Overfitting Risks: Because the sample size is small, a deep neural network will easily overfit to the specific imaging equipment, background noise, or hospital-specific lighting conditions instead of the actual pathology.
• Domain Fixes: Transfer learning using architectures pre-trained on massive datasets (like ImageNet) is crucial. Engineers apply domain-specific data augmentation (e.g., slight rotations or contrast adjustments that simulate different X-ray machines) to penalize memorization.

3. Natural Language Processing (NLP) & Legal/Medical Text:

• The Reality: Words and syntax change meaning drastically depending on the context. Legal briefs or clinical notes use highly specialized jargon that general language models do not understand natively.
• Underfitting Risks: Using a generic off-the-shelf model will underfit the specific nuances of the text, completely missing semantic legal clauses or critical drug-to-drug interactions.
• Overfitting Risks: Fine-tuning a massive model on a small set of proprietary legal documents will cause it to memorize exact phrasing, names, or dates rather than legal concepts.
• Domain Fixes: Incorporating domain-specific knowledge graphs or specialized vocabularies (like BioBERT for medicine) helps constrain the model. Regularization techniques like high dropout rates prevent the network from relying too heavily on specific keyword triggers.

4. Time-Series & Physical Systems

• The Reality: Data collected from sensors, weather stations, or IoT devices adheres to physical laws (e.g., thermodynamics, gravity) and temporal dependencies.
• Underfitting Risks: Models that ignore temporal sequence (treating rows as independent) underfit the systemic momentum and seasonal trends inherent in time-series data.
• Overfitting Risks: Standard random cross-validation will cause "data leakage" because future data points bleed into the training set, leading to severe overfitting and artificial high performance that collapses in production.
• Domain Fixes: Implementing Physics-Informed Neural Networks (PINNs) forces the model's loss function to obey mathematical laws, preventing it from predicting physically impossible scenarios. Custom time-series splitting (forward chaining) ensures the model is always validated on future data.

[More to come ...]