Overfitting

Definition

When a model performs well on training data but poorly on new, unseen data because it memorized patterns rather than learning generalizable features. Overfitting is a constant risk in fine-tuning and custom model development.

Why it matters

Overfitting is the most common failure mode in custom model training. A model that has memorized its training examples will ace your eval set but fail in production, creating a dangerous false sense of confidence. The risk is especially high when fine-tuning on small datasets, which is exactly the situation most enterprises face: they have hundreds or thousands of examples, not millions. Detecting overfitting requires disciplined train/eval splits and monitoring for divergence between training and evaluation metrics. Preventing it requires regularization techniques, data augmentation, and knowing when to stop training. Teams that skip these steps waste their fine-tuning budget on models that do not generalize.

In practice

In practice, overfitting manifests as a model that perfectly handles scenarios it saw during training but breaks on slight variations. For example, a customer support model fine-tuned on 500 examples might perfectly recite answers to those 500 questions but fail to generalize to the 501st. LoRA mitigates overfitting risk by limiting the number of trainable parameters. Standard practice includes: holding out 10-20% of data for evaluation, monitoring eval loss for divergence from training loss, using early stopping when eval performance plateaus, and testing on fully held-out examples that were not part of any training decision. Dropout and weight decay remain important regularization tools.