In 2019, Nakkiran et al. from OpenAI published a paper titled "Deep Double Descent: Where Bigger Models and More Data Hurt" that challenged a fairly fundamental assumption in machine learning. The classical view says that as a model trains longer or grows larger, at some point it starts overfitting and test performance degrades. Nakkiran et al. showed that if you keep going past that point, performance can actually recover. This is the double descent phenomenon, and it shows up across model size, dataset size, and crucially for this project, training time.
The epoch-wise version is the least discussed of the three but arguably the most surprising. A model trained on corrupted labels will, at some point during training, appear to be getting worse on the test set. If you stopped there you would call it overfitting and move on. But if you keep training, the model recovers. This project tries to reproduce that finding on CIFAR-10 with a ResNet-18, understand what is mechanistically happening during the peak, and test whether two common regularization techniques can suppress it.
The honest summary upfront is that the double descent phenomenon did show up, but only clearly at the two highest noise levels. The mechanistic story from Figure 3 is the strongest result in the project. The intervention experiments produced one clean rejection and one ambiguous partial confirmation. Not every hypothesis was right, and where things did not go as predicted that is documented honestly here.
double_descent/
├── core/
│ ├── data.py -- NoisyCIFAR10 dataset with noise injection and index tracking
│ └── metrics.py -- Peak detection and seed aggregation utilities
├── experiments/
│ ├── train.py -- Main training loop
│ └── sweep.py -- Multi-seed sweep across all noise levels and interventions
├── models/
│ └── resnet.py -- CIFAR-adapted ResNet-18
├── checkpoints/ -- Saved model states every 10 epochs
├── results/
│ ├── figures/ -- All six output plots
│ └── summary_table.csv -- Peak statistics per noise level
└── notebooks/
└── analysis.ipynb -- All plotting and analysis code
The project runs on Python 3.10 or higher. Install dependencies with:
pip install torch torchvision numpy matplotlib seaborn pandas tqdm scikit-learnTraining was done on a CUDA GPU. You can verify your setup with:
import torch
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))To reproduce the full sweep from scratch:
python experiments/sweep.pyThe sweep has skip logic built in so if it crashes or gets interrupted it will pick up from the last completed checkpoint rather than starting over. Checkpoints are saved every 10 epochs.
CIFAR-10 is a standard benchmark of 60,000 32x32 color images across 10 classes, with 50,000 for training and 10,000 for testing. It downloads automatically via torchvision.
Symmetric label noise is injected during training only. For each training sample, with probability η the true label is replaced with a uniformly random label drawn from the other nine classes. This follows the setup in Nakkiran et al. (2019). The indices of all corrupted samples are tracked throughout training so that clean and corrupted sample accuracy can be measured separately at every epoch. The test set always uses clean labels.
Noise levels tested were 0%, 10%, 20%, and 30%.
The ResNet-18 used here is adapted from the standard ImageNet version. The original 7x7 convolution with stride 2 was replaced with a 3x3 convolution with stride 1, and the maxpool layer was replaced with an identity operation. This preserves spatial resolution for the small 32x32 inputs of CIFAR-10. The model has approximately 11 million parameters, which makes it overparameterized relative to the 50,000 training samples. This is intentional since overparameterization is what places the model in the regime where double descent can occur.
Training used SGD with momentum 0.9, weight decay 5e-4, Nesterov momentum enabled, and an initial learning rate of 0.1. The learning rate followed a cosine annealing schedule over the full training run.
| Noise Rate | Epochs |
|---|---|
| 0% | 300 |
| 10% | 400 |
| 20% | 500 |
| 30% | 600 |
This is the central result. Test loss over training epochs is plotted for all four noise levels, averaged across three seeds, with shaded bands showing one standard deviation.
Starting with what worked cleanly. The η=30% curve shows a typical double descent pattern. From around epoch 350 onwards, test loss climbs steadily, peaks around epoch 473, and then recovers. The corresponding accuracy curve in the right panel drops from around 80% down to roughly 65% before bouncing back to 79%. This is reproducible across all three seeds given the tight confidence bands.
The η=20% curve also shows a bump, peaking around epoch 385, though it is softer and less dramatic than the 30% case. It is real, but it is not as visually obvious as the 30% result.
The η=10% case is where things get less clean. The curve largely decreases throughout training with only minor instability. There is no obvious double descent peak visible in the curve. The summary table identifies a detected peak at epoch 37, but this is almost certainly a small fluctuation in the early noisy phase of training rather than a genuine interpolation threshold. At 10% noise the phenomenon is very weak, and claiming a clear double descent there would be overstating what the data actually shows.
The η=0% case behaves exactly as classical theory predicts. Monotonic decrease, clean convergence, no instability.
The summary table below gives the quantitative peak statistics.
| Noise | Peak Epoch | Peak Loss | Peak Height | Recovery Epoch | Final Loss | Final Acc |
|---|---|---|---|---|---|---|
| 0% | N/A (no descent) | N/A | N/A | N/A | 0.1718 | 95.52% |
| 10% | N/A (no descent) | N/A | N/A | N/A | 0.3513 | 91.22% |
| 20% | 385 | 0.8559 | 0.2191 | 388 | 0.5716 | 85.50% |
| 30% | 473 | 1.2760 | 0.4999 | 474 | 0.8833 | 78.45% |
The 0% and 10% peak detections are not reliable findings. The peak detection algorithm finds the highest point in the smoothed curve after the first 30 epochs, and at low noise levels that just picks up a minor bump rather than a genuine double descent peak. The 20% and 30% results are genuine. Peak height nearly doubles going from 20% to 30% noise (0.22 to 0.50), which is consistent with the idea that higher noise creates a more severe memorization phase.
This figure shows train and test loss side by side for each noise level with the generalization gap shaded between them.
For η=0% the gap is small and grows slowly, which is ordinary generalization behavior. For η=20% and η=30% the story is completely different. Train loss drops toward zero in the later epochs while test loss is simultaneously rising. This is the memorization phase made explicit. The model is fitting the corrupted training labels perfectly at the exact moment it is generalizing worst to the clean test distribution. The gap then closes during recovery as the model finds a flatter solution where reducing train loss further is correlated with test loss also decreasing.
This is the strongest figure in the project and the result worth spending the most time on.
Clean training accuracy (blue) and corrupted training accuracy (red) are plotted separately over epochs for each noise level. The corrupted accuracy curves follow an almost perfect sigmoid in every panel. For hundreds of epochs corrupted accuracy sits near zero. The model is learning the underlying signal in the data and largely ignoring the corrupted labels. Then at some point it crosses the interpolation threshold and begins memorizing the noise. The sigmoid fires and corrupted accuracy climbs rapidly toward 100%.
The timing of this sigmoid is what makes the figure compelling. The epoch at which corrupted accuracy inflects corresponds closely to the epoch at which test loss peaks in Figure 1. The model memorizes the noise, test performance degrades, and then cosine annealing decays the learning rate enough that the model can no longer maintain the sharp noise-fitting solution and drifts toward a more generalizable one.
Higher noise delays the entire sequence. At η=10% the sigmoid fires around epoch 250. At η=20% it fires around epoch 400. At η=30% it fires around epoch 500. The interpolation threshold is harder to cross when more labels are wrong.
One caveat worth noting. The peak marker shown as a dotted vertical line in this figure is labeled "ep 3" which is incorrect. This was a bug in the peak detection algorithm that was fixed before the summary table was finalized. The visual alignment between the corrupted accuracy inflection and the actual test loss peaks in Figure 1 is still valid and can be confirmed by comparing the two figures directly. This was not fixed due to the extremely expensive compute of sweep.py
Mean softmax entropy was measured on the test set throughout training. Lower entropy means higher confidence predictions.
All four curves start high since the model has barely learned anything in the first few epochs. The clean case drops entropy quickly and keeps decreasing throughout training, converging near zero by epoch 300. The noisy cases behave differently. After the initial drop, entropy plateaus at a level that scales with the noise rate. At η=30%, entropy stays around 1.4 to 1.5 for hundreds of epochs before dropping sharply during the recovery phase.
The original hypothesis was that entropy would dip specifically at the peak epoch, reflecting overconfidence at the interpolation threshold. This did not happen. Entropy stays elevated through the entire memorization phase and only collapses during recovery. The hypothesis was wrong in its specifics. What the data actually shows is a model that is confident about training labels (corrupted accuracy reaches 100%) while remaining uncertain about test predictions (entropy stays high). This is a picture of memorization rather than generalization. The entropy drop during recovery marks the point at which the model reforms representations that are actually predictive on unseen data. The hypothesis got the timing wrong but the underlying signal is real and meaningful.
This figure compares the baseline (η=20%, no intervention) against label smoothing with ε=0.1 and mixup with α=0.2, all run for 500 epochs across three seeds.
The label smoothing result is a clear failure relative to the prediction. The smoothing curve sits above the baseline for almost the entire run and ends at a final test loss of around 0.87 compared to the baseline's 0.57. Label smoothing did not suppress double descent. It degraded overall performance significantly. At ε=0.1, smoothing appears to prevent the late-training recovery that produces good final performance, possibly by fighting against the cosine annealing schedule in a way that keeps the model perpetually uncertain. There is no visible peak in the smoothed curve, but this is not because smoothing prevented the interpolation dynamics. The model simply never recovers well regardless. This is a rejection of H2.
The mixup result is more nuanced. Mixup tracks the baseline quite closely through most of training and ends with a better final loss of around 0.48, which is a genuine improvement over the baseline's 0.57. The peak region looks slightly smoother for mixup compared to the baseline but the difference is not dramatic. Mixup's main benefit here is in final performance rather than peak suppression. It is not preventing the model from entering the memorization phase, it is helping it recover to a better solution. Whether this counts as suppressing double descent depends on how strictly you define the term. The honest answer is that mixup helped overall but did not do what was specifically predicted.
This figure examines per-class test accuracy at six key epochs during training at η=20%, using seed 42.
Cat is consistently the hardest class. It bottoms out at 0.58 at epoch 150, right in the period when the model is beginning to memorize noise most aggressively. Cat has always been hard on CIFAR-10 even under clean labels because of its visual similarity to dog and other animal classes. Under noisy labels this problem is compounded. Car and frog are relatively robust throughout training. These classes appear visually distinctive enough that the model retains reasonable generalization even during the memorization phase.
By epoch 500 most classes recover to reasonable accuracy with cat still the weakest at 0.76. The aggregate test loss curves in Figure 1 average over quite different per-class dynamics, and this heatmap is a useful reminder of that. Whether double descent dynamics are concentrated in the harder classes or uniform across all classes is an interesting question this project does not fully answer with only one noise level shown here.
| Hypothesis | Prediction | What Was Observed | Verdict |
|---|---|---|---|
| H1: higher noise shifts peak later | Higher η means later peak epoch | Clear peaks at ep 385 for η=20% and ep 473 for η=30%, no convincing peak at 0% or 10% | Confirmed at high noise only |
| H2: label smoothing reduces peak magnitude | Peak height decreases with smoothing | Smoothing degraded overall performance significantly, final loss 0.87 vs baseline 0.57 | Rejected |
| H3: mixup suppresses double descent | Peak nearly disappears with mixup | Better final performance (0.48 vs 0.57) but peak not clearly eliminated | Partially confirmed with caveats |
| H4: memorization peaks at double descent peak | Corrupted train accuracy peaks at the same epoch as test loss | Corrupted accuracy sigmoid inflects at approximately the test loss peak epoch across all noise levels | Confirmed, strongest result in the project |
| H5: entropy dips at the peak | Minimum entropy near peak epoch | Entropy stays elevated through memorization phase, drops during recovery not at peak | Hypothesis was wrong on timing, signal is real but marks a different thing |
Why does recovery happen at all
The most plausible explanation is cosine annealing. As the learning rate decays, the optimizer loses the ability to maintain sharp noise-fitting solutions. Early in training there is enough gradient signal to memorize individual corrupted labels. As the learning rate approaches zero these sharp minima become unreachable and the model drifts toward flatter regions of the loss landscape that generalize better. Figure 2 supports this interpretation because the generalization gap closes right as the learning rate enters its final decay phase. It is worth being clear that this explanation is interpretive rather than proven. Whether recovery would occur with a constant learning rate is one of the more interesting open questions.
Why does higher noise delay the peak
A higher fraction of corrupted labels means the optimizer has to work harder to overpower the clean signal. With 30% of training labels wrong, it takes considerably more epochs to begin memorizing noise. This pushes the interpolation threshold further into training which is exactly what Figures 1 and 3 show. The trend is consistent across all four noise levels tested, including the absence of a visible peak at 0% and 10% which itself is informative. Within this specific setup the direction of the relationship is clear, though whether the same quantitative pattern holds across other architectures and datasets remains an open question.
Why did label smoothing fail here
The intuition was that softening the target distribution would prevent overconfidence on corrupted labels. In practice at ε=0.1, smoothing prevented the model from becoming confident about anything including correct predictions. The result is a consistently elevated loss that does not recover well in the long run. This experiment tests one specific configuration and cannot speak to label smoothing in general, but at the strength tested it was clearly not the right tool for this setting.
What the entropy finding actually means
During the memorization phase the model is confident about training labels (corrupted accuracy reaches 100%) but uncertain about test predictions (entropy stays high). This is a coherent picture of a model that has memorized specific training examples rather than learned generalizable features. The entropy collapse during recovery corresponds to the model reforming representations that transfer to unseen data. The original hypothesis was wrong about when entropy would drop but the entropy signal is still informative and worth reporting accurately rather than trying to frame it as a confirmation of something it is not.
This project is entirely empirical. The mechanisms discussed above are plausible interpretations of the data and not formal derivations.
Double descent was only clearly observable at 20% and 30% noise. The 10% result did not produce a convincing peak, which limits how strongly H1 can be stated. If the claim is "higher noise shifts the peak later" then the cleaner claim is "high enough noise produces a visible peak, and that peak shifts later with more noise."
Only one dataset and one architecture were tested. CIFAR-10 with ResNet-18 is a reasonable benchmark but there is no guarantee that the same dynamics appear in other settings.
Only one regularization strength was tested for each intervention. The label smoothing rejection applies to ε=0.1 specifically. The mixup result applies to α=0.2 specifically.
Three seeds per configuration produces reasonable error bars but is a small sample. The confidence bands are tight which is encouraging, but three seeds would not catch rare outlier behaviors.
The peak detection algorithm produced incorrect annotations in Figure 3 (showing "ep 3" as the peak location for all panels). This was fixed in the summary table but the figure itself retains the incorrect markers. This should not affect interpretation since the visual signal in Figure 3 is clear regardless of the annotation.
Several directions seem worth pursuing if this project were to continue.
Comparing SGD with Adam would test whether optimizer choice shifts the peak location, since Adam's adaptive learning rates might prevent the memorization dynamics from developing in the same way.
Running experiments with a constant learning rate would directly test whether cosine annealing is necessary for recovery, which would either confirm or challenge the explanation offered in the discussion section.
A finer noise sweep at 5%, 15%, and 25% would show whether the onset of observable double descent is gradual or appears relatively suddenly past some threshold noise rate.
Testing different regularization strengths for label smoothing and mixup would give a more complete picture of the intervention results rather than a single data point per method.
The most technically interesting extension would be interpolating model weights between the peak checkpoint and the final checkpoint and plotting loss along the path. This would make the geometry of the loss landscape concrete rather than inferred from training curves.
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2019). Deep Double Descent: Where Bigger Models and More Data Hurt. OpenAI. arXiv:1912.02292.
Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2021). Understanding Deep Learning Requires Rethinking Generalization. ICLR.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. arXiv:1710.09412.
Muller, R., Kornblith, S., and Hinton, G. (2019). When Does Label Smoothing Help. NeurIPS.
Personal research project