Empirical Characterizations of Over-parameterization.

Numerical experiments for MIT 6.7910 - Statistical Learning Theory and Applications (Fall 2025). This repository can be used to run experiments on over-parameterization in deep neural networks. It provides functionality to:

• Track optimization metrics during training
• Visualize loss landscapes around converged minima (pytorch checkpoints)
• Analyze training dynamics and loss landscape geometry across different acrhitectures/ extents of over-parameterization.

Project Structure

• [src/]: Core source code

  • [models.py]: Neural network architectures
  • [train.py]: Training loop and metric tracking
  • [data.py]: Data loading and preprocessing
  • [hessian.py]: Estimate the top eigenvalue of the Hessian of the loss w.r.t. model parameters using the power iteration method. Use a single-mini-batch or subsample for the estimation.
  • [metrics.py]: Custom metrics and evaluation
  • [utils.py]: Mischellaneous, to set seed for reproducibility and log results in CSV

• [cli/]: Scripts that can actually be run from the command-line interface or be used to submit a job on an HPC cluster.

  • [run_experiment.py]: Main training script
  • [plot_metrics.py]: To create line graphs of metrics logged during different training runs
  • [plot_direction_profile_1d_distributed.py]
  • [plot_direction_profile_2d.py]
  • [merge_direction_profile_1d.py]
  • [count_parameters.py]

Setup

1. Clone the repository
2. Install dependencies:

   pip install -r requirements.txt
   

Example Usage from CLI

After training a set of models on a HPC cluster with the bash scripts:

python -m cli.plot_metrics --csv runs\cnn\cnn_fulldata_overparam_cifar10_w16_d2_20251205.csv runs\cnn\cnn_fulldata_overparam_cifar10_w32_d2_20251205.csv runs\cnn\cnn_fulldata_overparam_cifar10_w64_d2_20251205.csv runs\cnn\cnn_fulldata_overparam_cifar10_w128_d2_20251205.csv --labels "w=16" "w=32" "w=64" "w=128" --cos_labels "w=16" "w=32" "w=64" "w=128" --outdir plots --cos runs\cnn\cnn_fulldata_overparam_cifar10_w16_d2_20251205_final_cosines.npy runs\cnn\cnn_fulldata_overparam_cifar10_w32_d2_20251205_final_cosines.npy runs\cnn\cnn_fulldata_overparam_cifar10_w64_d2_20251205_final_cosines.npy runs\cnn\cnn_fulldata_overparam_cifar10_w128_d2_20251205_final_cosines.npy

Visualizing the Loss Landscape of Neural Nets

For visualizing the loss landscape around checkpoints, we implement the method of

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. NIPS, 2018.

The citation can be found below:

@inproceedings{visualloss,
  title={Visualizing the Loss Landscape of Neural Nets},
  author={Li, Hao and Xu, Zheng and Taylor, Gavin and Studer, Christoph and Goldstein, Tom},
  booktitle={Neural Information Processing Systems},
  year={2018}
}