Non-Record Submission: Random Subspace Optimization

[yangguohao]

Summary

• Introduce a random subspace optimizer based on a simple baseline without any further modifications for learning adapters on Random Linear Maps. The experiments were conducted on 4× Quadro RTX 8000 GPUs for 2000 iterations which couldn't beat the baseline.

Background

This change is aligned with randomized subspace optimization motivation for LLM training efficiency discussed in:

• A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (arXiv:2502.07222)