This version is built for Hugging Face Transformers v4.47.x.
New
Add AdapterPlus adapters (@julian-fong via #746, #775):
AdapterPlus (Steitz & Roth, 2024) is a new bottleneck adapter variant optimized for vision transformers. Check out our notebook for training AdapterPlus adapters for ViT models.
Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):
The newly added save_adapter_setup(), load_adapter_setup() and push_adapter_setup_to_hub() methods allow saving, loading and uploading complex adapter compositions - including AdapterFusion setups - with one line of code. Read our documentation for more.
Enabling full gradient checkpointing support with adapters (@lenglaender via #759)
Gradient checkpointing is a technique for enabling fine-tuning in very memory-limited setttings that nicely complements efficient adapters. It is now supported across all integrated adapter methods. Check out our notebook for fine-tuning Llama with gradient checkpointing and adapters.
More
- Custom names for AdapterFusion layers (@calpt via #774):
Allows distinguishing multiple fusions on the same adapter by name. See details. - Allow specifying adapter dtype in AdapterConfig (@killershrimp via #767)
Changed
- Major refactoring of library test suite (@TimoImhof via #740):
See tests readme for details. - Upgrade supported Transformers version (@calpt via #757; @lenglaender via #776)
Fixed
- Fix Mistral flash attention with adapters (@divyanshuaggarwal via #758)
- Fix Bottleneck configs to work with ln_before = True and init_weights = "mam_adapter" (@julian-fong via #761)
- Fix default Lora/ (IA)^3 scaling in forward (@calpt via #770) Note: This restores compatibility with adapter-transformers, but changes logic compared to previous Adapters versions. See #770 for details.
- Fixes for sequence generation & orthogonal projection with ReFT (@calpt via #778)
- Various minor compatibility and warning fixes (@calpt via #780, #787)
