Adapters v1.1.0

This version is built for Hugging Face Transformers v4.47.x.

New

Add AdapterPlus adapters (@julian-fong via #746, #775):

AdapterPlus (Steitz & Roth, 2024) is a new bottleneck adapter variant optimized for vision transformers. Check out our notebook for training AdapterPlus adapters for ViT models.

Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):

The newly added save_adapter_setup(), load_adapter_setup() and push_adapter_setup_to_hub() methods allow saving, loading and uploading complex adapter compositions - including AdapterFusion setups - with one line of code. Read our documentation for more.

Enabling full gradient checkpointing support with adapters (@lenglaender via #759)

Gradient checkpointing is a technique for enabling fine-tuning in very memory-limited setttings that nicely complements efficient adapters. It is now supported across all integrated adapter methods. Check out our notebook for fine-tuning Llama with gradient checkpointing and adapters.

More

• Custom names for AdapterFusion layers (@calpt via #774):
  Allows distinguishing multiple fusions on the same adapter by name. See details.
• Allow specifying adapter dtype in AdapterConfig (@killershrimp via #767)

Changed

• Major refactoring of library test suite (@TimoImhof via #740):
  See tests readme for details.
• Upgrade supported Transformers version (@calpt via #757; @lenglaender via #776)

Fixed

• Fix Mistral flash attention with adapters (@divyanshuaggarwal via #758)
• Fix Bottleneck configs to work with ln_before = True and init_weights = "mam_adapter" (@julian-fong via #761)
• Fix default Lora/ (IA)^3 scaling in forward (@calpt via #770) Note: This restores compatibility with adapter-transformers, but changes logic compared to previous Adapters versions. See #770 for details.
• Fixes for sequence generation & orthogonal projection with ReFT (@calpt via #778)
• Various minor compatibility and warning fixes (@calpt via #780, #787)

Adapters v1.0.1

This version is built for Hugging Face Transformers v4.45.x.

New

• Add example notebook for ReFT training (@julian-fong via #741)

Changes

• Upgrade supported Transformers version (@calpt via #751)

Fixed

• Fix import error for Huggingface-hub version >=0.26.0 & Update Notebooks (@lenglaender via #750)
• Fix output_embeddings get/ set for empty active head (@calpt via #754)
• Fix LoRA parallel composition (@calpt via #752)

Adapters v1.0.0

Blog post: https://adapterhub.ml/blog/2024/08/adapters-update-reft-qlora-merging-models

This version is built for Hugging Face Transformers v4.43.x.

New Adapter Methods & Model Support

• Add Representation Fine-Tuning (ReFT) implementation (LoReFT, NoReFT, DiReFT) (@calpt via #705)
• Add LoRA weight merging with Task Arithmetics (@lenglaender via #698)
• Add Whisper model support + notebook (@TimoImhof via #693; @julian-fong via #717)
• Add Mistral model support (@KorventennFR via #609)
• Add PLBart model support (@FahadEbrahim via #709)

Breaking Changes & Deprecations

• Remove support for loading from archived Hub repository (@calpt via #724)
• Remove deprecated add_fusion() & train_fusion() methods (@calpt via #714)
• Remove deprecated arguments in push_adapter_to_hub() method (@calpt via #724)
• Deprecate support for passing Python lists to adapter activation (@calpt via #714)

Minor Fixes & Changes

• Upgrade supported Transformers version (@calpt & @lenglaender via #712, #719, #727)
• Fix SDPA/ Flash attention support for Llama (@calpt via #722)
• Fix gradient checkpointing for Llama and for Bottleneck adapters (@calpt via #730)

Adapters v0.2.2

This version is built for Hugging Face Transformers v4.40.x.

New

• Add example notebook for embeddings training & update docs (@hSterz via #706)

Changed

• Upgrade supported Transformers version (@calpt via #697)
• Add download redirect for AH adapters to HF (@calpt via #704)

Fixed

• Fix saving adapter model with custom heads (@hSterz via #700)
• Fix moving adapter head to device with adapter_to() (@calpt via #708)
• Fix importing encoder-decoder adapter classes (@calpt via #711)

Adapters v0.2.1

This version is built for Hugging Face Transformers v4.39.x.

New

• Support saving & loading via Safetensors and use_safetensors parameter (@calpt via #692)
• Add adapter_to() method for moving & converting adapter weights (@calpt via #699)

Fixed

• Fix reading model info in get_adapter_info() for HF (@calpt via #695)
• Fix load_best_model_at_end=True for AdapterTrainer with quantized training (@calpt via #699)

Adapters v0.2.0

This version is built for Hugging Face Transformers v4.39.x.

New

• Add support for QLoRA/ QAdapter training via bitsandbytes (@calpt via #663): Notebook Tutorial
• Add dropout to bottleneck adapters (@calpt via #667)

Changed

• Upgrade supported Transformers version (@lenglaender via #654; @calpt via #686)
• Deprecate Hub repo in docs (@calpt via #668)
• Switch resolving order if source not specified in load_adapter() (@calpt via #681)

Fixed

• Fix DataParallel training with adapters (@calpt via #658)
• Fix embedding Training Bug (@hSterz via #655)
• Fix fp16/ bf16 for Prefix Tuning (@calpt via #659)
• Fix Training Error with AdapterDrop and Prefix Tuning (@TimoImhof via #673)
• Fix default cache path for adapters loaded from AH repo (@calpt via #676)
• Fix skipping composition blocks in not applicable layers (@calpt via #665)
• Fix Unipelt Lora default config (@calpt via #682)
• Fix compatibility of adapters with HF Accelerate auto device-mapping (@calpt via #678)
• Use default head dropout prob if not provided by model (@calpt via #685)

Adapters v0.1.2

This version is built for Hugging Face Transformers v4.36.x.

New

• Add MT5 support (@sotwi via #629)

Changed

• Upgrade supported Transformers version (@calpt via #617)
• Simplify XAdapterModel implementations (@calpt via #641)

Fixed

• Fix prediction head loading for T5 (@calpt via #640)

Adapters v0.1.1

This version is built for Hugging Face Transformers v4.35.x.

New

• Add leave_out to LoRA and (IA)³ (@calpt via #608)

Fixed

• Fix error in push_adapter_to_hub() due to deprecated args (@calpt via #613)
• Fix Prefix-Tuning for T5 models where d_kv != d_model / num_heads (@calpt via #621)
• [Bart] Move CLS rep extraction from EOS tokens to head classes (@calpt via #624)
• Fix adapter activation with skip_layers/ AdapterDrop training (@calpt via #634)

Docs & Notebooks

• Update notebooks & add new complex configuration demo notebook (@hSterz & @calpt via #614)

Adapters 0.1.0

Blog post: https://adapterhub.ml/blog/2023/11/introducing-adapters/

With the new Adapters library, we fundamentally refactored the adapter-transformers library and added support for new models and adapter methods.

This version is compatible with Hugging Face Transformers version 4.35.2.

For a guide on how to migrate from adapter-transformers to Adapters have a look at https://docs.adapterhub.ml/transitioning.md.
Changes are given compared to the latest adapters-transformers v3.2.1.

New Models & Adapter Methods

• Add LLaMA model integration (@hSterz)
• Add X-MOD model integration (@calpt via #581)
• Add Electra model integration (@hSterz via #583, based on work of @amitkumarj441 and @pauli31 in #400)
• Add adapter output & parameter averaging (@calpt)
• Add Prompt Tuning (@lenglaender and @calpt via #595)
• Add Composition Support to LoRA and (IA)³ (@calpt via #598)

Breaking Changes

• Renamed bottleneck adapter configs and config strings. The new names can be found here: https://docs.adapterhub.ml/overview.html (@calpt)
• Removed the XModelWithHeads classes (@lenglaender) (XModelWithHeads have been deprecated since adapter-transformers version 3.0.0)

Changes Due to the Refactoring

• Refactored the implementation of all already supported models (@calpt, @lenglaender, @hSterz, @TimoImhof)
• Separate the model config (PretrainedConfig) from the adapters config (ModelAdaptersConfig) (@calpt)
• Updated the whole documentation, Jupyter Notebooks and example scripts (@hSterz, @lenglaender, @TimoImhof, @calpt)
• Introduced the load_model function to load models containing adapters. This replaces the Hugging Face from_pretrained function used in the adapter-transformers library (@lenglaender)
• Sharing more logic for adapter composition between different composition blocks (@calpt via #591)
• Added Backwards Compatibility Tests which allow for testing if adaptations of the codebase, such as Refactoring, impair the functionality of the library (@TimoImhof via #596)
• Refactored the EncoderDecoderModel by introducing a new mixin (ModelUsingSubmodelsAdaptersMixin) for models that contain other models (@lenglaender)
• Rename the class AdapterConfigBase into AdapterConfig (@hSterz via #603)

Fixes and Minor Improvements

• Fixed EncoderDecoderModel generate function (@lenglaender)
• Fixed deletion of invertible adapters (@TimoImhof)
• Automatically convert heads when loading with XAdapterModel (@calpt via #594)
• Fix training T5 adapter models with Trainer (@calpt via #599)
• Ensure output embeddings are frozen during adapter training (@calpt #537)

adapter-transformers v.3.2.1

This is the last release of adapter-transformers. See here for the legacy codebase: https://github.com/adapter-hub/adapter-transformers-legacy.

Based on transformers v4.26.1

Fixed

• Fix compacter init weights (@hSterz via #516)
• Restore compatibility of GPT-2 weight initialization with Transformers (@calpt via #525)
• Restore Python 3.7 compatibility (@lenglaender via #510)
• Fix LoRA & (IA)³ implementation for Bart & MBart (@calpt via #518)
• Fix resume_from_checkpoint in AdapterTrainer class (@hSterz via #514)