FEAT add GraLoRA #2851

yeonjoon-jung01 · 2025-10-20T01:24:36Z

Summary

We opened the initial PR for the GraLoRA method (a granular low-rank adaptation that improves expression power and outlier handling, selected as a NeurIPS 2025 Spotlight), based on #2636

Test Codes

pytest tests/test_gralora.py -v
pytest tests/test_config.py -k "Gralora" -v
pytest tests/test_decoder_models.py -k "Gralora" -v
pytest tests/test_custom_models.py -k "Gralora" -v

BenjaminBossan

Thanks for contributing GraLoRA to PEFT. The method looks interesting and the implementation generally looks good.

I have added a bunch of comments, but many of these are just due to your fork being a bit older. We have simplified PEFT now so that you can remove a bunch of code, I have marked the code that can be deleted.

Apart from the comments that I added, to complete this PR, let's work on:

Extend tests: Add tests to test_custom_models.py, test_encoder_decoder_models.py, test_feature_extraction_models.py, and test_seq_classifier.py
Also, let's add documentation and ideally also at least one example.
Optional, but highly recommended: Add an experiment to our PEFT method comparison suite.

BenjaminBossan · 2025-10-22T12:04:10Z

src/peft/tuners/gralora/config.py

+
+@dataclass
+class GraloraConfig(PeftConfig):
+    r: int = field(default=8, metadata={"help": "gralora attention dimension"})


The description is not helpful IMO, please give a more complete description.

I’ve extended the description for better clarity.

BenjaminBossan · 2025-10-22T12:05:01Z

src/peft/tuners/gralora/config.py

+            )
+        },
+    )
+    gralora_alpha: int = field(default=8, metadata={"help": "gralora alpha"})


Same, let's extend the description

I’ve extended the description.

BenjaminBossan · 2025-10-22T12:05:05Z

src/peft/tuners/gralora/config.py

+    )
+    gralora_alpha: int = field(default=8, metadata={"help": "gralora alpha"})
+    gralora_dropout: float = field(default=0.0, metadata={"help": "gralora dropout"})
+    gralora_k: int = field(default=2, metadata={"help": "gralora k"})


Same, let's extend the description

I’ve extended the description.

BenjaminBossan · 2025-10-22T12:06:18Z

src/peft/tuners/gralora/config.py

+class GraloraConfig(PeftConfig):
+    r: int = field(default=8, metadata={"help": "gralora attention dimension"})
+    hybrid_r: int = field(
+        default=0, metadata={"help": "hybrid_r is the rank allocated to vanilla LoRA method when using Hybrid GraLoRA"}


IIUC, by passing a value > 0, hybrid GraLoRA is enabled. Let's make that clear.

I’ve extended the description, making it more clear.

BenjaminBossan · 2025-10-22T12:06:56Z

src/peft/tuners/gralora/layer.py

+        self.scaling = {}
+        self.gralora_dropout = nn.ModuleDict({})
+
+        # Set to `None` otherwise to avoid computation with random weight


Removed the wrong code comment.

BenjaminBossan · 2025-10-22T13:18:31Z

src/peft/tuners/gralora/model.py

+            config_dict[key] = config
+        return config_dict
+
+    def _set_adapter_layers(self, enabled=True):


Same as parent class, can be removed.

I’ve removed the inherited methods from the parent class.

BenjaminBossan · 2025-10-22T13:18:34Z

src/peft/tuners/gralora/model.py

+            if isinstance(module, (BaseTunerLayer, ModulesToSaveWrapper)):
+                module.enable_adapters(enabled)
+
+    def enable_adapter_layers(self):


Same as parent class, can be removed.

I’ve removed the inherited methods from the parent class.

BenjaminBossan · 2025-10-22T13:18:43Z

src/peft/tuners/gralora/model.py

+    def enable_adapter_layers(self):
+        self._set_adapter_layers(enabled=True)
+
+    def disable_adapter_layers(self):


Same as parent class, can be removed.

I’ve removed the inherited methods from the parent class.

BenjaminBossan · 2025-10-22T13:18:51Z

src/peft/tuners/gralora/model.py

+
+        self.active_adapter = new_adapter or []
+
+    def merge_and_unload(


Same as parent class, can be removed.

I’ve removed the inherited methods from the parent class.

BenjaminBossan · 2025-10-22T14:53:10Z

tests/test_gralora.py

Thanks a lot for very diligently adding unit tests for GraLoRA. However, I'm afraid that we can delete the majority of them :)

Most of what is tested here is already covered in existing tests (merging, disabling adapters, different dtypes, ...). This is especially true for the tests in test_custom_models.py, so please add the GraLoRA settings there (once without and once with hybrid) and delete them here.

There are some tests that check for errors or warnings at initialization time. Those should be moved to test_initialization.py. Check the tests there for inspiration

Some other tests, like checking the shape of parameters, are IMHO too granular and can be deleted without replacement. If the shapes are wrong, we would notice in the forward pass for instance.

There are a few tests that are not covered by existing tests and are not necessarily too granular, like
test_gralora_hybrid_forward_computation, test_gralora_information_exchange_via_permutation, and test_gralora_scaling_factor. Personally, I'd be fine with removing those too but keeping them would be fine as well.

Again, thank you for your kind review. I’ve added the GraLoRA test code to test_custom_models.py.
You can run it using:

pytest tests/test_custom_models.py -k "Gralora"

Just in case, I’ve kept the test_gralora.py file to test GraLoRA’s unique features.

BenjaminBossan · 2025-10-23T09:58:01Z

@yeonjoon-jung01 Please ping me when you're finished so that I know that I can give this another review. Also, if possible, please avoid force pushes or rebases, as those make reviews harder.

…hts parameter for flexible initialization

…est coverage

…ight calculation.

…ce, and more intuitive hybrid_r handling.

yeonjoon-jung01 · 2025-10-24T04:44:19Z

@yeonjoon-jung01 Please ping me when you're finished so that I know that I can give this another review. Also, if possible, please avoid force pushes or rebases, as those make reviews harder.

@BenjaminBossan I’ve finished updating the code 🙂. I saw your message a bit late — I had already rebased the branch to sync with the main stream, just in case there might be any conflicts. I’ll make sure to avoid force pushes or rebases from now on. Sorry about that!

yeonjoon-jung01 · 2025-10-24T10:15:13Z

@BenjaminBossan I have also resolved the previously missed features.

I’ve extended the test coverage to include test_custom_models.py, test_encoder_decoder_models.py, test_feature_extraction_models.py, and test_seq_classifier.py.
Additionally, I’ve added corresponding documentation and example code.

BenjaminBossan

Thanks for the updates to the PR. I did another review round, please check.

Also, before committing your changes, please call make style. Ensure that you have the correct version of ruff installed (0.12.12).

BenjaminBossan · 2025-10-24T12:10:46Z

docs/source/package_reference/gralora.md

@@ -0,0 +1,32 @@
+# GraLoRA


Please also add an entry to the toctree.

BenjaminBossan · 2025-10-24T12:12:32Z

src/peft/tuners/gralora/config.py

+        metadata={
+            "help": (
+                "gralora_k determines the number of subblocks in the GraLoRA adapter."
+                "The total parameter count is preserved regardles of gralora_k, while the expressivitiy is multiplied by gralora_k."


This sounds like the higher gralora_k, the better. Probably there is a tradeoff here. Could you please document it? Also, please mention the self.r % self.gralora_k != 0 constraint.

BenjaminBossan · 2025-10-24T12:13:00Z

src/peft/tuners/gralora/config.py

+
+
+@dataclass
+class GraloraConfig(PeftConfig):


Please also add a docstring here. You can just copy the help text for the parameter description.

BenjaminBossan · 2025-10-24T12:15:48Z

src/peft/tuners/gralora/layer.py

+        subblock_in_features = self.in_features // gralora_k
+        subblock_out_features = self.out_features // gralora_k


Nice. Could you please add tests for these errors to test_initialization.py? Check the other tests there for inspiration. Please also add a test for the self.r % self.gralora_k != 0 error.

BenjaminBossan · 2025-10-24T12:17:19Z

src/peft/tuners/gralora/layer.py

+            general_gralora_A = nn.Linear(self.in_features, hybrid_r, bias=False)
+            general_gralora_B = nn.Linear(hybrid_r, self.out_features, bias=False)


As a small optimization, you could use the torch.utils.integration.init_empty_weights context manager to initialize these parameters with empty tensors too.

BenjaminBossan · 2025-10-24T13:01:27Z

tests/test_gralora.py

+        # Outputs should be different due to hybrid component
+        assert not torch.allclose(output_hybrid, output_pure, atol=1e-3)
+
+    def test_gralora_invalid_rank_zero(self):


Move to test_initialization.py

BenjaminBossan · 2025-10-24T13:01:37Z

tests/test_gralora.py

+        with pytest.raises(ValueError, match="`r` should be a positive integer"):
+            get_peft_model(mlp, config)
+
+    def test_gralora_invalid_rank_negative(self):


Move to test_initialization.py

BenjaminBossan · 2025-10-24T13:02:18Z

tests/test_gralora.py

+        assert model.base_model.model.lin1.bias.requires_grad
+        assert not model.base_model.model.lin0.bias.requires_grad
+
+    def test_gralora_multiple_adapters_with_bias_raises(self):


Move to test_initialization.py

BenjaminBossan · 2025-10-24T13:04:15Z

tests/test_gralora.py

+        # Should match base model output (no merge)
+        assert torch.allclose(base_output, unloaded_output, atol=1e-5)
+
+    def test_gralora_merge_with_hybrid_component(self):


If you add a setting to test_custom_models.py with hybrid GraLoRA, this test would become redundant.

BenjaminBossan · 2025-10-24T13:06:20Z

tests/test_gralora.py

+        return X
+
+
+class TestGralora:


There are still a lot of tests here that are unnecessary because they're already covered by existing tests. Please remove them:

test_gralora_forward_pass

test_gralora_backward_pass

test_gralora_save_load_roundtrip

test_gralora_merge_and_unload

test_gralora_merge_unmerge

test_gralora_multiple_adapters

test_gralora_dtype_compatibility

test_gralora_disable_adapters

test_gralora_trainable_parameters_only

test_gralora_save_pretrained_files

test_gralora_safe_merge_success

test_gralora_safe_merge_detects_nan

test_gralora_cpu_fp16_merge

test_gralora_cpu_bf16_merge

test_gralora_delete_adapter

test_gralora_delete_nonexistent_adapter_raises

test_gralora_unload_without_merge

test_gralora_merge_with_adapter_names

test_gralora_enable_disable_adapter_layers

test_gralora_forward_with_merged_adapter

test_gralora_forward_with_disable_adapters_and_merged

test_gralora_add_non_active_adapter

I have simply removed test_gralora.py, and integrated into existing files

yeonjoon-jung01 · 2025-10-24T18:23:25Z

@BenjaminBossan I’ve resolved all of your comments and applied the suggested changes. The main update is that I removed tests/test_gralora.py and integrated the related test cases into the existing test_initialization and test_custom_models files, including additional scenarios for Hybrid GraLoRA.

yeonjoon-jung01 mentioned this pull request Oct 20, 2025

Support for GraLoRA #2636

Closed

BenjaminBossan requested changes Oct 22, 2025

View reviewed changes

yeonjoon-jung01 force-pushed the gralora_support branch from 134e6f0 to a24d156 Compare October 23, 2025 06:34

yeonjoon-jung01 and others added 5 commits October 24, 2025 00:00

feat: Add Gralora configuration and basic implementation

6dfa24e

ENH Support merge/unmerge in GraLoRA functionality; support init_weig…

bfa1ef7

…hts parameter for flexible initialization

TST Add test suite for GraLoRA.

9813b17

FIX & TEST: Fix GraLoRA bugs in get_peft_config_as_dict and improve t…

c1fe6c4

…est coverage

Refactor GraLoRA weight computation to improve efficiency in delta-we…

4f1444f

…ight calculation.

yeonjoon-jung01 force-pushed the gralora_support branch from c53ffce to 2618a8a Compare October 23, 2025 15:21

yeonjoon-jung01 added 2 commits October 24, 2025 00:29

Refactor GraLoRA code for clearer documentation, simplified inheritan…

9431502

…ce, and more intuitive hybrid_r handling.

Update test code for the GraLoRA method

dec25f5

yeonjoon-jung01 force-pushed the gralora_support branch from 2618a8a to dec25f5 Compare October 23, 2025 15:31

yeonjoon-jung01 requested a review from BenjaminBossan October 24, 2025 01:36

ADD: documentations, examples, and test code for GraLoRA method

925ad72

BenjaminBossan requested changes Oct 24, 2025

View reviewed changes

REFACTOR: integrate GraLoRA tests into existing test files

3f69d8f

yeonjoon-jung01 requested a review from BenjaminBossan October 24, 2025 18:24


		self.active_adapter = new_adapter or []

		def merge_and_unload(

		subblock_in_features = self.in_features // gralora_k
		subblock_out_features = self.out_features // gralora_k

		general_gralora_A = nn.Linear(self.in_features, hybrid_r, bias=False)
		general_gralora_B = nn.Linear(hybrid_r, self.out_features, bias=False)



		@dataclass
		class GraloraConfig(PeftConfig):

FEAT add GraLoRA #2851

Are you sure you want to change the base?

FEAT add GraLoRA #2851

Uh oh!

Conversation

yeonjoon-jung01 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Codes

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BenjaminBossan commented Oct 23, 2025

Uh oh!

yeonjoon-jung01 commented Oct 24, 2025

Uh oh!

yeonjoon-jung01 commented Oct 24, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeonjoon-jung01 commented Oct 20, 2025 •

edited

Loading