Running evo2_40b model

[zane529]

Hi, I try to run the evo2_40b in aws's g6e.12xlarge whit has 4 L40S, and i get the error.

/home/ubuntu/evo2_venv/lib/python3.12/site-packages/transformer_engine/pytorch/module/base.py:630: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(state, map_location="cuda")
/home/ubuntu/evo2/vortex/vortex/model/utils.py:153: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
return torch_load(state, map_location=device)
Using device: cuda:0
Traceback (most recent call last):
File "/home/ubuntu/evo2/./test/test_evo2.py", line 133, in
main()
File "/home/ubuntu/evo2/./test/test_evo2.py", line 105, in main
accuracies, losses = test_forward_pass(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/./test/test_evo2.py", line 49, in test_forward_pass
logits, _ = model.model.forward(input_ids.unsqueeze(0))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/vortex/vortex/model/model.py", line 696, in forward
x, inference_params_dict_out = self.stateless_forward(x, padding_mask=padding_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/vortex/vortex/model/model.py", line 770, in stateless_forward
x, _ = block(x, inference_params=None, padding_mask=padding_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/vortex/vortex/model/model.py", line 519, in forward
z = self.proj_norm(u)
^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/vortex/vortex/model/model.py", line 485, in proj_norm
projected = self.projections(normalized)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2/vortex/vortex/model/layers.py", line 78, in forward
out = super().forward(x)
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/transformer_engine/pytorch/module/linear.py", line 1024, in forward
out = linear_fn(*args)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/transformer_engine/pytorch/module/linear.py", line 212, in forward
_ = fp8_gemm(
^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/transformer_engine/pytorch/cpp_extensions/gemm.py", line 180, in fp8_gemm
_ = fn(*args)
^^^^^^^^^
File "/home/ubuntu/evo2_venv/lib/python3.12/site-packages/torch/_ops.py", line 1116, in call
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: /TransformerEngine/transformer_engine/common/gemm/cublaslt_gemm.cu:102 in function cublas_gemm: cuBLAS Error: the resource allocation failed

Please help me troubshooting, and the evo2 support mutil cards?