FP16 and deepspeed zero3 both break training in my testing ([rank0]: RuntimeError: weight should have at least three dimensions), this config works#15

Open

Julian2002 wants to merge 1 commit into2U1:masterfrom Julian2002:8-bit_config_patch

Commits on Nov 6, 2024

FP16 and deepspeed zero3 both break training in my testing ([rank0]: RuntimeError: weight should have at least three dimensions), this config works
Julian2002
authored