Gradient shape unexpected

[snykral]

`File ~\Python\PyTorch\RL\utils\optim.py:61, in Sophia.hutchinson(self, p, grad)
59 def hutchinson(self, p, grad):
60 u = torch.randn_like(grad)
---> 61 hessian_vector_product = torch.autograd.grad(grad.dot(u), p, retain_graph=True)[0]
62 return u * hessian_vector_product

RuntimeError: 1D tensors expected, but got 4D and 4D tensors`
Does it run on any network architecture?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[bbbxyz]

Looks like a bug. torch.dot() only works on 1D vectors. You could try using torch.sum(grad * u) instead.
Unless you need this urgently, I'd suggest waiting for the official implementation to be released tomorrow.

[kyegomez]

Just fixed it try upgrading to new version please!

[kyegomez]

Looks like a bug. torch.dot() only works on 1D vectors. You could try using torch.sum(grad * u) instead. Unless you need this urgently, I'd suggest waiting for the official implementation to be released tomorrow.

"we are aiming to release tomorrow" -Lol aiming

[snykral]

Just fixed it try upgrading to new version please!

It worked, but now I'm facing the same issue as #7. Somehow, grad.requires_grad is False when it arrives at the optimizer.

Also, I had to comment some lines at init.py, because their files didn't come with the library:
#from experiments.training import trainer
#from Sophia.decoupled_sophia.decoupled_sophia import DecoupledSophia

[bbbxyz]

FYI https://github.com/Liuhong99/Sophia

[kyegomez]

I've upgraded it, now try upgrading with pip 😊

[Kingsleyandher]

did you solve this question? I also meet this question in megatron...