About gradient balancing

[miranghimire]

There are three backward calls inside gardient balancing between generator loss & OCR loss:

• convolutional-handwriting-gan/models/ScrabbleGAN_baseModel.py

  Line 354 in f7daa50

  self.loss_T.backward(retain_graph=True)

• convolutional-handwriting-gan/models/ScrabbleGAN_baseModel.py

  Line 368 in f7daa50

  self.loss_T.backward(retain_graph=True)

• convolutional-handwriting-gan/models/ScrabbleGAN_baseModel.py

  Line 374 in f7daa50

  self.loss_T.backward()

Won't these calls accumulate the gradients during the call of optimizer.step(); I thought our objective here was to simply compute the gardient balancing terms and multiply those to the loss or could you please give overview of what's going on here inside gradient balancing incase I misunderstood something?

[sharonFogel]

I just looked at the code, I think you're right and there should be self.netG.zero_grad() between the first and second backprop. The third one is performed without gradient accumulation just so that the graph won't be retained.