YOLOv3 model doesn't converge

[JVD9kh96]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

[zimengxueying]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

[JVD9kh96]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

I solved the problem by using tensorflow 2.2.0. It turned out that in tensorflow 2.5.0 which I was using, after the first prediction, the weights somehow got random and the only true prediction was for the first test image and then the weights got random. When I used tensorflow 2.2.0 the issue was solved. however, I was able to solve the issue in tensorflow 2.5.0 by freezing the weghts after loading the model.

I was gonna close the issue, so let me know if your problem is solved.

[mastergao57]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

I solved the problem by using tensorflow 2.2.0. It turned out that in tensorflow 2.5.0 which I was using, after the first prediction, the weights somehow got random and the only true prediction was for the first test image and then the weights got random. When I used tensorflow 2.2.0 the issue was solved. however, I was able to solve the issue in tensorflow 2.5.0 by freezing the weghts after loading the model.

I was gonna close the issue, so let me know if your problem is solved.

When I run video_demo.py, I met the same problem. Only the first prediction is true, the others are nan. I guess the problem is in the 'utils.load_weights(model, "./yolov3.weights")'. How to solve the problem? I use tensorflow==2.7.0

[giakoumoglou]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

I solved the problem by using tensorflow 2.2.0. It turned out that in tensorflow 2.5.0 which I was using, after the first prediction, the weights somehow got random and the only true prediction was for the first test image and then the weights got random. When I used tensorflow 2.2.0 the issue was solved. however, I was able to solve the issue in tensorflow 2.5.0 by freezing the weghts after loading the model.

I was gonna close the issue, so let me know if your problem is solved.

What command did you added and where? Thanks!

[JVD9kh96]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

I solved the problem by using tensorflow 2.2.0. It turned out that in tensorflow 2.5.0 which I was using, after the first prediction, the weights somehow got random and the only true prediction was for the first test image and then the weights got random. When I used tensorflow 2.2.0 the issue was solved. however, I was able to solve the issue in tensorflow 2.5.0 by freezing the weghts after loading the model.
I was gonna close the issue, so let me know if your problem is solved.

When I run video_demo.py, I met the same problem. Only the first prediction is true, the others are nan. I guess the problem is in the 'utils.load_weights(model, "./yolov3.weights")'. How to solve the problem? I use tensorflow==2.7.0

One way that helped me fix this issue was down-grading tensorflow to 2.2.0. Another solution wich I found was freezing the weights after loading them. These two solutions was helpful for me.

[giakoumoglou]

Can you paste the command that freezes the weights?

[JVD9kh96]

Hello. First of all, thank you for your amazing code. While trying to train the YOLOv3 model from scratch on yymnist dataset, I noticed that the loss is fluctuating between 20 and 40 and it doesn't converge. I didnt't change any of the config settings except the _C.YOLO.classes = "data/classes/voc.names" which I changed it to _C.YOLO.classes = "data/classes/yymnist.names". Am I doing something wrong?
I use tensorflow==2.5.0

I met the same situation,Have you solved？

I solved the problem by using tensorflow 2.2.0. It turned out that in tensorflow 2.5.0 which I was using, after the first prediction, the weights somehow got random and the only true prediction was for the first test image and then the weights got random. When I used tensorflow 2.2.0 the issue was solved. however, I was able to solve the issue in tensorflow 2.5.0 by freezing the weghts after loading the model.
I was gonna close the issue, so let me know if your problem is solved.

What command did you added and where? Thanks!

After you load the weights with model.set_weights(weights) or model.load_weights("path/to/weights"), just add the following command:
model.trainable = False