Car Ids get mixed and confused

[yasenOfficial]

Hello, thanks for your work on this model, I'm trying to rework it to work for Bulgarian license plates.

Frame Number,Car ID,License Plate,License Plate Score
5,1.0,CB8720TB,0.19673294579956319
21,4.0,KH5888BB,0.9785588194941117
28,4.0,KH5888BB,0.9739094052919942
29,1.0,CB8721TB,0.7660037101535193
33,1.0,CB8721TB,0.9926543276777186
35,4.0,KH5888BB,0.727899136655683
40,2.0,KH5838BB,0.5232681246358013
41,2.0,KH5338BB,0.1993557146048851
46,2.0,KH5888BB,0.9483313948229857
47,4.0,KH5888BB,0.8478755559926319
52,2.0,KH5888PB,0.6605058769015101
53,4.0,KH5888BB,0.7585433100466406
109,5.0,CA8978CX,0.18090781188533808
112,5.0,CA8942CX,0.23740360470947774
122,5.0,C8943CX,0.15154307079896012
135,5.0,CA5933CX,0.49990012910380177
136,5.0,CA3243CX,0.5319690946654391
138,5.0,CA8323CY,0.10956470000898931
315,13.0,CA2060AM,0.3455165278029194

The problem is that one car gets confused with different id's. Look at frames 21,28,40,41,46,47 it confuses the ids (2, and 4) Has anyone had this problem? Can someone provide a solution?