Refinement Module

[ahm-nq]

First of all, thank you for your incredible work on BiRefNet and for making it publicly available—it’s truly inspiring to see such dedication and innovation in the field.

I’ve been fine-tuning BiRefNet on a car segmentation dataset using the General_244.pth checkpoint and the same configuration provided in the original GitHub repository. However, after 164 epochs of fine-tuning, the results I’m achieving don’t seem to match the quality of the pretrained model.

While exploring the configuration file, I noticed an option to choose a Refinement Module, which is set to an empty string by default. Could this be a potential reason why my results aren’t as refined as those of the pretrained model?

Additionally, I couldn’t find explicit information in the paper about whether Refinement Modules were used during the training of the pretrained BiRefNet. Did you incorporate Refinement Modules to achieve the results shared in the repository? Any insights or guidance would be immensely helpful.

Thank you for your time and for creating such a remarkable contribution to segmentation research!

Let me know if you would like to seee some sample results.

[ZhengPeng7]

Hi, thanks for your interest in looking deeper into my codes.

Some refinement blocks can help, but the relevant codes are in chaos, and GPU memory costs could be extremely high. Based on these concerns, they were not used in my implementation.

I tested some refiners with SwinB as the backbone for lower GPU memory cost during training. In a relatively fair comparison, the one with a refiner can truly bring some improvement, as the screenshot shows below.

[ahm-nq]

Ok. Let me add some context. I am fine tuning your model to remove the background through the window regions of the car. I have attached comparison of results original legacy model and fine tuned version. Results are a bit compromised on finetuned results.

Brief settings :
bb is freezed
loss recipe : 30* bce, 0.5iou, 30ssim
augmentations: ['flip', 'enhance', 'rotate', 'crop']

Original image:

Background removed image using legacy weights:

Results i obtained after finetuning legacy:

Can it be the case introducing this new problem of removing bg through the window can affect existing learning? I have tried multiple fixes, but still finetuned results are not as good as original, though i get bg removed through the window inthe fine-tuned model.

[ZhengPeng7]

First, freezing the backbone was not a good idea in my experiments long ago.
Besides, you can try training from scratch if you have hundreds of images.

[ZhengPeng7]

Oh, I forgot one important thing -- I updated the default training to FP16. However, all the existing weights were trained in FP32, if not mentioned. I'm not very sure, but it should do some influence in fine-tuning with previous weights. You can easily comment the --use_accelerate in train.sh to avoid the FP16 training.