Problem description:
Around the precompute_text_embedding/precompute_image_embedding, both processors would access the code section and use the full dataset lists. That cause some collision when writing to a same filename, which can halt the training sometimes/randomly.
Solving:
1. add a line of code ---->if accelerator.main_process_first():
(just under if args.precompute_image_embeddings: or if args.precompute_text_embeddings:)
2. add 4 extra SPACE to the code section following the precompute conditioins.
Problem description:
Around the precompute_text_embedding/precompute_image_embedding, both processors would access the code section and use the full dataset lists. That cause some collision when writing to a same filename, which can halt the training sometimes/randomly.
Solving:
1. add a line of code ---->if accelerator.main_process_first():
(just under if args.precompute_image_embeddings: or if args.precompute_text_embeddings:)
2. add 4 extra SPACE to the code section following the precompute conditioins.