Still can't make Step 5 right

[lizimu-gzlab]

I'm working on a CryoSPARC project with preferred orientation problem. After Referenced Based Motion Correction, overall resolution reaches 3.80Å, while worst orientation resolution is about 8Å. I use this command to combine particle stacks:

relion_stack_create --i autorefinement_mrcs.star --o J53particle_stack

Then start step 4 using commands:

cryopros-train
--opt /opt/miniconda3/envs/cryopros/lib/python3.12/site-packages/cryoPROS/options/train.json
--gpu_ids 0 1 2 3
--task_name HAtrimer_iteration_1
--box_size 168
--Apix 1.245
--volume_scale 50
--init_volume_path J53_008_volume_map_lowpass13.mrc
--data_path J53particle_stack.mrcs
--param_path J53particle_stack.star
--invert
--dataloader_batch_size 8

We have about 140000 particles, each 1000 iterations took about 2.5 hours on 4 GTX3090 GPUs. While it is still training, I continue to try step 5 with these commands:

cryopros-uniform-pose
--input ./J53particle_stack.star
--output ./unipose.star

cryopros-generate
--model_path /storage_data/data_s/CS-20240523-pdcovrbd-3f7-9c1/cryopros/generate/HAtrimer_iteration_1/models/10000.pth
--output_path generated_HAtrimer_iteration_1
--gen_name HAtrimer_iteration_1_generated_particles.mrcs
--batch_size 50
--box_size 168
--Apix 1.245
--param_path unipose.star
--invert
--gen_mode 0

The unipose.star looks like it works as intended, but the created particles looks quite wrong,

and their 2D Classes looks even worse,

I'm not quite sure about the vim command, so I change all "J53particle_stack.mrcs" to "generated_HAtrimer_iteration_1/HAtrimer_iteration_1_generated_particles.mrcs" in Notepad++, then import the modified star file to CryoSPARC.

I think the 10000 iterations model should output decent results, and the 13Å lowpassed volume could be enough. I also tried gen_mode 1, but created particles always look like stars in the sky, not actual particles.
Any idea how to fix it?

[zhengdihan]

Hi,
May I take a look at the log file? It should be located in ./generate/{task_name}/train.log. I suspect you might be encountering the posterior collapse issue, where the KL loss is extremely low (approximately 1e-9). This is a frequent issue when training a VAE model. Typically, rerunning the training procedure will resolve this problem. The expected KL loss is usually around 1e-4 or 2e-4. If this is not the case, please rerun the "cryopros-train" program.

Regarding the running time, could you try adding the option "--dataloader_num_workers 0" when using "cryopros-train"? For example:

cryopros-train
--opt /opt/miniconda3/envs/cryopros/lib/python3.12/site-packages/cryoPROS/options/train.json
--gpu_ids 0 1 2 3
--task_name HAtrimer_iteration_1
--box_size 168
--Apix 1.245
--volume_scale 50
--init_volume_path J53_008_volume_map_lowpass13.mrc
--data_path J53particle_stack.mrcs
--param_path J53particle_stack.star
--invert
--dataloader_batch_size 8
--dataloader_num_workers 0

This may improve the running speed. On our server with 4x V100 GPUs, it takes about 10 minutes for 1000 iterations.

[lizimu-gzlab]

KL loss looks alright. I will try --dataloader_num_workers 0 next time. There are warnings when running cryopros-generate: /opt/miniconda3/envs/cryopros/lib/python3.12/site-packages/torch/nn/functional.py:4404: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.   warnings.warn( /opt/miniconda3/envs/cryopros/lib/python3.12/site-packages/torch/nn/functional.py:4343: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.   warnings.warn( I wonder if this is the case.   ------------------ Original ------------------ From: ***@***.***>; Date:  Fri, Jun 21, 2024 02:50 PM To: ***@***.***>; Cc: ***@***.***>; ***@***.***>; Subject:  Re: [mxhulab/cryopros] Still can't make Step 5 right (Discussion #11)   Hi, May I take a look at the log file? It should be located in ./generate/{task_name}/train.log. I suspect you might be encountering the posterior collapse issue, where the KL loss is extremely low (approximately 1e-9). This is a frequent issue when training a VAE model. Typically, rerunning the training procedure will resolve this problem. The expected KL loss is usually around 1e-4 or 2e-4. If this is not the case, please rerun the "cryopros-train" program. Regarding the running time, could you try adding the option "--dataloader_num_workers 0" when using "cryopros-train"? For example: cryopros-train --opt /opt/miniconda3/envs/cryopros/lib/python3.12/site-packages/cryoPROS/options/train.json --gpu_ids 0 1 2 3 --task_name HAtrimer_iteration_1 --box_size 168 --Apix 1.245 --volume_scale 50 --init_volume_path J53_008_volume_map_lowpass13.mrc --data_path J53particle_stack.mrcs --param_path J53particle_stack.star --invert --dataloader_batch_size 8 --dataloader_num_workers 0 This may improve the running speed. On our server with 4x V100 GPUs, it takes about 10 minutes for 1000 iterations. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[zhengdihan]

This warning should not be an issue. May I take a look at your training.log? Just a few lines from the bottom of the file will be sufficient.

You can also find some generated images in ./generate/{task_name}/images/xxxx/. Could you check whether the images generated during the training procedure still look like stars?

Another option is to use --gen_mode 2, which will employ the float type to represent the particles.

[lizimu-gzlab]

train.log

24-06-17 15:13:09.340 : <epoch: 0, iter: 1,000, lr:1.000e-04> Loss: 3.615e-04 Reconstruction loss: 3.155e-04 KL loss: 4.601e-05 Model loss: 0.000e+00
24-06-17 17:42:55.896 : <epoch: 0, iter: 2,000, lr:1.000e-04> Loss: 8.491e-05 Reconstruction loss: 3.335e-05 KL loss: 5.156e-05 Model loss: 0.000e+00
24-06-17 20:12:49.160 : <epoch: 0, iter: 3,000, lr:1.000e-04> Loss: 9.107e-05 Reconstruction loss: 2.833e-05 KL loss: 6.274e-05 Model loss: 0.000e+00
24-06-17 22:44:46.052 : <epoch: 0, iter: 4,000, lr:1.000e-04> Loss: 1.042e-04 Reconstruction loss: 2.724e-05 KL loss: 7.701e-05 Model loss: 0.000e+00
24-06-18 01:14:32.555 : <epoch: 0, iter: 5,000, lr:1.000e-04> Loss: 1.196e-04 Reconstruction loss: 2.988e-05 KL loss: 8.969e-05 Model loss: 0.000e+00
24-06-18 03:44:09.634 : <epoch: 0, iter: 6,000, lr:1.000e-04> Loss: 1.351e-04 Reconstruction loss: 3.249e-05 KL loss: 1.026e-04 Model loss: 0.000e+00
24-06-18 06:13:49.459 : <epoch: 0, iter: 7,000, lr:1.000e-04> Loss: 1.491e-04 Reconstruction loss: 3.680e-05 KL loss: 1.123e-04 Model loss: 0.000e+00
24-06-18 08:44:00.497 : <epoch: 0, iter: 8,000, lr:1.000e-04> Loss: 1.626e-04 Reconstruction loss: 4.186e-05 KL loss: 1.207e-04 Model loss: 0.000e+00
24-06-18 11:14:36.808 : <epoch: 0, iter: 9,000, lr:1.000e-04> Loss: 1.762e-04 Reconstruction loss: 4.592e-05 KL loss: 1.303e-04 Model loss: 0.000e+00
24-06-18 13:44:50.443 : <epoch: 0, iter: 10,000, lr:1.000e-04> Loss: 1.921e-04 Reconstruction loss: 5.129e-05 KL loss: 1.408e-04 Model loss: 0.000e+00
24-06-18 13:44:50.445 : Saving the model.
24-06-18 16:14:34.199 : <epoch: 0, iter: 11,000, lr:1.000e-04> Loss: 1.928e-04 Reconstruction loss: 5.312e-05 KL loss: 1.396e-04 Model loss: 0.000e+00
24-06-18 18:45:22.045 : <epoch: 0, iter: 12,000, lr:1.000e-04> Loss: 1.938e-04 Reconstruction loss: 5.218e-05 KL loss: 1.416e-04 Model loss: 0.000e+00
24-06-18 21:16:22.056 : <epoch: 0, iter: 13,000, lr:1.000e-04> Loss: 1.917e-04 Reconstruction loss: 5.089e-05 KL loss: 1.408e-04 Model loss: 0.000e+00
24-06-18 23:47:02.968 : <epoch: 0, iter: 14,000, lr:1.000e-04> Loss: 1.917e-04 Reconstruction loss: 5.032e-05 KL loss: 1.414e-04 Model loss: 0.000e+00
24-06-19 05:25:51.436 : <epoch: 0, iter: 15,000, lr:1.000e-04> Loss: 1.940e-04 Reconstruction loss: 5.073e-05 KL loss: 1.432e-04 Model loss: 0.000e+00
24-06-19 11:07:53.103 : <epoch: 0, iter: 16,000, lr:1.000e-04> Loss: 1.907e-04 Reconstruction loss: 4.993e-05 KL loss: 1.407e-04 Model loss: 0.000e+00
24-06-19 16:50:20.186 : <epoch: 0, iter: 17,000, lr:1.000e-04> Loss: 1.927e-04 Reconstruction loss: 4.965e-05 KL loss: 1.431e-04 Model loss: 0.000e+00
24-06-19 22:33:07.691 : <epoch: 1, iter: 18,000, lr:1.000e-04> Loss: 1.924e-04 Reconstruction loss: 4.997e-05 KL loss: 1.424e-04 Model loss: 0.000e+00
24-06-20 04:15:56.021 : <epoch: 1, iter: 19,000, lr:1.000e-04> Loss: 1.933e-04 Reconstruction loss: 5.137e-05 KL loss: 1.419e-04 Model loss: 0.000e+00
24-06-20 09:58:55.678 : <epoch: 1, iter: 20,000, lr:5.000e-05> Loss: 1.901e-04 Reconstruction loss: 5.089e-05 KL loss: 1.392e-04 Model loss: 0.000e+00
24-06-20 09:58:55.682 : Saving the model.
24-06-20 15:41:38.057 : <epoch: 1, iter: 21,000, lr:5.000e-05> Loss: 1.896e-04 Reconstruction loss: 4.975e-05 KL loss: 1.399e-04 Model loss: 0.000e+00
24-06-20 21:23:54.203 : <epoch: 1, iter: 22,000, lr:5.000e-05> Loss: 1.899e-04 Reconstruction loss: 4.951e-05 KL loss: 1.403e-04 Model loss: 0.000e+00
24-06-21 03:06:41.570 : <epoch: 1, iter: 23,000, lr:5.000e-05> Loss: 1.910e-04 Reconstruction loss: 5.043e-05 KL loss: 1.406e-04 Model loss: 0.000e+00
24-06-21 08:49:15.206 : <epoch: 1, iter: 24,000, lr:5.000e-05> Loss: 1.900e-04 Reconstruction loss: 5.023e-05 KL loss: 1.398e-04 Model loss: 0.000e+00
24-06-21 14:31:44.210 : <epoch: 1, iter: 25,000, lr:5.000e-05> Loss: 1.892e-04 Reconstruction loss: 4.986e-05 KL loss: 1.393e-04 Model loss: 0.000e+00

It seems my e-mail reply didn't attached the train.log, should be ok now.
Looks like the 0010_20000_G.png is much better than imported particles, but I'm not sure.

[zhengdihan]

The generated images and train.log appear to be fine. Perhaps you could try using --gen_mode 2 and the checkpoint 20000.pth. The generated particles should look similar to the images you've shown above.

[lizimu-gzlab]

Looks good now, --gen_mode 2 seems fixed the problem, and output a particle stack about the same size as imported J53particle_stack.mrcs.