Classification fails

[JuhaHuiskonen]

I am continuing from an existing refinement block. I use otherwise the same parameters, but have added classification to 4 classes. The inputs to this new classifacition block are the template map and parameters from the previous refinement. The first iteration (iteration 2) runs ok, but I get only 1 class. The second iteration (iteration 3) gives this error when the jobs starts:

1	2024-03-24 12:01:23 [INFO] Job 20950679_1 (v0.6.3) launching on r01c13.bullx using 4 task(s) and 370 GB of RAM
2	2024-03-24 12:01:23 [INFO] Filesystem                                      Size  Used Avail Use% Mounted on
3	2024-03-24 12:01:23 [INFO] 10.140.15.146@o2ib:10.140.15.145@o2ib:/scratch   20T  1.9T   18T  10% /scratch/project_2009485/nextPYP/tmp
4	2024-03-24 12:01:23 [WARNING] Could not find frealign statistics file in maps folder
5	2024-03-24 12:01:23 [WARNING] Could not find frealign statistics file in maps folder
6	2024-03-24 12:01:23 [WARNING] Could not find frealign statistics file in maps folder
7	2024-03-24 12:01:23 [WARNING] Could not find frealign statistics file in maps folde

And then these errors when it tries to refine class 2:

2024-03-24 13:29:40 [�INFO�] �pyp/align/core.py�:1623 | Running refinement for class 2 of 4
2024-03-24 13:29:41 [�INFO�] �pyp/align/core.py�:57 | CSP Total time elapsed: 01h 27m 30s
2024-03-24 13:29:41 [�INFO�] �/opt/pyp/bin/run/pyp�:57 | Total time elapsed (csp_swarm): 01h 28m 18s
2024-03-24 13:29:41 [�INFO�] �/opt/pyp/bin/run/pyp�:3178 | Filesystem                                      Size  Used Avail Use% Mounted on
2024-03-24 13:29:41 [�INFO�] �/opt/pyp/bin/run/pyp�:3178 | 10.140.15.146@o2ib:10.140.15.145@o2ib:/scratch   20T  1.9T   18T  10% /scratch/project_2009485/nextPYP/tmp
Traceback (most recent call last):
  File "/opt/pyp/bin/run/pyp", line 3873, in <module>
    csp_swarm(args.file, parameters, int(args.iter), args.skip, args.debug)
  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
    return func(*args, **kwargs)
  File "/opt/pyp/bin/run/pyp", line 2433, in csp_swarm
    align.csp_refinement(
  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
    return func(*args, **kwargs)
  File "/opt/pyp/src/pyp/align/core.py", line 1668, in csp_refinement
    frealign_parfile.Parameters.addFrameIndexInScanord(class_parxfile, class_parxfile, False)
  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 997, in addFrameIndexInScanord
    input_object = Parameters.from_file(input_parfile)
  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 1446, in from_file
    fieldwidths, fieldstring, version, extended = cls.format_from_parfile(filename)
  File "/opt/pyp/src/pyp/inout/metadata/frealign_parfile.py", line 1550, in format_from_parfile
    with open(par_filename) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/project_2009485/nextPYP/tmp/huiskone/20950679_1/lamella11_ts_004/frealign/maps/lamella11_ts_004_r02_02.parx'
2024-03-24 13:29:41 [�ERROR�] �/opt/pyp/bin/run/pyp�:3887 | PYP (cspswarm) failed
2024-03-24 13:29:48 [�INFO�] �/opt/pyp/bin/run/pyp�:3232 | Job 20950679_1 (v0.6.3) launching on r01c13.bullx using 4 task(s) and 370 GB of RAM
2024-03-24 13:29:48 [�INFO�] �pyp/refine/csp/particle_cspt.py�:57 | Total time elapsed (csp_local_merge): 00h 00m 00s
2024-03-24 13:29:48 [�INFO�] �/opt/pyp/bin/run/pyp�:3178 | Filesystem                                      Size  Used Avail Use% Mounted on
2024-03-24 13:29:48 [�INFO�] �/opt/pyp/bin/run/pyp�:3178 | 10.140.15.146@o2ib:10.140.15.145@o2ib:/scratch   20T  1.9T   18T  10% /scratch/project_2009485/nextPYP/tmp
Traceback (most recent call last):
  File "/opt/pyp/bin/run/pyp", line 3953, in <module>
    particle_cspt.merge_movie_files_in_job_arr(
  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
    return func(*args, **kwargs)
  File "/opt/pyp/src/pyp/refine/csp/particle_cspt.py", line 380, in merge_movie_files_in_job_arr
    with open(movie_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'stacks.txt'
2024-03-24 13:29:48 [�ERROR�] �/opt/pyp/bin/run/pyp�:3974 | PYP (csp_local_merge) failed
2024-03-24 13:29:48 [�INFO�] �/opt/pyp/bin/run/pyp�:3980 | Deleted temporary files from /scratch/project_2009485/nextPYP/tmp/huiskone/20950679_1

Any idea why the first iteration (iteration 2) does not produce 4 classes but only 1?

[yezhou867]

Hi, Classification was not designed to run from iteration 2 so far, mainly because the initialization needs a reference. So, at iteration 2, you have to do class=1, and from iteration 3 you can change to 4 classes.

[abartesaghi]

Yes, iteration 2 should only recapitulate the consensus model and you do not need to explicitly set the number of classes to 1 for that iteration. The actual problem is that the file frealign/maps/lamella11_ts_004_r02_02.parx is missing and I'm wondering if this is because the scratch was cleared by another process. Can you share the entire log without gaps? Also, the warnings at the top of the log should be safe to ignore.