Tutorial runs, but not our data

[stav-ros]

Hi,
Firstly I wanna say I love the one-click solution that nextPYP provides, and as we saw with the tutorial dataset, it can give excellent results in a matter of minutes.

However, when we tried to align our own data, we run into some issues.
Firstly, when we want to define "FRAME PATTERN" during the pre-processing step, there are no clear instructions on what the keywords available, and no way to test whether the ones written there can read all of our data.
In our case we had something like position__TILTSERIES_SCANORD_ANGLE__date_identifier.tiff and it wasn't recognised by the Frame Pattern.
When we tried to rename all our files to match the Tutorial data names, we run into some other issues:
Traceback (most recent call last):

98 File "/opt/pyp/bin/run/pyp", line 3478, in
99 tomo_swarm(args.path, args.file, args.debug, args.keep, args.skip)
100 File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
101 return func(*args, **kwargs)
102 File "/opt/pyp/bin/run/pyp", line 1429, in tomo_swarm
103 ] = preprocess.read_tilt_series(
104 File "/opt/pyp/src/pyp/preprocess/core.py", line 685, in read_tilt_series
105 mpi.submit_function_to_workers(
106 File "/opt/pyp/src/pyp/system/mpi.py", line 185, in submit_function_to_workers
107 parallel(delayed(wrapper)(func, *arg, current_directory=current_directory) for idx, func in enumerate(funcs) for arg in args[idx])
108 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 1098, in call
109 self.retrieve()
110 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/parallel.py", line 975, in retrieve
111 self._output.extend(job.get(timeout=self.timeout))
112 File "/usr/local/envs/pyp/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
113 return future.result(timeout=timeout)
114 File "/usr/local/envs/pyp/lib/python3.8/concurrent/futures/_base.py", line 444, in result
115 return self.__get_result()
116 File "/usr/local/envs/pyp/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
117 raise self.exception
118 FileNotFoundError: [Errno 2] No such file or directory: 'TS_43_039-60.0.mrc'
119 2024-01-26 11:51:12 [ERROR] /opt/pyp/bin/run/pyp:3483 | PYP (tomoswarm) failed

Even tho the File is there in the directory. So that was confusing.
Plus at times some random:

CUDA User Warning: Unable to start Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? System Has Unsupported driver/graphics card combination....

Again, running MotionCorr3 (with GPU) and Aretomo (GPU) with the tutorial data, was just fine, but not for our data. So that was confusing.

But by far the most common error we get is this:

ValueError: could not convert string to float: 'RO'
10
11 The above exception was the direct cause of the following exception:
12
13 Traceback (most recent call last):
14 File "/opt/pyp/bin/run/pyp", line 3478, in
15 tomo_swarm(args.path, args.file, args.debug, args.keep, args.skip)
16 File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
17 return func(*args, **kwargs)
18 File "/opt/pyp/bin/run/pyp", line 1429, in tomo_swarm
19 ] = preprocess.read_tilt_series(
20 File "/opt/pyp/src/pyp/preprocess/core.py", line 377, in read_tilt_series
21 tilts = np.loadtxt("{0}.rawtlt".format(name))
22 File "/usr/local/envs/pyp/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1356, in loadtxt
23 arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
24 File "/usr/local/envs/pyp/lib/python3.8/site-packages/numpy/lib/npyio.py", line 999, in _read
25 arr = _load_from_filelike(
26 ValueError: could not convert string 'RO' to float64 at row 0, column 1.
27 2024-01-26 12:01:40 [ERROR] /opt/pyp/bin/run/pyp:3483 | PYP (tomoswarm) failed

Which could potentially mean there is some issue with the way our Frames tiff output is written, but we are not sure what it is. Hopefully someone more expert can help us identify these issues, and help others that might encounter them :)

Thanks

[tofushan]

Hi,

Thanks for asking! We intend to write an instruction on how to use the frame pattern but have not yet had a chance.

The frame pattern uses regular expression to group the movie files that have the same TILTSERIES. SCANORD (integers; i.e., 000, 001) and ANGLE (signed floating points; i.e., -10.9, 1.00)
are used to extract the "acquisition order" and "tilt angle" respectively. If you have delimiters such as _, you should provide them in the pattern as well. Otherwise, these information cannot be detected correctly. For entries that do not belong to one of these categories, put a *.

Therefore, in your case, the pattern of position__TILTSERIES_SCANORD_ANGLE__date_identifier.tiff should be position__TILTSERIES_SCANORD_ANGLE__*_*.tiff. We normally see position comes with a ID such as position_1. Then the pattern is TILTSERIES_SCANORD_ANGLE__*_*.tiff.

However, I don't think .tiff can be recognized in current nextPYP. You probably need to convert all of your files into .tif or .mrc before doing the pre-processing.

Hope this helps!

[stav-ros]

Hi, and thanks for the quick reply.

Correct me if I'm wrong, but there is no difference between tiff and tif right? Because, as I mentioned, we renamed our files to EXACTLY match your tutorial data files but we have (primarily) getting this ValueError: could not convert string 'RO' to float64 at row 0, column 1.

Also, we tried "TILTSERIES_SCANORD_ANGLE__*_*.tiff" but we get the "Cannot find any files with this pattern" error, which is confusing. We tried using the asterisk * using your tutorial data, ánd it doesn't seem to recognize it as a variable.

Lastly, Does nextPYP only support SerialEM Files?

Hope this clears things up!

[tofushan]

Yes, .tiff and .tif are the same, but currently we do not consider .tiff in our application code. For example, files with .tiff extension are not moved to the local scratch, where nextPYP performs the image processing. We'll include it and test it thoroughly in the future release.

It's difficult to tell where RO comes from. I guess it's simply caused by using the wrong pattern. Could you show how your raw data and the renamed ones look like at least for one tilt-series?

For the tutorial data, there's no redundant entry that you can use * to represent it. Take TS_54_008_12.0.tif as an example, TS_54 matches TILTSERIES, 008 matches SCANORD and 12.0 matches ANGLE.

I guess you mean SerialEM files? Without .mdoc (metadata file), solely using the pattern can generate initial angles (.rawtlt in IMOD) and acquisition order, which will be used in the downstream refinement steps. If you have .mdoc, you can put them together with your raw data, and the angles and acquisition order will be extracted from the metadata rather than the filename of your movie files.

[abartesaghi]

The new version 0.6.1 fixes all the issues with handling of *.tiff files. Let us know if you are still having problems.

[stav-ros]

The new version 0.6.1 fixes all the issues with handling of *.tiff files. Let us know if you are still having problems.

Hello! Unfortunately the error still persists. Are there any restrictions/conditions for the micrographs? Attached is the list for one tilt-series:

And here is the log file for the run:

2024-01-31 09:57:30 [INFO] /opt/pyp/bin/run/pyp:3273 | Job 3940340880032593672_1 (v0.6.1) launching on cryo1 using 1 task(s) 
2	2024-01-31 09:57:30 [INFO] /opt/pyp/bin/run/pyp:3200 | Filesystem                 Size  Used Avail Use% Mounted on
3	2024-01-31 09:57:30 [INFO] /opt/pyp/bin/run/pyp:3200 | /dev/mapper/vgroot-lvroot   98G   42G   52G  45% /tmp/pyp
4	2024-01-31 09:57:30 [INFO] /opt/pyp/bin/run/pyp:1348 | Working path: /tmp/pyp/3940340880032593672_1/temp
5	2024-01-31 09:57:30 [INFO] pyp/inout/utils/core.py:17 | Retrieving .pyp_config.toml
6	2024-01-31 09:57:30 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-ewUEV6Zql6xz3d0s.micrographs
7	2024-01-31 09:57:30 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-ewUEV6Zql6xz3d0s.films
8	2024-01-31 09:57:30 [INFO] /opt/pyp/bin/run/pyp:57 | Loading results took: 00h 00m 00s
9	2024-01-31 09:57:31 [INFO] /opt/pyp/bin/run/pyp:57 | Total time in tomo_swarm 00h 00m 00s seconds
10	ValueError: could not convert string to float: 'RO'
11	
12	The above exception was the direct cause of the following exception:
13	
14	Traceback (most recent call last):
15	  File "/opt/pyp/bin/run/pyp", line 3490, in <module>
16	    tomo_swarm(args.path, args.file, args.debug, args.keep, args.skip)
17	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
18	    return func(*args, **kwargs)
19	  File "/opt/pyp/bin/run/pyp", line 1432, in tomo_swarm
20	    ] = preprocess.read_tilt_series(
21	  File "/opt/pyp/src/pyp/preprocess/core.py", line 370, in read_tilt_series
22	    tilts = np.loadtxt("{0}.rawtlt".format(name))
23	  File "/usr/local/envs/pyp/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1356, in loadtxt
24	    arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
25	  File "/usr/local/envs/pyp/lib/python3.8/site-packages/numpy/lib/npyio.py", line 999, in _read
26	    arr = _load_from_filelike(
27	ValueError: could not convert string 'RO' to float64 at row 0, column 1.
28	2024-01-31 09:57:31 [ERROR] /opt/pyp/bin/run/pyp:3495 | PYP (tomoswarm) failed
29	

Hope this helps.

[tofushan]

Hi,

Thanks for providing the information.

It looks like you select the Single-file tilt-series option in the Frame alignment tab. This option should not be selected because your tilted images are separated (each file represents one tilted image). If this option is selected, Frame pattern won't take any effect.

According to the data your provide, you should use TILTSERIES_SCANORD_ANGLE_*_*_*.tif for Frame pattern. Be aware of the number of _. Using __ (two _s) in Frame pattern, but your files do not have __, it won't work because they do not match.

I would suggest doing everything from scratch instead of re-running in the existing block. You can do this by clicking Copy in the existing block to create a new one.

Hope you can get it to work!

[stav-ros]

Thank you... Such a silly mistake to make. Now it works!

[xingzhe1129]

Hi, I still met the same errors. my data format are Position_10_001_0.00_20240510_005648_Fractions.tiff and I use TILTSERIES_SCANORD_ANGLE___*.tiff for the Frame pattern but it always failed with Cannot find tilted movies. Do yo know why it failed? Thanks.

[abartesaghi]

For the filenames you have, the correct frame pattern to use is: TILTSERIES_SCANORD_ANGLE_*_*_*.tiff
If you used a different pattern that resulted in a failed run, you may need to Delete files and data before re-runing the pre-processing block using correct frame pattern.

[xingzhe1129]

Thank you so much. It does work.

[abartesaghi]

Closing this issue since it seems to be resolved.