Issue with class types in pyaldata "idx_" fields

[CeciliaGallego]

In pyaldata, "idx_" variables contain both "int" and "numpy.ndarray" values, which triggers an error in pyaldata library when using "mat2dataframe" to load the file. It should either be all entries "int" or "numpy.ndarray" (I think numpy.ndarray is more flexible).

Example of one pyaldata file triggering the error:

" idx_trial_start idx_trial_end idx_cursor_update
0 <class 'int'> <class 'int'> <class 'int'>
1 <class 'int'> <class 'int'> <class 'int'>
2 <class 'int'> <class 'int'> <class 'numpy.ndarray'>
3 <class 'int'> <class 'int'> <class 'int'>
4 <class 'int'> <class 'int'> <class 'numpy.ndarray'>

idx_before_camera_trigger idx_CPI idx_lick
0 <class 'int'> <class 'int'> <class 'numpy.ndarray'>
1 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
2 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
3 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
4 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>

            idx_motion        idx_reward_number        idx_session_timer  

0 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
1 <class 'numpy.ndarray'> <class 'int'> <class 'numpy.ndarray'>
2 <class 'numpy.ndarray'> <class 'int'> <class 'numpy.ndarray'>
3 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
4 <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> "

Error from pyaldata repo triggered:

---

AttributeError Traceback (most recent call last)
Cell In[11], line 5
1 session = params.root / "raw" / "M056" / "M056_2025_02_20_11_30"
3 pyal_file = session / f"{session.name}_pyaldata_0.mat"
----> 5 df = pyal.mat2dataframe(pyal_file, shift_idx_fields=True)

File ~/Documents/decorrelation-bci/packages/pyaldata/pyaldata/io.py:52, in mat2dataframe(path, shift_idx_fields, td_name)
48 td_name = real_keys[0]
50 df = pd.DataFrame(mat[td_name])
---> 52 df = data_cleaning.clean_0d_array_fields(df)
53 df = data_cleaning.clean_integer_fields(df)
55 if shift_idx_fields:

File ~/Documents/decorrelation-bci/packages/pyaldata/pyaldata/utils.py:28, in copy_td..wrapper(*args, **kwargs)
25 if not isinstance(args[0], pd.DataFrame):
26 raise ValueError(f"first argument of {func.name} has to be a pandas DataFrame")
---> 28 return func(args[0].copy(), *args[1:], **kwargs)

File ~/Documents/decorrelation-bci/packages/pyaldata/pyaldata/data_cleaning.py:45, in clean_0d_array_fields(df)
43 for c in df.columns:
44 if isinstance(df[c].values[0], np.ndarray):
---> 45 if all([arr.ndim == 0 for arr in df[c]]):
46 df[c] = [arr.item() for arr in df[c]]
48 return df

AttributeError: 'int' object has no attribute 'ndim'

[AtMostafa]

I've been going around this by commenting out ---> 52 df = data_cleaning.clean_0d_array_fields(df). But I agree, needs a proper fix

[martinesparza]

isn't this fixed here??

[CeciliaGallego]

It is! We didn't have the last version. Sorry!! You can close the issue.