Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider a different serialization format for saving sessions #24

Closed
dweindl opened this issue Jan 13, 2025 · 1 comment · Fixed by #149
Closed

Consider a different serialization format for saving sessions #24

dweindl opened this issue Jan 13, 2025 · 1 comment · Fixed by #149

Comments

@dweindl
Copy link
Member

dweindl commented Jan 13, 2025

Trying to load an old session fails with ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'. It might be nice to have something that works across pandas versions (or across different versions of other dependencies).
@dweindl
Copy link
Member Author

dweindl commented Feb 11, 2025

Serializing everything to yaml/json is inconvenient due to large number of somewhat large DataFrames.
Serialization to yaml + extra csv files and zipping everything up works, but is pretty slow. After dropping some unnecessary data (e.g., #141), this should become more practicable.

dweindl added a commit that referenced this issue Feb 12, 2025
@dweindl
Session serialization to zipped yaml/tsv
0c6dddd 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
@dweindl dweindl mentioned this issue Feb 12, 2025
Merged
dweindl added a commit that referenced this issue Feb 14, 2025
@dweindl
Session serialization to zipped yaml/tsv
aade94f 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
@dweindl dweindl mentioned this issue Feb 24, 2025
Closed
dweindl added a commit that referenced this issue Feb 25, 2025
@dweindl
Session serialization to zipped yaml/tsv
31b12d9 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
dweindl added a commit that referenced this issue Feb 25, 2025
@dweindl
Session serialization to zipped yaml/tsv
a4dd061 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
dweindl added a commit that referenced this issue Feb 25, 2025
@dweindl
Session serialization to zipped yaml/tsv
2fedb9a 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
dweindl added a commit that referenced this issue Feb 25, 2025
@dweindl
Session serialization to zipped yaml/tsv
f90d296 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
dweindl added a commit that referenced this issue Feb 25, 2025
@dweindl
Session serialization to zipped yaml/tsv
123022f 
Related to #24. Serialize session data to a zipped yaml file and tsv files instead of pickling, so the data is portable across different versions of Python and other dependencies.

Currently, this is still too slow. If we can reduce the amount of data without losing much, this could be become viable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Session serialization to zipped yaml/tsv ICB-DCM/C-COMPASS
1 participant
@dweindl