Supporting serde serialization and deserialization of encoder object

[Some1and2-XC]

Hey, I'm not really sure how difficult this would be but would it be at all possible to make the ParCompress<Zlib> encoder support serialization?

The use case for this is to allow for saving the encoder state using something like this to disk so that application errors can be recovered.

[sstadick]

Hi!

I think that would be pretty tricky on the full object:

gzp/src/par/compress.rs

Line 150 in f31a4dd

pub struct ParCompress<F>

. Specifically I don't think the channels and joinhandle will serialize nicely.

You could theoretically grab the dictionary though. Can you describe more about application errors can be recovered? The threads can crash, but you should be able to get errors out of the joinhandles still. What kind of error are you seeing? Or is it an error elsewhere in your app that that causes the failure and you want the encoder dictionary to pick up where it left off?

[Some1and2-XC]

Yeah, I was more so referring to errors in other sections of the application, specifically a long running application that does compression. I want to be able to recreate the state of the compressor even if the entire application crashes and I need to relaunch the binary.

I suppose this is a good explanation of what I had in mind.

Or is it an error elsewhere in your app that that causes the failure and you want the encoder dictionary to pick up where it left off?

Do you think serde is even the right tool for this? Maybe making some of the attributes like the dictionary public and adding something to ParCompressBuilder to allow for setting of the dictionary? I'd love to hear what you think.

[sstadick]

Making the dictionary accessible and also allowing a dict to be passed in do seem like the way to go for that.

You may have to just dump the dictionary at some interval, but the Bytes crate has a serde feature, so that should all wire together okay.

I don't think that I'm going to be able to implement this myself anytime soon, but would be totally open to PRs! I think you described the work already:

• getter for returning an Option<&Bytes> from the ParCompress.
• update for the ParCompressBuilder to set some Option<Bytes> as the dictionary value, default to None.
• add serde as a top level feature that in turn sets the serde feature on the Bytes crate.
• a few tests, gated by the serde feature.

[Some1and2-XC]

Okay, sounds great! I'll see what I can do.