Feature Request: decompress from input `Iterator` to output `Iterator` (for low RAM usage in embedded Rust)

[neil-morrison44]

Hi,

Feel free to point me in the right direction if this is supported just now (or is implemented in another crate I've not found yet), but what I'm looking for is a function with an input similar to decompress_slice_iter_to_slice but that returns an iterator I can iterate over to get decompressed data. I'm trying to run on RAM constrained devices in no_std, eventually reading image files and writing pixels directly to the display.

Ideally, and I might be far off here from how it'd actually need to work:

• the decompress_* function gets called & is given an Iterator that returns u8s
• it returns a struct implementing core::iter::Iterator over u8s
• calling next() on the returned iter will fill a (small) internal buffer by calling the input Iterator, decompressing, then transparently returning values from the output buffer until it's consumed, refilling it as needed

Ideally the size of the internal buffer would be definable by the user (const generic?, heapless does something similar) within whatever bounds make sense.

This is potentially related to #25 but more targeted towards resource-constrained devices (embedded Rust etc)

[oyvindln]

Yeah it seems reasonable to have something like this for cases where flate2 is too heavy. I've not personally done much work on this library in quite a while, might look into it at some point if no one else starts tinkering with it but might take a while.

[neil-morrison44]

I'm still thinking about this one occasionally - I think it'd be fairly easy/nice to do with a generator which'll be available in the 2024 edition. Since iterator in / iterator out things should become really neat with generators.

I'll probably have free some time at the end of the year to look into it, unless anyone else wants to.
(Which'll align with the 2024 edition's planned release in early 2025)

[peckpeck]

Please be aware that even with an iterator you'd still need a buffer of the size of the compression window (32k max if not known in advance).

I just made a function that could help to implement this : https://github.com/Frommi/miniz_oxide/blob/master/miniz_oxide/src/inflate/core.rs#L1403

[neil-morrison44]

For embedded things knowing that you'll require a 32k buffer ahead of time is completely fine (unless you're building for a chip with less than 32k obviously) since that's a size known at compile time.

It's the input buffer & output buffer being unknown sizes that makes embedded use impossible, assuming a use case like reading a compressed image from an SD card and drawing to a display or reading compressed sound from a network socket & playing it on a speaker.

---

Also, didn't realise I didn't attach this at the time but I built out a iterator -> Iterator decompressor using generators for a much simpler algorithm to see what it's like https://github.com/neil-morrison44/heatshrink-gen-rs

Though I thought generators were closer to being included in non-nightly Rust at the time, so I've not revisited it since.

[peckpeck]

In this case it is easy to develop, here is how I would do it 29d3a0f

Please note 2 things:

• there is no error handling inside iterators, so an error is translated into the end of the stream
• the buffer is fixed and always 32k (this could be refactored)

[oyvindln]

That looks reasonable, maybe you could make a PR? The buffer size could be made changeable using const generics.

I've been thinking about adding the bits to make compression work without allocation as well if there is a use case for it, mainly with the "fast" algorithm that uses a 4k window and the dumb RLE one that is less efficient but just needs to look at the previous byte, though the algorithm using the full window could be made to work too.