Skip to content

modules/zstd: add rle block decoder #1213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 16 commits into from
Closed

modules/zstd: add rle block decoder #1213

wants to merge 16 commits into from

Conversation

lpawelcz
Copy link
Contributor

@lpawelcz lpawelcz commented Nov 30, 2023 •
edited
Loading

This PR adds RleBlockDecoder Proc which is responsible for decoding ZSTD RLE Blocks. More information about RLE Block's format can be found in Section 3.1.1.2.2 of RFC8788.
RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

  • RleDataPacker
  • BatchPacker

Those are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

NOTE: this is based on #1212 , please ignore commits from that branch when reviewing.

This is part of #1211

CC @proppy

@lpawelcz lpawelcz mentioned this pull request Nov 30, 2023
Closed
@lpawelcz lpawelcz changed the title modules/zstd: add RLE block decoder modules/zstd: add rle block decoder Nov 30, 2023
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch 2 times, most recently from 1f02f00 to 842cfcf Compare December 12, 2023 11:18
@rw1nkler rw1nkler mentioned this pull request Dec 13, 2023
Open
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch 4 times, most recently from 823d5a8 to ad311e0 Compare December 19, 2023 11:38
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch from ad311e0 to f1c4596 Compare December 27, 2023 14:22
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch 2 times, most recently from 81fca94 to 4c1bb56 Compare January 4, 2024 14:27
@lpawelcz lpawelcz mentioned this pull request Jan 5, 2024
Closed
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch 3 times, most recently from afbdd66 to 80b5bc7 Compare January 15, 2024 14:53
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch from 80b5bc7 to 2b45f83 Compare February 15, 2024 11:44
rw1nkler and others added 13 commits February 21, 2024 14:57
@rw1nkler @lpawelcz
modules/zstd: Add buffer library
a8c0510 
This commit adds a DSLX Buffer library that provides the Buffer struct,
and helper functions that can be used to operate on it. The Buffer
is meant to be a storage for data coming from the channel. It acts like
a FIFO, allowing data of any length to be put in or popped out of it.
Provided DSLX tests verify the correct behaviour of the library.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add Buffer use-case example
7828870 
This commit adds a simple test that shows, how one can use the Buffer
struct inside a Proc.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
@lpawelcz
modules/zstd/buffer: Add benchmarking rules
3748388 
Signed-off-by: Pawel Czarnecki <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add library for parsing magic number
fc57626 
This commit adds the library with functions for parsing a magic number and
tests that verify its correctness.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <[email protected]>
@rw1nkler @rdob-ant @lpawelcz
modules/zstd: Add library for parsing frame header
1d11225 
This commit adds the library with functions for parsing a frame header.
The provided tests verify the correcness of the library.

Internal-tag: [#49967]
Co-authored-by: Roman Dobrodii <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz
modules/zstd/frame_header: Add benchmarking rules
374b000 
Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz
dependency_support/libzstd: Make zstd_errors.h public
4f0a064 
Required for expected_status inference in C++ tests for ZSTD decoder
components

Internal-tag: [#53465]
Signed-off-by: Pawel Czarnecki <[email protected]>
@rw1nkler @lpawelcz
dependency_support: Add decodecorpus binary
6ddea32 
Internal-tag: [#50967]
Signed-off-by: Robert Winkler <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add data generator library
b4fb1ec 
This commit adds a binary that calls decoding to generate data and loads
it into a vector of bytes.

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add zstd frame header tests
e25cbe1 
Internal-tag: [#50967]
Co-authored-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add common zstd definitions
dd34d02 
Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
@rw1nkler @lpawelcz
modules/zstd: Add raw block decoder
e6a0e0f 
Internal-tag: [#51343]
Signed-off-by: Robert Winkler <[email protected]>
@lpawelcz
modules/zstd/raw_block_decoder: Add benchmarking rules
09267ae 
Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz
modules/zstd: Add rle block decoder
d31c6eb 
Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz
modules/zstd/rle_block_dec: Specify fifo depths for internal channels
4c19f10 
Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz
modules/zstd/rle_block_decoder: Add benchmarking rules
5689ed9 
Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
@lpawelcz lpawelcz force-pushed the 51473-rle-block-parser branch from 2b45f83 to 5689ed9 Compare February 21, 2024 14:06
lpawelcz added a commit to antmicro/xls that referenced this pull request Feb 21, 2024
@lpawelcz
[TEMP] Add RLE Block Decoder
f1ed899 
google#1213

modules/zstd: Add rle block decoder

Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/rle_block_dec: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/rle_block_decoder: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
lpawelcz added a commit to antmicro/xls that referenced this pull request Mar 7, 2024
@lpawelcz
[TEMP] Add RLE Block Decoder
a0412c7 
google#1213

modules/zstd: Add rle block decoder

Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/rle_block_dec: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/rle_block_decoder: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <[email protected]>
@cdleary cdleary added the app Application level functionality (examples, uses of XLS stack) label Mar 27, 2024
@proppy
Copy link
Member

proppy commented Mar 29, 2024

should we close this and focus on reviewing #1315 ?

@lpawelcz
Copy link
Contributor Author

Closing. The review will take place in #1315

@lpawelcz lpawelcz closed this Mar 29, 2024
@tmichalak tmichalak deleted the 51473-rle-block-parser branch April 29, 2025 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
app Application level functionality (examples, uses of XLS stack)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lpawelcz @proppy @cdleary @rw1nkler