Skip to content

Polish better decoder (Testing) #7428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 65 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
cc6dd14
update
XiangpengHao Sep 4, 2024
5837fc7
update
XiangpengHao Dec 22, 2024
fec6313
update
XiangpengHao Dec 28, 2024
948db87
update
XiangpengHao Dec 29, 2024
8c50d90
poc reader
XiangpengHao Dec 30, 2024
f5422ce
update
XiangpengHao Dec 31, 2024
dfdc1b6
avoid recreating new buffers
XiangpengHao Dec 31, 2024
3c526f8
update
XiangpengHao Dec 31, 2024
53f5fad
bug fix
XiangpengHao Dec 31, 2024
56980de
selective cache
XiangpengHao Jan 1, 2025
4dd1b6b
clean up changes
XiangpengHao Jan 1, 2025
f8f983e
clean up more and format
XiangpengHao Jan 1, 2025
882aaf1
cleanup and add docs
XiangpengHao Jan 1, 2025
c8bdbcf
switch to mutex instead of rwlock
XiangpengHao Jan 2, 2025
cdb1d85
revert irrelevant changes
XiangpengHao Jan 2, 2025
69720e5
submodule
XiangpengHao Jan 3, 2025
a9550ab
update
XiangpengHao Jan 3, 2025
be1435f
rebase
XiangpengHao Jan 6, 2025
e4d9eb7
Merge remote-tracking branch 'upstream/main' into better-decoder
XiangpengHao Jan 8, 2025
21e015b
remove unrelated changes
XiangpengHao Jan 8, 2025
bbc3595
Merge remote-tracking branch 'upstream/main' into better-decoder
XiangpengHao Jan 10, 2025
547fb46
fix clippy
XiangpengHao Jan 10, 2025
05c8c8f
make various ci improvements
XiangpengHao Jan 10, 2025
314fda1
Merge remote-tracking branch 'apache/main' into better-decoder
alamb Mar 21, 2025
c895dd2
whitespace
alamb Mar 21, 2025
3cf0a98
Reduce some ugliness, avoid unwrap
alamb Mar 21, 2025
7b72f9d
more factory
alamb Mar 21, 2025
5bdf51a
lint
alamb Mar 22, 2025
a77e1e7
Merge remote-tracking branch 'apache/main' into better-decoder
alamb Mar 26, 2025
90a55d5
Isolate reader cache more
alamb Mar 26, 2025
9ffa81c
Merge remote-tracking branch 'apache/main' into better-decoder
alamb Mar 27, 2025
7c10b4a
Merge remote-tracking branch 'apache/main' into better-decoder
alamb Mar 28, 2025
822760c
Add benchmark for parquet reader with row_filter and project settings
zhuqi-lucas Apr 10, 2025
31a544f
fix clippy
zhuqi-lucas Apr 10, 2025
b16428d
change bench mark to use asyn read to trigger the page cache
zhuqi-lucas Apr 11, 2025
1aacd01
fix
zhuqi-lucas Apr 11, 2025
2d58006
Merge remote-tracking branch 'upstream/main' into benchmark_row_filter
zhuqi-lucas Apr 11, 2025
768826e
fix
zhuqi-lucas Apr 11, 2025
f624b91
Update comments, add background
alamb Apr 11, 2025
6c28e44
incremently addressing the comments
zhuqi-lucas Apr 11, 2025
69a2617
Fix bool random
zhuqi-lucas Apr 11, 2025
b044813
Merge commit '69a2617' into alamb/docs_for_bench
alamb Apr 11, 2025
6a37818
fixup
alamb Apr 11, 2025
2f6ccbb
Add fn switch and project enum
zhuqi-lucas Apr 11, 2025
994c747
Merge pull request #1 from alamb/alamb/docs_for_bench
zhuqi-lucas Apr 11, 2025
d0a656b
Fix clippy
zhuqi-lucas Apr 11, 2025
67480b9
Address comment
zhuqi-lucas Apr 12, 2025
16bc1bf
Add float(half set) and int(full set) change
zhuqi-lucas Apr 12, 2025
a4bedbd
Merge branch 'benchmark_row_filter' of github.com:zhuqi-lucas/arrow-r…
zhuqi-lucas Apr 12, 2025
d0ab2fe
Fix corner case: skipping page should also make dic page to none
zhuqi-lucas Apr 12, 2025
7638c41
Address comments
zhuqi-lucas Apr 13, 2025
8fc992b
Merge branch 'benchmark_row_filter' into better-decoder
zhuqi-lucas Apr 14, 2025
9271cc9
Set compression
zhuqi-lucas Apr 14, 2025
8e00ac5
fix
zhuqi-lucas Apr 14, 2025
36346aa
Merge branch 'benchmark_row_filter' into better-decoder
zhuqi-lucas Apr 14, 2025
890519e
Update comments
alamb Apr 14, 2025
7eb0476
refactor filter column indexes
alamb Apr 14, 2025
22c7b39
Read from in memory buffer
alamb Apr 14, 2025
86878ab
Merge remote-tracking branch 'apache/main' into benchmark_row_filter
alamb Apr 14, 2025
5ae9b58
celanu
alamb Apr 14, 2025
1effe88
Test both sync and async readers
alamb Apr 14, 2025
74abec0
Merge branch 'benchmark_row_filter' into better-decoder
zhuqi-lucas Apr 15, 2025
6ea0eef
Merge branch 'main' into better-decoder
zhuqi-lucas Apr 18, 2025
0c3aa9b
Improve the performance for skip record
zhuqi-lucas Apr 19, 2025
d26de88
Perf: make the cache not missing to avoid some clickbench regression
zhuqi-lucas May 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion parquet/src/arrow/array_reader/byte_view_array.rs
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,8 @@ impl ByteViewArrayDecoderDictionary {
}
}

/// Reads the next indexes from self.decoder
/// Reads the next `len` indexes from self.decoder
///
/// the indexes are assumed to be indexes into `dict`
/// the output values are written to output
///
Expand Down Expand Up @@ -464,6 +465,8 @@ impl ByteViewArrayDecoderDictionary {
}
}

output.views.reserve(len);

// Calculate the offset of the dictionary buffers in the output buffers
// For example if the 2nd buffer in the dictionary is the 5th buffer in the output buffers,
// then the base_buffer_idx is 5 - 2 = 3
Expand Down
1 change: 1 addition & 0 deletions parquet/src/arrow/arrow_reader/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,7 @@ impl<T: ChunkReader + 'static> ReaderPageIterator<T> {
.map(|i| i[rg_idx][self.column_idx].page_locations.clone());
let total_rows = rg.num_rows() as usize;
let reader = self.reader.clone();
// todo: add cache???

SerializedPageReader::new(reader, column_chunk_metadata, total_rows, page_locations)?
.add_crypto_context(
Expand Down
Loading