dl/translator: preparatory change for translator porting #25077
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/dt |
CI test resultstest results on build#61812
test results on build#61827
|
|
/dt |
This method loops through the column writers to check if any of them are flush worthy, computes the memory usage in the same loop. Useful for a latter commit that avoids this loop again and needs stats right after append.
The interface implementations keep track of the current memory used by the writer and related reservations.
Adds the following - flush() - flushes all the buffered bytes to the output stream - methods to fetch buffered/flushed bytes
The implementation works with the scheduler to reserve memory as needed.
.. instead of lazy_abort_source. To be used later, they are both connected anyway.
Currently multiplexer is a one shot class with pattern as follows mux = create_mux(); co_await reader.consume(mux...) With the new changes, we want multiplexer to multiplex across scheduling iterations and release resouces inbetween. This commit makes changes to the API support this port. The new pattern would look something like this.. mux = create_mux(); mux.multiplex(reader1...) mux.flush_writers(); // optional mux.multiplex(reader2..) mux.flush_writers(); // optional ... ... result = co_await std::move(mux).finish(); The ability to temporarily flush all the intermediate state and multiplex across multiple readers enables porting to the new scheduler API.
Make the task long running to support batching of data across multiple iterations of scheduler
|
/dt |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backports Required
Release Notes