concurrent compression and IO for segment writer #3182
Conversation
| Ok(layout) | ||
| }; | ||
| // compress chunks and flush concurrently, former is cpu bound latter is io. | ||
| let (layout, (mut write, segment_specs)) = future::try_join(write_chunks, flush).await?; |
There was a problem hiding this comment.
Given that compressing chunks is CPU bound, is a concurrent wait for write_chunks, flush the right mechanism opposed to putting the CPU bound work on a diff thread? Say a task is currently busy compressing, this also means that no progress can be made on issuing writes, right?
There was a problem hiding this comment.
Yea this assumes we don't block that much while polling compression. My thinking was that compressing an average chunk would be shorter compared to writing the previous compressed chunk into remote storage, so the combined future would have some await points during the write.
Spawning here would be better and I have to add that while doing the next thing I plan to do (multithreaded compression), but that requires changing the VortexWrite a bit, as to spawn we need it to be Send + 'static
This will only have an effect on remote storage. Unfortunately we don't have those on our benchmarks yet