chore: event count throttle for squashed commands #4924
Conversation
Signed-off-by: kostas <[email protected]>
| thread_local size_t MultiCommandSquasher::throttle_size_limit_ = | ||
| absl::GetFlag(FLAGS_throttle_squashed); | ||
|
|
||
| thread_local util::fb2::EventCount MultiCommandSquasher::ec_; | ||
|
|
||
| MultiCommandSquasher::MultiCommandSquasher(absl::Span<StoredCmd> cmds, ConnectionContext* cntx, |
There was a problem hiding this comment.
This is used not only from async fiber but also directly from the connection. If we preempt, the connection will also "freeze". I guess this is fine, just mentioning it here for completeness.
There are 3 calls of this and all of them should be ok if we preempt from these flows.
| await cl.execute_command("exec") | ||
|
|
||
| # With the current approach this will overshoot | ||
| # await client.execute_command("multi") |
There was a problem hiding this comment.
I wish we also handled this case as well
src/server/multi_command_squasher.h
Outdated
| @@ -94,6 +104,9 @@ class MultiCommandSquasher { | |||
|
|
|||
| // we increase size in one thread and decrease in another | |||
| static atomic_uint64_t current_reply_size_; | |||
| static thread_local size_t throttle_size_limit_; | |||
| // Used to throttle when memory is tight | |||
| static thread_local util::fb2::EventCount ec_; | |||
There was a problem hiding this comment.
We need this to avoid ThisFiber::Yield, ThisFiber::SleepFor in while(true).
There was a problem hiding this comment.
since it's thread local, it's more efficient to use NoOpLock together with CondVarAny
There was a problem hiding this comment.
Actually, I added a bug here:
static atomic_uint64_t current_reply_size_;
so current_reply_size is not thread local. So what can happen is:
Core 0 -> starts multi/exec
Core 1 -> starts multi/exec but needs to throttle so it goes to sleep waiting on the thread local cond variable
Core 0 -> is done, notifies the thread local
Core 1 -> the fiber never awakes even though we decremented current_reply_size.
Since current_reply_size is global then so should ec_.
P.s. not very happy with this extra synchronization but we only pay it when we are under memory pressure
kostasrim
changed the title
|
@adiholden pinging for an early discussion here |
kostasrim
changed the title
| @@ -15,6 +16,8 @@ | |||
| #include "server/transaction.h" | |||
| #include "server/tx_base.h" | |||
|
|
|||
| ABSL_FLAG(size_t, throttle_squashed, 0, ""); | |||
There was a problem hiding this comment.
@adiholden I will adjust as we said f2f. Looking for some early feedback based on our discussion
src/server/multi_command_squasher.cc
Outdated
| @@ -63,6 +66,10 @@ size_t Size(const facade::CapturingReplyBuilder::Payload& payload) { | |||
| } // namespace | |||
|
|
|||
| atomic_uint64_t MultiCommandSquasher::current_reply_size_ = 0; | |||
| thread_local size_t MultiCommandSquasher::throttle_size_limit_ = | |||
| absl::GetFlag(FLAGS_throttle_squashed); | |||
There was a problem hiding this comment.
As discussed this morning multiply by thread count. The limit should be per thread and the current_reply_size_ is global counter
There was a problem hiding this comment.
Yes I know, I even wrote a comment above that I will follow up with this 😄
I wanted to know if you have anything else to add 😄
Throttle/preempt flows that use multi command squasher and crb crosses the limit.