Dynamically change replication throttle for an on-going rebalance

[stanislavkozlovski]

Now that we have support for replication throttle (#748, thanks again @aaronlevin-stripe), I thought it would be great to have a way to change a throttle in the middle of a rebalance.

In cases where you set one throttle but later realized it wasn't satisfactory (e.g things going too slow or too unstable), it is invaluable to have a quick way to revert it.

I thought adding a new endpoint for updating throttle (or maybe extending /admin?) may be straightforward to do.
What are people's thoughts on this?

[ShubhamRwt]

Hello, Is this issue still valid? If yes, then can I work upon it ?

[StarKnightt]

Hello, Is this issue still valid? If yes, then can I work upon it ?

Nah, It's fixed.

[imans777]

it's not fixed I believe. we can't change replication_throttle in the middle of a rebalance operation. Only max partition numbers and concurrent movements can be changed via /admin API.
I still think it's an important feature that should be implemented. I'm now in the middle of a rebalance on a very high load cluster (~10M records/s) and I'm struggling to find a good value for this. I have to stop execution each time and wait for lags to drop (which takes hours) and then try again with new value.

[oleg-vlsk]

@stanislavkozlovski @imans777
Hi, is this issue still relevant? If so, can I work on it?

[imans777]

@stanislavkozlovski @imans777 Hi, is this issue still relevant? If so, can I work on it?

Well I'm not a contributor yet, but to me it's an invaluable feature 😅
I still don't get the point @StarKnightt mentioned, because I don't see it fixed.

[oleg-vlsk]

@efeg @stanislavkozlovski
I'd be glad to take a shot at this. Should someone assign me to the issue first, or can I just start working on it and submit a PR when it's done?

[imans777]

@efeg @stanislavkozlovski I'd be glad to take a shot at this. Should someone assign me to the issue first, or can I just start working on it and submit a PR when it's done?

I'd be very glad to have this feature too 😀
As I've looked in this project for a while, it seems that if an issue gets labels as this issue has ("good first issue", etc), it is going to be accepted if a PR is submitted. If the change is big, it needs a design proposal, but I believe this issue is not one of them. So I believe if you just add this feature (e.g. to the current /admin API) and write a test for it, the contributors would accept it. Because change is not going to be a big one, if you ping them they'll probably be accessible (for big changes I think they're too busy to review and submit :D).