[consensus/simplex] Spend one fast-skip per leader between finalizations - #4541
Open
ControlCplusControlV wants to merge 3 commits into
Open
[consensus/simplex] Spend one fast-skip per leader between finalizations#4541ControlCplusControlV wants to merge 3 commits into
ControlCplusControlV wants to merge 3 commits into
Conversation
A latched timeout expires its view immediately, so any sustained condition that prevents finalization churned views as fast as nullifications circulated. Each churned view writes a voter journal section that cannot be pruned while `last_finalized` is frozen, so a stall grew the journal (and open file descriptors) without bound. Track which participants have been fast-skipped since the last finalization and latch at most once per participant, leaving later timeouts for that leader to expire on the round's own deadlines. Bits only accumulate while nothing finalizes, so a stall shorter than one pass through the participant set behaves exactly as before.
Contributor
|
This is an interesting idea 👀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fix was mostly for our network which runs a single-leader configurations, where when the single proposer goes down, after SKIP_TIMEOUT progresses views begin churning incredibly fast at network speed. The fix is to basically only fast timeout a leader once and otherwise let views progress at their normal rate, as skipping the same leader on repeat accomplishes nothing and just pushes the network to failure faster.
Issues #626 and #2520 are also both related, where the network will eventually crash because of first too many open file descriptors, as well as out of memory.