Make log servers return an empty version range only when it is correct to do so #12188

sbodagala · 2025-06-09T19:38:10Z

In the context of version vector/unicast, make log servers return an empty version range (on peeks) only when it is correct to do so. This is so the receiver will receive all versions (even though the sender is sending an empty version range) that it is supposed to receive.

Changes:

Make the cluster controller collect the set of log servers participated in recovery and propagate that information to the other processes.
Extend the ServerPeekCursor to take a flag that tells the source log server whether it can return an empty version range or not (in the context of version vector/unicast), and make the source log server return an empty version range only when this flag is set.
Make the peek APIs set the flag appropriately when initializing ServerPeekCursors.
Also, take the internal logic used by SetPeekCursor and MergedPeekCursor into account while initializing the above mentioned flag.

Testing:

Joshua id (with version vector disabled): 20250609-200159-sre-0c15ba89fee21207 (no failures).

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

an empty version range (on peeks) only when it is correct to do so. This is so the receiver will receive all versions (even though the sender is sending an empty version range) that it is supposed to receive. Changes: - Make the cluster controller collect the set of log servers participated in recovery and propagate that information to the other processes. - Extend the ServerPeekCursor to take a flag that tells the source log server whether it can return an empty version range or not (in the context of version vector/unicast), and make the source log server return an empty version range only when this flag is set. - Make the peek APIs set the flag appropriately when initializing ServerPeekCursors. - Also, take the internal logic used by SetPeekCursor and MergedPeekCursor into account while initializing the above mentioned flag.

foundationdb-ci · 2025-06-09T20:04:03Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4ec7e4d
Duration 0:25:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-09T20:17:22Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4ec7e4d
Duration 0:38:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-09T20:27:47Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4ec7e4d
Duration 0:49:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-09T20:29:47Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 4ec7e4d
Duration 0:51:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-09T20:38:11Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 4ec7e4d
Duration 0:59:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-09T20:40:04Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 4ec7e4d
Duration 1:01:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

dlambrig · 2025-06-11T19:16:19Z

fdbserver/TagPartitionedLogSystem.actor.cpp

+    int bestSet,
+    int& bestServer,
+    Optional<Version> end,
+    Optional<std::map<uint8_t, std::vector<uint16_t>>> knownLockedTLogIds) {


todo - do both these have to be optional?

TagPartitionedLogSystem::peekLogRouter(), an invoker of this, takes these arguments as Optional. Outside the recovery context, I don't think these arguments need to be specified so I think it makes sense to have these as optional. Also I plan to feed the locked server list from the serialized TagPartitionedLogSystem to this method from TagPartitionedLogSystem::peekLogRouter() in a follow up PR, so I think it makes sense to have this argument as Optional in that sense too. The "Optional end" argument has a value assigned to it in both the contexts that this method is currently getting called from, so it may not need to be Optional (but we may not want to change this purely based on the current invokers).

Some more context, I added the argument Optional<Version> end to TagPartitionedLogSystem::peekLogRouter(). In that iteration end would have to be present and not equal to -1 to indicate we are in recovery and should return up to end.

It would be good to not add any extra work for the non-version-vector case. ie. only call resetBestServerIfNotLocked if version vector is enabled, as was done for resetBestServerIfNotAvailable()

I think knownLockedTLogIds could be a reference, maybe const.

dlambrig · 2025-06-23T20:15:47Z

fdbserver/TagPartitionedLogSystem.actor.cpp

@@ -1997,11 +2038,13 @@ Optional<DurableVersionInfo> TagPartitionedLogSystem::getDurableVersion(UID dbgi
 	std::vector<TLogLockResult> results;
 	std::string sServerState;
 	LocalityGroup unResponsiveSet;
+	std::vector<uint16_t> lockedTLogIds;


We can lockedTLogIds.reserve(logSet->logServers.size()); to prevent repeated reallocations.

dlambrig · 2025-06-23T20:25:59Z

fdbserver/TagPartitionedLogSystem.actor.cpp

+			                          results[new_safe_range_begin].end,
+			                          results,
+			                          failedLogsCompletePolicy,
+			                          lockedTLogIds);


to avoid copying the vector into the constructor here, I think you can define DurableVersionInfo::std::vector<uint16_t> knownLockedTLogIds; to be DurableVersionInfo::std::vector<uint16_t> &&knownLockedTLogIds;. Or alternatively, make a constructor for DurableVersionInfo(...lockedTLogIds) that sets lockedTLogIds = std::move(lockedTLogIds); The same could be done with results.

dlambrig · 2025-06-23T22:54:11Z

fdbserver/include/fdbserver/TLogInterface.h


 	TLogPeekRequest(Version begin,
 	                Tag tag,
 	                bool returnIfBlocked,
 	                bool onlySpilled,
 	                Optional<std::pair<UID, int>> sequence = Optional<std::pair<UID, int>>(),
-	                Optional<Version> end = Optional<Version>())
+	                Optional<Version> end = Optional<Version>(),
+	                Optional<bool> returnEmptyIfStopped = Optional<bool>())


Is the new variable returnEmptyIfStopped necessary? Prior to the PR, Optional end being present indicated "return empty if stopped" and end should only be set in that case. The Optional end could be renamed to clarify that intention.

dlambrig · 2025-06-24T01:42:46Z

fdbserver/TagPartitionedLogSystem.actor.cpp

+    int bestSet,
+    int& bestServer,
+    Optional<Version> end,
+    Optional<std::map<uint8_t, std::vector<uint16_t>>> knownLockedTLogIds) {


Some more context, I added the argument Optional<Version> end to TagPartitionedLogSystem::peekLogRouter(). In that iteration end would have to be present and not equal to -1 to indicate we are in recovery and should return up to end.

It would be good to not add any extra work for the non-version-vector case. ie. only call resetBestServerIfNotLocked if version vector is enabled, as was done for resetBestServerIfNotAvailable()

I think knownLockedTLogIds could be a reference, maybe const.

dlambrig · 2025-06-24T01:53:41Z

fdbserver/TagPartitionedLogSystem.actor.cpp

+    Tag tag,
+    bool useSatellite,
+    Optional<Version> end,
+    Optional<std::map<uint8_t, std::vector<uint16_t>>> knownStoppedTLogIds) {


I think knownStoppedTLogIds can be a reference, maybe pipe it through as a const as well.

dlambrig · 2025-06-24T01:58:20Z

fdbserver/TagPartitionedLogSystem.actor.cpp

 		quorumResults.push_back(quorum(tLogCommitResults, tLogCommitResults.size() - it->tLogWriteAntiQuorum));
 		logGroupLocal++;
 	}

 	return minVersionWhenReady(waitForAll(quorumResults), allReplies);
 }

+void TagPartitionedLogSystem::resetBestServerIfNotLocked(


can you add a few comments describing this function's purpose, so maintainers do not have to dig through PR history, thanks

dlambrig · 2025-06-24T02:03:39Z

fdbserver/TagPartitionedLogSystem.actor.cpp

@@ -667,13 +668,31 @@ Future<Version> TagPartitionedLogSystem::push(const ILogSystem::PushVersionSet&
 			tLogCommitResults.push_back(commitSuccess);
 			location++;
 		}
+


sbodagala requested review from dlambrig and jzhou77 June 9, 2025 19:38

sbodagala mentioned this pull request Jun 9, 2025

Make log servers return an empty version range only when it is correct to do so #12171

Closed

5 tasks

dlambrig reviewed Jun 11, 2025

View reviewed changes

Abdullah4345 approved these changes Jun 21, 2025

View reviewed changes

dlambrig requested changes Jun 24, 2025

View reviewed changes

Make log servers return an empty version range only when it is correct to do so #12188

Are you sure you want to change the base?

Make log servers return an empty version range only when it is correct to do so #12188

Uh oh!

Conversation

sbodagala commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 9, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbodagala commented Jun 9, 2025 •

edited

Loading