Hot shard design documentation #12413

dlambrig · 2025-10-02T19:51:03Z

Hot shard documentation

[X]explains what a hot shard is
[X] - explains how to avoid a hot shard (e.g. structure data and/or access patterns in a certain way)
[X] - explains the server management side options for fixing a hot shard, should it arise. Please emphasize that the options are rather bad / near non-existent and hot shards should be avoided by design for this reason

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2025-10-02T20:27:54Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 6c10d8b
Duration 0:36:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:31:40Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 7094c6e
Duration 0:34:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:39:30Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 6c10d8b
Duration 0:48:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:39:35Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 6c10d8b
Duration 0:48:20
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:43:18Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 6c10d8b
Duration 0:52:03
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/foundationdb-kubernetes-sidecar:${FDB_VERSION}-${COMMIT_SHA}-1 --file Dockerfile --target foundationdb-kubernetes-sidecar .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-02T20:44:45Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 7094c6e
Duration 0:47:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:47:45Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 7094c6e
Duration 0:50:29
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/ycsb:${FDB_VERSION}-${COMMIT_SHA} --file Dockerfile --target ycsb .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-02T20:57:02Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 6c10d8b
Duration 1:05:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T20:57:24Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 7094c6e
Duration 1:00:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-02T21:00:10Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 7094c6e
Duration 1:02:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77

This looks great!

I think it's also worth documenting the max shard size is dynamically calculated in

int64_t getMaxShardSize(double dbSizeEstimate) {
    int64_t size = std::min((SERVER_KNOBS->MIN_SHARD_BYTES + (int64_t)std::sqrt(std::max<double>(dbSizeEstimate, 0)) *
                                                                 SERVER_KNOBS->SHARD_BYTES_PER_SQRT_BYTES) *
                                SERVER_KNOBS->SHARD_BYTES_RATIO,
                            (int64_t)SERVER_KNOBS->MAX_SHARD_BYTES);

i.e., a formula of (MIN_SHARD_BYTES + sqrt(DB_size) * SHARD_BYTES_PER_SQRT_BYTES) * SHARD_BYTES_RATIO, and then max'ed at MAX_SHARD_BYTES.

design/hotshard.md

foundationdb-ci · 2025-10-05T19:36:52Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 88560df
Duration 0:04:14
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (88560df8d4f849299775dd5a6b143796c510acf8) for primary source and source version 88560df8d4f849299775dd5a6b143796c510acf8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T19:36:52Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 88560df
Duration 0:04:14
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (88560df8d4f849299775dd5a6b143796c510acf8) for primary source and source version 88560df8d4f849299775dd5a6b143796c510acf8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-05T19:36:54Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 88560df
Duration 0:04:14
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (88560df8d4f849299775dd5a6b143796c510acf8) for primary source and source version 88560df8d4f849299775dd5a6b143796c510acf8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T19:36:54Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 88560df
Duration 0:04:14
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (88560df8d4f849299775dd5a6b143796c510acf8) for primary source and source version 88560df8d4f849299775dd5a6b143796c510acf8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T19:37:00Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 88560df
Duration 0:04:19
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (88560df8d4f849299775dd5a6b143796c510acf8) for primary source and source version 88560df8d4f849299775dd5a6b143796c510acf8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T19:48:01Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 4bfc61f
Duration 0:24:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T20:04:34Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4bfc61f
Duration 0:40:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-05T20:12:05Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4bfc61f
Duration 0:48:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-09T16:38:34Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 0bef483
Duration 0:49:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-09T18:09:21Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 0bef483
Duration 2:19:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-09T18:12:08Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 0bef483
Duration 2:22:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

dlambrig

I would put this under Server Knobs Related to Hot Shards and probably group it with Hot Shard Throttling Knobs

I want to call out that this is a separate feature. We will be testing it soon and once it makes production we can update the doc

dlambrig

it sounds like this should go under a different section? or maybe make the section not read-specific

probably makes sense for the section to not be read-specific

foundationdb-ci · 2025-10-10T15:41:27Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 181049e
Duration 0:11:51
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-10T15:54:25Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 181049e
Duration 0:24:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-10T16:12:27Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 181049e
Duration 0:42:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-10T16:14:28Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 181049e
Duration 0:44:51
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/foundationdb-kubernetes-sidecar:${FDB_VERSION}-${COMMIT_SHA}-1 --file Dockerfile --target foundationdb-kubernetes-sidecar .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-10T16:19:58Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 181049e
Duration 0:50:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-10T18:16:16Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 181049e
Duration 2:46:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-10T18:24:36Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 181049e
Duration 2:55:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2025-10-13T17:41:43Z

design/hotshard.md

+
+### Storage Server Impact
+
+- The storage servers hosting the hot shard become saturated with write traffic, causing their write queues to grow ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


This appears to be blobWorker, not storage servers. Durability lag causes rate limiting at line 910.

jzhou77 · 2025-10-13T17:44:19Z

design/hotshard.md

+
+- The storage servers hosting the hot shard become saturated with write traffic, causing their write queues to grow ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).
+
+- Storage server metrics report high `bytesInput` and increasing queue depths, which are monitored by Ratekeeper ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).


Sees to be line 835 and 845.

jzhou77 · 2025-10-13T17:45:00Z

design/hotshard.md

+
+- Storage server metrics report high `bytesInput` and increasing queue depths, which are monitored by Ratekeeper ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).
+
+- If the storage server's write queue size exceeds thresholds relative to available space and target ratios, it becomes the bottleneck for the entire cluster ([Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


jzhou77 · 2025-10-13T17:45:47Z

design/hotshard.md

+
+### Cluster-Wide Rate Limiting
+
+- Ratekeeper continuously monitors storage server queues and bandwidth metrics to determine cluster-wide transaction rate limits ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


line numbers are wrong

jzhou77 · 2025-10-13T17:46:14Z

design/hotshard.md

+
+- Ratekeeper continuously monitors storage server queues and bandwidth metrics to determine cluster-wide transaction rate limits ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).
+
+- When a storage server's write queue becomes excessive, Ratekeeper sets the limit reason to `storage_server_write_queue_size` and reduces the cluster-wide transaction rate proportionally ([Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973), [Ratekeeper:1440](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L1440)).


jzhou77 · 2025-10-13T17:47:18Z

design/hotshard.md

+
+- This causes **all transactions across the cluster** to be throttled, even those not touching the hot shard, because the cluster cannot commit faster than its slowest storage server can durably persist data.
+
+- The rate limit is calculated as: `limitTps = min(actualTps * maxBytesPerSecond / inputRate, maxBytesPerSecond * MAX_TRANSACTIONS_PER_BYTE)`, ensuring the cluster doesn't overwhelm the saturated storage server ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).


line 795

limitTps = std::min(actualTps * maxBytesPerSecond / std::max(1.0e-8, inputRate), maxBytesPerSecond * SERVER_KNOBS->MAX_TRANSACTIONS_PER_BYTE);

foundationdb-ci · 2025-10-13T21:14:43Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 3c97f14
Duration 0:11:55
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-13T21:32:55Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 3c97f14
Duration 0:30:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77

Looks great and thank you!

foundationdb-ci · 2025-10-13T21:49:30Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 3c97f14
Duration 0:46:40
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/ycsb:${FDB_VERSION}-${COMMIT_SHA} --file Dockerfile --target ycsb .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-13T21:51:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 3c97f14
Duration 0:48:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-13T22:01:34Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 3c97f14
Duration 0:58:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-13T22:03:59Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 3c97f14
Duration 1:01:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

kakaiu · 2025-10-13T23:10:33Z

design/hotshard.md

+- Prevents moving tiny/idle shards during rebalancing
+- Location: [ServerKnobs:368](https://github.com/apple/foundationdb/blob/7.3.0/fdbclient/ServerKnobs.cpp#L368)
+
+**`DD_ENABLE_REBALANCE_STORAGE_QUEUE_WITH_LIGHT_WRITE_SHARD`** (bool, default: `true`)


The feature flag is ENABLE_REBALANCE_STORAGE_QUEUE instead of DD_ENABLE_REBALANCE_STORAGE_QUEUE_WITH_LIGHT_WRITE_SHARD. This feature is off by default.

dlambrig requested a review from jzhou77 October 2, 2025 19:52

Hot shard design documentation

7094c6e

dlambrig force-pushed the hotshard-doc branch from 6c10d8b to 7094c6e Compare October 2, 2025 19:57

jzhou77 requested changes Oct 3, 2025

View reviewed changes

dlambrig requested a review from nicmorales9 October 4, 2025 20:43

dlambrig force-pushed the hotshard-doc branch 2 times, most recently from 88560df to dd60954 Compare October 5, 2025 19:33

dlambrig marked this pull request as ready for review October 5, 2025 19:33

dlambrig force-pushed the hotshard-doc branch from dd60954 to e05b8fb Compare October 5, 2025 19:38

Added resolution section, review comments

9594771

dlambrig force-pushed the hotshard-doc branch from e05b8fb to 9594771 Compare October 5, 2025 19:44

dlambrig marked this pull request as draft October 5, 2025 19:45

dlambrig commented Oct 10, 2025

View reviewed changes

Respond to review comments

181049e

dlambrig requested a review from nicmorales9 October 10, 2025 15:29

jzhou77 requested changes Oct 13, 2025

View reviewed changes

Fix line numbers to 7.3

3c97f14

dlambrig requested review from jzhou77 and removed request for johscheuer and jzhou77 October 13, 2025 21:06

jzhou77 approved these changes Oct 13, 2025

View reviewed changes

dlambrig merged commit 5ed6a2a into apple:main Oct 13, 2025
4 of 6 checks passed

dlambrig deleted the hotshard-doc branch October 13, 2025 22:45

kakaiu reviewed Oct 13, 2025

View reviewed changes


		### Storage Server Impact

		- The storage servers hosting the hot shard become saturated with write traffic, causing their write queues to grow ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


		- The storage servers hosting the hot shard become saturated with write traffic, causing their write queues to grow ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).

		- Storage server metrics report high `bytesInput` and increasing queue depths, which are monitored by Ratekeeper ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).


		- Storage server metrics report high `bytesInput` and increasing queue depths, which are monitored by Ratekeeper ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).

		- If the storage server's write queue size exceeds thresholds relative to available space and target ratios, it becomes the bottleneck for the entire cluster ([Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


		### Cluster-Wide Rate Limiting

		- Ratekeeper continuously monitors storage server queues and bandwidth metrics to determine cluster-wide transaction rate limits ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).


		- Ratekeeper continuously monitors storage server queues and bandwidth metrics to determine cluster-wide transaction rate limits ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933), [Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973)).

		- When a storage server's write queue becomes excessive, Ratekeeper sets the limit reason to `storage_server_write_queue_size` and reduces the cluster-wide transaction rate proportionally ([Ratekeeper:973](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L973), [Ratekeeper:1440](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L1440)).


		- This causes all transactions across the cluster to be throttled, even those not touching the hot shard, because the cluster cannot commit faster than its slowest storage server can durably persist data.

		- The rate limit is calculated as: `limitTps = min(actualTps * maxBytesPerSecond / inputRate, maxBytesPerSecond * MAX_TRANSACTIONS_PER_BYTE)`, ensuring the cluster doesn't overwhelm the saturated storage server ([Ratekeeper:933](https://github.com/apple/foundationdb/blob/7.3.0/fdbserver/Ratekeeper.actor.cpp#L933)).

Hot shard design documentation #12413

Hot shard design documentation #12413

Conversation

dlambrig commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 2, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

jzhou77 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 5, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 9, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 9, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 9, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

dlambrig commented Oct 2, 2025 •

edited

Loading