Skip to content

feat: allow overriding actual shard values in auto-sharding - #3963

Draft
igor-sirotin wants to merge 1 commit into
masterfrom
feat/autosharding-shard-override
Draft

feat: allow overriding actual shard values in auto-sharding#3963
igor-sirotin wants to merge 1 commit into
masterfrom
feat/autosharding-shard-override

Conversation

@igor-sirotin

Copy link
Copy Markdown
Contributor

Summary

Auto-sharding currently produces shard ids in the range [0..NUM_SHARDS_IN_NETWORK-1], derived from the content-topic hash. To make this work in Status — which uses a specific set of shard values — we need to be able to force the actual shard values while keeping the auto-sharding distribution.

This PR introduces an optional shard override array of length NUM_SHARDS_IN_NETWORK:

  • Auto-sharding still hashes the content topic to an index in [0..NUM_SHARDS_IN_NETWORK-1].
  • When an override is configured, the shard id placed in the pubsub topic becomes shardOverride[index] instead of the index itself.
  • When unset, behaviour is unchanged (shard id == index).

How to use

CLI:

--num-shards-in-network=8 --shard-override=32 --shard-override=64 ... (8 values)

Provide exactly NUM_SHARDS_IN_NETWORK values (repeatable arg). The node also subscribes to the override values by default (instead of [0..n-1]), so it relays/receives on the forced shards.

The same is exposed via the (deprecated) AutoShardingConfig JSON config as shardOverride.

Changes

  • waku_core/topics/sharding.nimSharding.shardOverride; getGenZeroShard remaps the computed index through the override.
  • factory/networks_config.nimShardingConf.shardOverride; new shards() helper (effective shard set); validateShards validates against the actual shard values and checks override length.
  • factory/conf_builder/waku_conf_builder.nimwithShardOverride + buildShardingConf threading; default subscribe shards use the actual shards.
  • node/waku_node.nimmountAutoSharding accepts the override (validates length).
  • factory/node_factory.nim — passes the override through.
  • tools/confutils/cli_args.nim — new --shard-override arg.
  • api/api_conf.nimAutoShardingConfig.shardOverride (+ JSON read/write/wiring).
  • tests/waku_core/topics/test_sharding.nim — unit tests for the remapping and the length-mismatch fallback.

Notes / open questions

  • Not compiled locally: the Nim toolchain + RLN (Rust) + vendored submodules were not available in this environment, so I relied on static review and added unit tests. Please let CI validate the build (and nph formatting).
  • Semantics decision worth a look: with an override set, the default subscribe shards and shard validation both switch to the override values (so a node lands on the forced shards). Flag if you'd prefer --shard to keep meaning indices.

🤖 Generated with Claude Code

Auto-sharding currently produces shard ids in the range
[0..NUM_SHARDS_IN_NETWORK-1], derived from the content topic hash. To
interoperate with Status, which uses a specific set of shard values, we
need to be able to force the actual shard values while keeping the
auto-sharding distribution.

This adds an optional shard override array (length NUM_SHARDS_IN_NETWORK):
auto-sharding still hashes the content topic to an index, but the shard id
placed in the pubsub topic becomes shardOverride[index] instead of the
index itself.

Wired through:
- Sharding.getGenZeroShard remaps the computed index via shardOverride
- ShardingConf gains a shardOverride field; default subscribe shards and
  shard validation use the actual (override) shard values
- WakuConfBuilder.withShardOverride + buildShardingConf threading
- CLI arg --shard-override (repeatable)
- mountAutoSharding accepts the override (validates length)
- api_conf AutoShardingConfig gains shardOverride (+ JSON)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:3963

Built from 3fa6045

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant