refactor(meta): include upstream partial graph id in exchange request

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

part of the downstream PR #24239.

assume that an actor belongs to a fixed partial graph, and modify the local barrier worker state accordingly.

copilot gen

This pull request introduces support for tracking and propagating the partial_graph_id throughout the system, especially in actor info and fragment edge management. This change is important for distinguishing different partial graphs within a database, which can improve scalability and correctness in distributed streaming jobs. The update touches protocol buffer definitions, actor and fragment info structures, and various places where actors and fragments are constructed and referenced.

The most important changes are:

Protocol and Data Model Updates

• Added partial_graph_id to the ActorInfo message in proto/common.proto and to relevant request messages in proto/task_service.proto, allowing this information to be transmitted in RPCs. [1] [2]
• Updated the FragmentInfo struct and FragmentEdgeBuilder logic to include and propagate partial_graph_id for each fragment, ensuring that edges and actor info are correctly associated with their partial graph. [1] [2] [3] [4]

Actor and Fragment Construction

• Modified actor info construction in several places to include the correct partial_graph_id, especially when scaling or rescheduling database partial graphs, and updated relevant helper functions to support this. [1] [2] [3] [4]
• Changed the collection of actor upstreams to focus specifically on database partial graphs, renaming and updating the logic to use collect_database_partial_graph_actor_upstreams. [1] [2]

Fragment Edge Building and Relations

• Updated fragment edge building to handle cases where upstream or downstream fragments may not be present, improving robustness and correctness of edge construction.

Integration and Propagation

• Ensured that the partial_graph_id is correctly propagated through checkpoint control, command, and info modules, including during snapshot backfill and replace job scenarios. [1] [2] [3] [4] [5]

These changes collectively enhance the system's ability to manage and distinguish partial graphs, which is critical for supporting scalable and reliable distributed streaming jobs.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

• My PR needs documentation updates.

Release note

[wenym1]

• feat(snapshot-backfill): support alter rate limit for snapshot backfill #24302 : 2 dependent PRs (#24116 , #24359 )
• feat(meta): support drop/cancel creating snapshot backfill job #24286
• refactor: only handle partial graphs in local barrier worker #24239
• refactor(meta): include upstream partial graph id in exchange request #24342 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.