Skip to content

feat(kv-offload): use external PegaFlow server#729

Open
xiaguan wants to merge 5 commits into
mainfrom
feat/external-pegaflow-server
Open

feat(kv-offload): use external PegaFlow server#729
xiaguan wants to merge 5 commits into
mainfrom
feat/external-pegaflow-server

Conversation

@xiaguan

@xiaguan xiaguan commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #724

Depends on novitalabs/pegaflow#407.

Summary

  • replace the embedded PegaFlow engine with an external CUDA IPC/RPC client
  • keep host, SSD, RDMA, and routing configuration in the PegaFlow server; OpenInfer exposes only the endpoint and content namespace
  • support Qwen3 and GLM5.2 arena registration with opt-in exportable CUDA allocations
  • carry block stride inside each CUDA IPC tensor and fail closed on mixed registration-wire revisions
  • use terminal native Load completion and flush submitted saves before unregistering the client instance

Validation on final PR heads

  • prek run
  • cargo test --release -p openinfer-kv-offload --lib (9 passed)
  • PegaFlow server tests (23 passed) and core tests (124 passed, 1 ignored)
  • PegaFlow server all-targets and Python client CUDA 13 checks

Cross-process gate status

The Qwen3 byte-exact restore and GLM5.2 EP8 host-restore gates passed before the PegaFlow PR reached its final scope. Those results remain design evidence, but the gates must be replayed on the final heads before deployment.

@xiaguan
xiaguan force-pushed the feat/external-pegaflow-server branch from 4d4f98b to 5a2c6ba Compare July 19, 2026 19:47

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d4f98b04e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread openinfer-kv-offload/src/external.rs Outdated
Comment on lines +313 to +315
.release(deadline_request(ReleaseRequest { lease }))
.await
.map_err(|err| rpc_error("release", err))?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check release RPC status before dropping leases

When a lease release reaches the server but is rejected at the application layer, this call currently drops the RPC response and returns Ok(()); unlike health/save/load/flush, it never validates the returned status. In the reservation-pressure and abandoned-prefetch paths this makes failed releases look successful, so host blocks can remain pinned until the server TTL instead of being surfaced as a release failure.

Useful? React with 👍 / 👎.

Signed-off-by: xiaguan <751080330@qq.com>
@xiaguan
xiaguan force-pushed the feat/external-pegaflow-server branch from 5a2c6ba to 38a6201 Compare July 19, 2026 19:59
xiaguan added 4 commits July 20, 2026 03:54
Signed-off-by: xiaguan <751080330@qq.com>
Signed-off-by: xiaguan <751080330@qq.com>
Signed-off-by: xiaguan <751080330@qq.com>
Signed-off-by: xiaguan <751080330@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support out-of-process PegaFlow

1 participant