Dataplane integration fixes (combined view) - #5514
Closed
lucarlig wants to merge 32 commits into
Closed
Conversation
Signed-off-by: lucarlig <luca.carlig@ibm.com>
The dataplane publisher interval was a hardcoded 60s module constant, so runtime-created users and servers stayed invisible to the dataplane for up to a full minute. Expose it as DATAPLANE_PUBLISHER_INTERVAL_SECONDS (default 60, unchanged) and derive the UserConfig key TTL as interval + 10, preserving the current 60/70 relationship. Functional test stacks can now run a short interval while load benchmarks keep the default. Signed-off-by: lucarlig <luca.carlig@ibm.com>
A 5 x 100ms retry window cannot cover the dataplane publisher snapshot cycle (default 60s), so the allow-path tests still failed on the split stack. Switch the helper to a deadline-based wait that covers one full publish interval plus slack by default (75s), tunable via MCP_E2E_PUBLISHER_SYNC_DEADLINE for stacks that run a short DATAPLANE_PUBLISHER_INTERVAL_SECONDS. Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com> # Conflicts: # .secrets.baseline
The deadline only bounds how long a genuine failure takes; passing runs return as soon as the publisher converges, so a fixed 75s covering one default publish interval plus slack works on every stack and the knob adds nothing. Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com> # Conflicts: # .secrets.baseline
DataplanePublisherService._build_user_data was building tool_name_by_id using tool.name, which is the slugged hybrid property (e.g. 'compliance-reference-progress-reporter'). Upstream MCP servers advertise the bare original name (e.g. 'progress_reporter'), so the dataplane filtered every runtime-registered vhost's tools to zero. Changing to tool.original_name publishes the exact name the upstream advertises, which is what the dataplane needs for its prefix-strip routing in tools/list and tools/call. This makes contextforge-gateway-rs PR #55 unnecessary. Signed-off-by: Luca <lucarlig@protonmail.com>
The allowed_tool_names change reads tool.original_name, but the bulk minimal-column query did not select it, so every publish cycle failed with 'Could not build dataplane payload data from the database: original_name' and no UserConfig ever reached Redis. Signed-off-by: lucarlig <luca.carlig@ibm.com>
Give the mock tools distinct name/original_name values so the payload assertions verify the publisher exports the upstream-facing original_name rather than the gateway-prefixed name. Signed-off-by: lucarlig <luca.carlig@ibm.com>
The compliance virtual_server fixture connected to /servers/{id}/mcp
immediately after creation. On split deployments that route is served
from asynchronously published dataplane config, so early gateway_virtual
rows failed with 404/400 before the first snapshot landed. Probe the
route until it accepts (75s deadline, one default publish interval plus
slack); control-plane-only deployments return on the first probe.
Signed-off-by: lucarlig <luca.carlig@ibm.com>
WORKER_ID was evaluated at module import in the gunicorn master before fork, so every worker published with the same id and the lock's compare-and-swap release could not distinguish owners; compute it per service instance instead. Raise the UserConfig key TTL from interval + 10 to 2 x interval + 10 so a single late publish cycle can no longer expire every key at once and drop dataplane config for all subjects until the next snapshot. Signed-off-by: lucarlig <luca.carlig@ibm.com>
The dataplane proxies streamable-HTTP upstreams only; publishing other transports (e.g. SSE, deprecated and removed in the 2026-07-28 MCP protocol update) advertised backends it cannot serve, which surfaced as successful tools/list responses with an empty list. Exclude non-streamable gateways from the payload and omit virtual hosts left with no publishable backends, so the dataplane 404s them and split deployments can route those requests back to the control plane. Signed-off-by: lucarlig <luca.carlig@ibm.com>
pytest-playwright's runtest hook collides with pytest-asyncio
('Runner.run() cannot be called from a running event loop'), so running
the whole live_gateway tree under -p playwright failed every async test
regardless of gateway behavior. Run the asyncio suites with
-p no:playwright and keep a second pass for the two suites that need
the playwright plugin (sso and the RBAC transport tests).
Signed-off-by: lucarlig <luca.carlig@ibm.com>
The per-server allow-path checks only asserted that tools/list succeeded, so a deployment that silently drops the server's backend passed them with an empty list. Assert the visibility servers actually advertise their associated tools. Signed-off-by: lucarlig <luca.carlig@ibm.com>
…er/luca/dataplane-integration-fixes # Conflicts: # .secrets.baseline Signed-off-by: lucarlig <luca.carlig@ibm.com>
…aplane-integration-fixes # Conflicts: # .secrets.baseline Signed-off-by: lucarlig <luca.carlig@ibm.com>
…ca/dataplane-integration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
…taplane-integration-fixes # Conflicts: # mcpgateway/services/dataplane_publisher.py Signed-off-by: lucarlig <luca.carlig@ibm.com>
…aplane-integration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
The live visibility test still asserted the pre-#4694 semantics where the admin-bypass path collapsed to anonymous scoping and hid even the caller's own private servers. Owner matching (issue #4694, commit 8c186c5) deliberately keeps the requester email on that path, so the listing includes the caller's own private rows; the fixture's private server is created by the same admin and correctly appears. Update the assertion and docstring to the current contract. Signed-off-by: lucarlig <luca.carlig@ibm.com>
The plugin suites are designed for a gateway booted out-of-band with a plugin enforce config. Against a general-purpose stack every test errored at fixture setup (gateway self-registration 502) instead of skipping. Probe /admin/plugins once per session and skip the suites unless the subsystem is enabled and at least one plugin is enabled. Signed-off-by: lucarlig <luca.carlig@ibm.com>
…a/dataplane-integration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
…ataplane-integration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
The dataplane uses the backends map key as the namespace prefix for the federated tool names it advertises, so keying by gateway id produced names like '<gateway-uuid>-echo' where the control plane advertises 'fast-time-echo'. Name-based clients and the protocol-compliance checks (exact slug-prefix matching) then failed whenever the dataplane served the virtual server. Key the map by the gateway slug — the same value the control plane composes tool names from — with the id as fallback, and keep the tool-association lookup on the id. Signed-off-by: lucarlig <luca.carlig@ibm.com>
…ntegration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
…ataplane-integration-fixes Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
lucarlig
force-pushed
the
user/luca/dataplane-integration-fixes
branch
from
July 9, 2026 15:27
9597c5b to
b83963f
Compare
(cherry picked from commit 398a44d) Signed-off-by: lucarlig <luca.carlig@ibm.com>
Collaborator
Author
|
all but 1 pr have been merged this is not necessary anymore |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft combined branch for validating the dataplane/control-plane integration changes together before or while the smaller component PRs land.
Use this PR for CI and
cf-integrationtesting across the stacked changes. Prefer the linked component PRs as the review/merge units unless reviewers decide the combined branch is the right landing path.Rebased onto latest
origin/main(cd10eaf0b) on 2026-07-09.Active component PRs
Note: this is a combined testing branch, not guaranteed to be byte-for-byte at every component PR head. The current #5514 head carries the functional changes used for the latest
cf-integrationrun; #5482 and #5517 have later component-branch housekeeping/test commits outside this exact combined head.Branch-history PRs
These are already merged, closed, or reverted, but are useful for understanding the combined branch history:
Latest local validation
uv run pytest tests/unit/mcpgateway/services/test_dataplane_publisher.py -q->22 passeduv run pytest tests/unit/mcpgateway/services/test_resource_service.py -k 'template_proxy or scoped_lookup or include_inactive_true_skips_enabled_filter' -q->4 passedPrevious full integration validation
./scripts/cf-integration.sh test-all-up-> passed all lanesprobe: passsmoke: passlive-mcp:19 passed, 3 skippedlive-rbac:40 passedlive-protocol:28 passed, 2 skipped, 76 deselected, 31 xfailed, 5 xpassedlive-all:112 passed, 57 skipped, 37 xfailed, 4 xpassedand40 passed, 20 skippedThe previously failing templated resource proxy case now passes:
tests/live_gateway/protocol_compliance/test_resources.py::test_templated_resource_registered_and_resolves[gateway_proxy-http]