[gateway] HTTP 400 on tools/list for Atlassian MCP HTTP backend (Streamable HTTP transport)

[lpcox]

Problem

When an HTTP MCP backend is configured with custom headers (e.g., Authorization: Basic … for Atlassian), the gateway returns:

Failed to register tools from atlassian: backend error listing tools: code=-32603, message=HTTP 400: Bad Request

The gateway successfully establishes a connection (✓ atlassian: connected) but the subsequent tools/list RPC call fails with HTTP 400.

Source: github/gh-aw#22913

Analysis

Root cause: when custom headers are present, NewHTTPConnection skips the SDK-managed Streamable HTTP and SSE transports and falls back to a plain JSON-RPC-over-POST implementation (internal/mcp/connection.go, NewHTTPConnection() lines 183–260; internal/mcp/http_transport.go).

The plain JSON-RPC transport sends:

POST /v1/mcp HTTP/1.1
Content-Type: application/json
Accept: application/json, text/event-stream

{"jsonrpc":"2.0","id":1,"method":"tools/list","params":null}

This is incorrect for a Streamable HTTP (MCP 2025-03-26 spec) server for several reasons:

1. params: null vs params: {} — The MCP 2025-03-26 spec and many implementations require an empty object {} rather than null for parameterless methods. Atlassian's server likely rejects null as a malformed request.
2. Missing Accept: application/json, text/event-stream ordering — Streamable HTTP servers use this header to decide whether to stream or return a single JSON response. Some servers are strict about the value.
3. No initialize handshake before tools/list — The plain JSON-RPC path may send tools/list without first completing the initialize/initialized handshake required by the 2025-03-26 spec.
4. Transport bypass — The SDK's StreamableClientTransport handles protocol negotiation correctly, but it is bypassed when len(headers) > 0. The condition at NewHTTPConnection() that skips to plain JSON-RPC when custom headers are provided is overly conservative.

Relevant files:

• internal/mcp/connection.go — NewHTTPConnection() (lines 183–260): transport selection logic
• internal/mcp/http_transport.go — tryStreamableHTTPTransport() (lines 336–350), trySSETransport() (lines 352–365), tryPlainJSONTransport() (lines 367–393), setupHTTPRequest() (lines 232–249), createJSONRPCRequest() (lines 200–208)
• internal/server/tool_registry.go — registerToolsFromBackend() (lines 120–138): the caller that triggers tools/list

Proposed Solution

Option A (preferred) — pass custom headers through the SDK transports:

Modify tryStreamableHTTPTransport() and trySSETransport() in internal/mcp/http_transport.go to inject custom headers via a custom http.Client or request transformer, so the SDK transport is used even when headers are provided:

func tryStreamableHTTPTransport(ctx context.Context, serverID, url string, headers map[string]string) (*Connection, error) {
    httpClient := buildHTTPClientWithHeaders(headers) // new helper
    return trySDKTransport(ctx, serverID, url, headers,
        HTTPTransportStreamable, "streamable HTTP",
        func(url string, _ *http.Client) sdk.Transport {
            return &sdk.StreamableClientTransport{
                Endpoint:   url,
                HTTPClient: httpClient,
                MaxRetries: 0,
            }
        },
    )
}

Implement buildHTTPClientWithHeaders() as an http.Client with a custom RoundTripper that injects the configured headers into every outgoing request.

Option B (fallback fix) — fix the plain JSON-RPC path:

If the SDK transport cannot easily accept custom headers, at minimum fix the plain JSON-RPC path:

1. In createJSONRPCRequest() (http_transport.go lines 200–208): use map[string]interface{}{} (empty object) instead of nil for parameterless methods.
2. Ensure initialize + initialized handshake is sent before tools/list in tryPlainJSONTransport().

Option A is preferred because it preserves correct protocol negotiation (Streamable HTTP) while supporting authentication headers.

Testing

• Add or update integration test in test/integration/ that configures an HTTP backend with an Authorization header and verifies tools/list succeeds.
• Unit test in internal/mcp/ that confirms the Streamable HTTP transport is selected (not plain JSON-RPC) even when custom headers are present.
• Verify against Atlassian MCP Server ((mcp.atlassian.com/redacted) with a valid Authorization: Basic …` token.

Generated by Gateway Issue Dispatcher · ◷