diff --git a/.github/workflows/sdk-types-ci.yml b/.github/workflows/sdk-types-ci.yml index e08e0c04..4cfa67c1 100644 --- a/.github/workflows/sdk-types-ci.yml +++ b/.github/workflows/sdk-types-ci.yml @@ -22,8 +22,15 @@ on: # no git-subdirectory support, so the package installs whole-repo). Without # it here, editing the shipped export map triggers no job at all. - 'package.json' - - 'sdk/go/helpers/**' - - 'sdk/go/resolvers/**' + # Every Go SDK package, because this workflow owns the API-surface gate: + # it walks the LIVE Go surface with `go doc` and fails when an export is + # neither mapped nor excluded. Listing only two packages meant a change + # that added exports to any of the others — and edited the symbol map to + # match — triggered no job at all, so nothing checked the edit. + - 'sdk/go/**' + - 'sdk/parity/**' + - 'docs/sdk-parity-matrix.md' + - 'scripts/gen-parity-matrix.py' - '.github/workflows/sdk-types-ci.yml' pull_request: paths: *paths diff --git a/docs/design-history.md b/docs/design-history.md index d3ab9e9a..95b3c448 100644 --- a/docs/design-history.md +++ b/docs/design-history.md @@ -626,3 +626,211 @@ reason enum a Go exchange emitted, never on a human string. Emit and decode are to one shared oracle corpus (`error-detail-vectors.json`) replayed by all three languages, so the typed-failure contract is verified end-to-end across the language boundary rather than trusted to match by inspection. + +## Discovery answers are grouped per URI, not flattened + +A discovery call is per-URI: an agent asks about several resources at once, and both +`DiscoverResources` and the Broker's `Resolve` answer with one `OfferGroup` per +requested URI, each either carrying offers or carrying a typed `OfferAbsenceReason` +explaining why it carries none. The SDK returns that shape rather than a flat list, +with the fail-closed `{verified, rejected}` split preserved **inside** each group and +produced by the one `Verifier` — never a second verification path. + +Flattening loses two things, and the second is unrecoverable. It drops the +attribution — which offer answers which URI — and it erases a REFUSED URI entirely, +because an empty group has no offer to carry its identity back. That matters because +the absence vocabulary is a set of different actions: `NOT_IN_CATALOG` means give up, +`SCOPE_INSUFFICIENT` means acquire an entitlement and retry, `CONTENT_BLOCKED` means +never retry. Flattened, all three read as "found nothing", which is the trial-and-error +the field exists to prevent. + +Two consequences worth stating. The absence reasons are carried as POINTERS, because a +responder may legitimately withhold a reason where the existence of a resource must +itself stay hidden — so "absent" and "unspecified" have to stay distinguishable, and a +generated getter collapses them. And `ResourceResponse` carries a grouped list *and* a +flat one, with the flat one a single-URI convenience mirror; the two are read as +alternatives, never concatenated, because a responder that populates both would +otherwise have every offer counted twice. + +## A signed leg refuses redirects; the guarded fetch follows them + +The SSRF-guarded HTTP client follows up to five redirects, re-pinning the address and +re-vetting the scheme at each hop. That is correct for what it was built for: fetching +public well-known documents and key directories, where there is nothing to leak. + +It is wrong for any request carrying a credential. A usage report, a dispute and a +delivery fetch all take only the guarded `.Transport` and install their own refusal. +For the RPC legs, following a redirect would re-sign the caller's request for a target +the peer chose — after the endpoint check had already passed, which is precisely the +window that check exists to close. For the delivery fetch it is worse: the proof of +possession covers `@target-uri`, so replaying it at a new location fails the edge's own +check, and re-signing per hop would hand a fresh proof of possession of the agent's key +to whatever host the first hop named. Redirect support on those legs would need per-hop +re-signing plus host anchoring, and is deliberately not attempted. + +The transport error is rebuilt before it surfaces, because the HTTP client wraps every +failure in a value carrying the full URL it was dialing — query included. On a refused +redirect that is a credential belonging to a URL the *first hop* chose, so the wrapper +leaks even when the SDK's own message is already redacted. + +## A usage report's destination comes off the message, not from configuration + +A usage report must reach the Exchange that ISSUED the offer, and that Exchange's +address is read from its own `/.well-known/ramp.json` — never from configuration. A +signature covers the `exchange` DOMAIN; it says nothing about where that domain's +endpoint lives or where its DNS points. + +The SDK takes the domain off `UsageReport.exchange` rather than as an argument, and +offers no option to supply an endpoint. Leaving no configuration slot for it is what +makes the rule structural instead of a convention someone can quietly reverse: there is +no parameter a configured origin could be passed as. (`Dispute` is the exception, and +only because `DisputeRequest` carries no `exchange` field to read — it takes the domain +as an argument and runs the identical checks.) + +Five checks precede the send, in order: refuse anything that is not a plain hostname, +because the value is concatenated into a URL and a smuggled path would choose what gets +fetched; resolve the endpoint from that host's own manifest, cached per host; require +the endpoint to be that host or a subdomain of it, since the manifest is only as +trustworthy as the host serving it and a dial-time address guard has no objection to an +unrelated PUBLIC host; dial through the SSRF guard, applied to the report itself and +not only to the manifest fetch; and refuse redirects. The per-origin client pool that +follows is bounded and evicts least-recently-used, because which Exchanges appear is +driven by incoming offers — an open-ended, caller-influenced key space. + +## Host anchoring compares host and port, but not scheme + +A value a remote document supplies — an Exchange's advertised endpoint, a WBA +directory's revocation URL — is checked against the host that served that document +before anything signed is sent to it: it may name itself or one of its own +subdomains, and nothing else. The match is on a full dot-delimited label boundary, +so `evil-a.com` is not a subdomain of `a.com`; a bare suffix comparison gets that +wrong, and it is the mistake an attacker registers a domain to exploit. + +The rule is now normative and stated where an implementer will find it: the +`WellKnownManifest.endpoint` proto comment, and the manifest page on the docs site. +It was not always — for most of this work it lived only in one client's code, +inherited from a port rather than decided, which is why an Exchange advertising a +separate domain could be conformant and then stop being so. Enforcement sits in +the shared endpoint resolver, so every consumer of that resolver inherits it. + +The anchor is the host that SERVED the manifest, not the `domain` member inside +it. That member is self-asserted, so anchoring to it would let a hostile manifest +name whatever endpoint it liked and validate itself. For a conformant Exchange the +two agree, which is exactly why the distinction only shows against a document +worth refusing. + +One SDK enforces it. Python and TypeScript ship endpoint resolvers that do not, +and the predicate they would need does not exist there — only private +near-namesakes in their WBA modules, and neither is a counterpart. TypeScript's +compares `URL.host`, which folds a default port away as this rule now requires, +but it is not exported; Python's compares `netloc`, which keeps an explicit `:443` +verbatim and carries any userinfo along with it. Closing that is tracked +separately. + +The comparison includes the PORT. An earlier version of this rule ignored it, on +the reasoning that TLS binds hostnames rather than ports, so a service on another +port of the same name is not another host. That was overruled deliberately: what +is being anchored is a place a signed call is sent, another port is another +service that the party publishing the anchor need not control, and having one rule +compare the port while another ignored it was the worse outcome of the two — the +predicate existed twice in Go and the copies had already drifted apart on exactly +this question. An Exchange reachable on a non-default port now names that port on +both sides. + +One detail carries the whole decision: a DEFAULT port and an omitted port are the +same port. `url.Parse` does not materialize an implicit port, so a comparison of +raw authorities would refuse an operator who merely wrote `:443` out in full — a +spelling check wearing a security check's clothes. The folding is scheme-relative, +which is also what keeps it from becoming something it is not: the scheme is still +not compared here — that is the guarded transport's decision, in one place, driven +by one flag — and `http://x` and `https://x` both reduce to no port rather than +diverging on 80 versus 443. + +Scheme-relative folding needs a scheme on both sides, and both ANCHORS in this SDK +arrive without one: a WBA directory's authority and an `Offer.exchange` host are +bare `host[:port]` values. Reading those as https — the assumption that lets a bare +domain be told apart from a path — quietly broke the rule for plaintext +deployments: an anchor of `a.example:80` kept its port, because 80 is not https's +default, while the candidate `http://a.example:80` folded the same port away, so +one authority reached two answers and every plaintext WBA directory that spelled +`:80` in full stopped anchoring its own revocation URL. A skipped revocation poll +leaves a revoked key resolving, which is a great deal worse than the spelling it +was refusing. A side that named no scheme therefore borrows the other's. That +decides only WHICH port is the default, never whether two different ports are +equal: an anchor of `x:443` still refuses `http://x:80`, since 443 is not http's +default. + +## What the manifest fetch does not guarantee, and why that is bounded + +The address a usage report goes to is read from the issuing Exchange's own +`/.well-known/ramp.json`. That fetch runs on the guarded client, which FOLLOWS up +to five redirects — re-pinning the address and re-vetting the scheme at each hop, +but not anchoring the host. So the party that answers for the manifest can be one +a redirect chose, and the answer is cached per host for the TTL. + +That is deliberate and it is bounded, but the bound comes from somewhere else: +whatever the manifest says, the endpoint it advertises must still anchor to the +ORIGINAL offer domain before a signed call goes there. A redirect can therefore +change who answers the question, never where the report lands. What it can weaken +is the assumption that a manifest served over TLS from the provider's own domain +is thereby endorsed by it. + +Refusing redirects on that one fetch was considered and not taken here. The +five-hop posture is stated as identical across all three languages, so a Go-only +refusal would create a divergence in the transport policy rather than remove a +risk — and the risk it removes is already contained by the anchoring above. Worth +revisiting as a three-language change. + +## One agent identity, one key + +The protocol carries a single agent identity and the SDK does not offer a second. +`agent_identity_hash` is defined as the RFC 7638 thumbprint of the agent's +request-signing key; an Exchange verifies the detached offer acceptance against the +key registered for whichever caller the request signature identified; and the +delivery URL is bound to that same thumbprint, which a later fetch must prove +possession of. A separately-custodied acceptance key would be refused at execute, +and any URL it did produce could never be fetched — the presented key would not +match the binding. So the client takes one Signer, and the public half of that +same key for the fetch header, which a Signer cannot yield. + +One consequence for the cross-language surface: Go's `SignAgentBinding` takes a +`Signer` plus the public half, while Python's counterpart takes raw seed bytes. +The parity map records them as counterparts because the face exists in both, but +the custody posture differs — the Go seam exists precisely so the SDK never holds +key material, and closing that gap belongs with the TypeScript/Python client work. + +## Bounds on a leg that dials wherever an offer points + +`ReportUsage` and `Dispute` reach an Exchange named inside an offer, so the origin +is discovered at runtime and chosen by another party. Everything about that leg is +therefore bounded rather than open-ended: the response size (Connect treats an +unset cap as "any size" while compressing every exchange, so an unbounded read is +an unbounded decompression into the caller's memory), the call deadline (an +Exchange that accepts a connection and never answers would otherwise hold a call, +a goroutine and a socket indefinitely), the per-origin client pool, and the +endpoint cache beneath it. The key space for both caches is the same open-ended, +caller-influenced set of hosts, so both evict least-recently-used at a fixed cap. + +The base-transport option cannot remove the guard on that leg. A caller can supply +a transport — its own connection tuning, its own client certificates — and it is +composed UNDERNEATH the address and scheme guards rather than in place of them. +The only opt-out through that seam is the deployment-level SKIP_SSRF / +ALLOW_INSECURE pair, which is one decision recorded in one place instead of a +per-caller copy of it. An option that could silently disarm the guard is exactly +how a security property becomes advisory. + +Two settings on a supplied transport are dropped rather than carried, because +each would route the dial around the address check rather than under it: a proxy, +which would have the dialer resolve and vet the PROXY instead of the destination, +and a custom TLS dialer, which `net/http` prefers over the pinned dialer whenever +the scheme is https — which is every RAMP leg. The second is the more dangerous +of the two because it fails silently and on the ordinary path: a transport that +carries one dials wherever it likes and no error says the pin never ran. TLS +itself stays configurable through `TLSClientConfig`, which is kept, so the +customisation the seam exists for survives and only the dialer is refused. + +The claim is scoped to that seam deliberately. A caller that injects a whole +`*http.Client` into a resolver, or supplies its own endpoint resolver, has taken +ownership of that fetch and the guard is that caller's to install — which is a +different bargain from an option that quietly weakens a fetch the SDK still +performs. diff --git a/docs/sdk-parity-matrix.md b/docs/sdk-parity-matrix.md index 632cfb6a..a334301e 100644 --- a/docs/sdk-parity-matrix.md +++ b/docs/sdk-parity-matrix.md @@ -12,7 +12,7 @@ Go is the oracle (`sdk/go/{helpers,resolvers,core,connect,connectserver}`); Python and TS mirror it. This document is **generated** from the same two artifacts CI already enforces against the code, so it cannot drift from the real surface — a mismatch fails the API-surface gate or the corpus-completeness gate before it can reach this file. -**At a glance:** 75 symbols at cross-language parity · 14 documented divergences · 99 Go-idiomatic exclusions · 23 conformance corpora, each tri-replayed. +**At a glance:** 77 symbols at cross-language parity · 14 documented divergences · 142 Go-idiomatic exclusions · 23 conformance corpora, each tri-replayed. Layering (L1 pure trust core vs L2 I/O resolvers), the SSRF transport-wiring invariant, and naming conventions are recorded in [`design-history.md`](./design-history.md). @@ -25,6 +25,7 @@ Legend: a name = the public face in that language · `—` = intentionally none | Go | python | ts | |---|---|---| | `AcceptanceSignatureAlgorithm` | `ACCEPTANCE_SIGNATURE_ALGORITHM` | `ACCEPTANCE_SIGNATURE_ALGORITHM` | +| `AgentKeyHeader` | `AGENT_KEY_HEADER` | `AGENT_KEY_HEADER` | | `AppendSignature` | `append_signature` | `appendSignature` | | `ApplyScopes` | `apply_scopes` | `applyScopes` | | `CanonicalAcceptanceBytes` | `jcs_acceptance_payload` | `acceptancePayload` | @@ -52,6 +53,7 @@ Legend: a name = the public face in that language · `—` = intentionally none | `RequestIDHeader` | `RequestIDHeader` | `RequestIDHeader` | | `RetrievalAuthFailureDetail` | `retrieval_auth_failure_detail` | `retrievalAuthFailureDetail` | | `ScopesSubset` | `scopes_subset` | `scopesSubset` | +| `SignAgentBinding` | `sign_agent_binding` | `signInbound` | | `SignOffer` | `sign_offer_jcs` | `signOffer` | | `SignOfferAcceptance` | `sign_offer_acceptance_jcs` | `signOfferAcceptance` | | `SignRequest` | `sign_request` | `signRequest` | @@ -127,14 +129,14 @@ Deliberate, reason-backed asymmetries. The allowlist is **shrink-only** — a ne ### Architectural DECISIONs - **DECISION — full Connect handler binding.** Go-only full Connect Broker handler binding — OPEN DECISION in docs/sdk-parity-matrix.md _(symbols: `connectserver.NewBrokerServiceHandler`, `connectserver.NewExchangeServiceHandler`)_ -- **DECISION — typed Connect **client**.** Go-only typed Connect client (Discover->Execute orchestration) — deliberate runtime-native divergence, DECISION resolved in docs/sdk-parity-matrix.md _(symbols: `connect.Client`, `connect.NewClient`)_ +- **DECISION — typed Connect **client** — OPEN.** Go-only typed Connect client covering the agent verb set (Discover, Resolve, Execute, ReportUsage, Dispute, Fetch) — OPEN DECISION in docs/sdk-parity-matrix.md: the API-surface design governs and specifies a thin Connect-unary JSON client for TypeScript and Python with the SAME verb names, so this is an implementation difference pending that work, not a settled API divergence _(symbols: `connect.Client`, `connect.NewClient`)_ ### Mapped symbols with an intentional per-language gap | Go symbol | python | ts | rationale | |---|---|---|---| -| `connect.Client` | — | — | Go-only typed Connect client (Discover->Execute orchestration) — deliberate runtime-native divergence, DECISION resolved in docs/sdk-parity-matrix.md | -| `connect.NewClient` | — | — | Go-only typed Connect client constructor — deliberate runtime-native divergence, DECISION resolved in docs/sdk-parity-matrix.md | +| `connect.Client` | — | — | Go-only typed Connect client covering the agent verb set (Discover, Resolve, Execute, ReportUsage, Dispute, Fetch) — OPEN DECISION in docs/sdk-parity-matrix.md: the API-surface design governs and specifies a thin Connect-unary JSON client for TypeScript and Python with the SAME verb names, so this is an implementation difference pending that work, not a settled API divergence | +| `connect.NewClient` | — | — | Go-only typed Connect client constructor (NewClient plus NewBrokerClient) — OPEN DECISION in docs/sdk-parity-matrix.md, pending the TypeScript and Python unary client that carries the same verb names | | `connectserver.NewBrokerServiceHandler` | — | — | Go-only full Connect Broker handler binding — OPEN DECISION in docs/sdk-parity-matrix.md | | `connectserver.NewExchangeServiceHandler` | — | — | Go-only full Connect Exchange handler binding — OPEN DECISION in docs/sdk-parity-matrix.md | | `core.NewVerifier` | — | `createVerifier` | Go NewVerifier factory folds into the Python class constructor (Verifier(...)); idiomatic Python exposes the class, not a separate factory symbol. TS keeps the createVerifier factory. | @@ -154,16 +156,34 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | Go symbol | why no py/ts face | |---|---| -| `connect.ClientOption` | Part of the Go-only typed Connect client; see the Connect-client DECISION in docs/sdk-parity-matrix.md. | -| `connect.ExecuteOption` | Part of the Go-only typed Connect client; see the Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.BrokerClient` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.CallError` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.CallErrorKind` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.CallOption` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.ClientOption` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.DefaultCallTimeout` | Go default deadline for a call to an offer-derived Exchange. Python and TS ship no Connect client at all, so neither holds a counterpart today; the default arrives with their unary client work. | +| `connect.DefaultMaxRPCReadBytes` | Go default response-size bound for a Connect call. Python and TS ship no unary client to hold one; the bound arrives with their unary client work. | +| `connect.EndpointResolver` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.ExecuteOption` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.NewBrokerClient` | Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | | `connect.NewValidateInterceptor` | Go-only protovalidate interceptor (matrix SERVER-role validation row: TS/Py absent). | -| `connect.Validation` | Part of the Go-only typed Connect client validation option; see the Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.Validation` | Part of the Go-only typed Connect client validation option; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md. | +| `connect.WithAgentKey` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithClientOptions` | Go escape hatch for raw connectrpc.ClientOption values, mirroring connectserver.WithHandlerOptions; py/ts have no Connect option type to pass through. | +| `connect.WithContentTimeout` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithEndpointResolver` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithGuardedBaseTransport` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithHTTPClient` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithIdempotencyKey` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithInterceptors` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithKeyResolver` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithMaxContentBytes` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithOfferKey` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithProofWindow` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithRequestIDFunc` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithRequester` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithSignWindow` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `connect.WithSignatureAgent` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithSigner` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithValidation` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connect.WithVerification` | Go functional-option builder; py/ts pass options via kwargs/options objects. | @@ -189,7 +209,9 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `connectserver.WithVerifyGate` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `connectserver.WithoutReplayStore` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `core.DefaultRequestID` | Go default request-id minter; py/ts mint request-ids inline. | +| `core.DiscoveryResult` | Go per-URI discovery result carrying the fail-closed split plus the typed absence reasons; py/ts gain the same shape with their client verbs. | | `core.ErrOfferExpired` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | +| `core.OfferGroupResult` | Go per-URI group within a discovery result; py/ts gain the same shape with their client verbs. | | `core.RequestIDFunc` | Go request-id function type; py/ts pass a callable inline. | | `core.RequestIDMiddleware` | Go-only request-id middleware (matrix SERVER-role request-id row: TS/Py absent). | | `core.SigningOption` | Go functional-option type for the signing transport; py/ts pass options objects. | @@ -197,6 +219,7 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `core.WithSignPredicate` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `core.WithSignatureAgent` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `core.WithWindow` | Go functional-option builder; py/ts pass options via kwargs/options objects. | +| `helpers.AgentBinding` | Go value struct holding the three proof header values; Python returns a tuple of them and TS returns a prepared request, so neither names a public type. | | `helpers.AgentIDParam` | Signed-URL query-parameter name; language-idiomatic inline constant, no cross-language public face. | | `helpers.AlgEd25519` | RFC 9421 alg tag constant; inlined per language. | | `helpers.AllSignaturesFromContext` | Go context.Context accessor; py/ts thread multisig state explicitly. | @@ -210,7 +233,10 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `helpers.ErrEmptyMoney` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrExpired` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrFutureCreated` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | +| `helpers.ErrInvalidHost` | Go errors.Is sentinel for an unusable host reference; py/ts raise/throw instead of exporting sentinels. | | `helpers.ErrInvalidKeyLength` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | +| `helpers.ErrInvalidPoPInput` | Go errors.Is sentinel for a proof input that cannot be written into a signature base; py/ts raise/throw instead of exporting sentinels. | +| `helpers.ErrKeyIDMismatch` | Go errors.Is sentinel for a keyid that is not the presented key's thumbprint; py/ts raise/throw instead of exporting sentinels. | | `helpers.ErrMalformedSignatureInput` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrMissingContentDigest` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrMissingCreated` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | @@ -218,6 +244,7 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `helpers.ErrMissingRequiredComponent` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrMissingSignature` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrMissingSignatureInput` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | +| `helpers.ErrMissingTargetURI` | Go errors.Is sentinel for a proof requested without the URL it binds; py/ts raise/throw instead of exporting sentinels. | | `helpers.ErrOfferExpired` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrOfferSignatureInvalid` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrProofOfPossessionMismatch` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | @@ -232,11 +259,18 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `helpers.ErrUnknownFields` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.ErrUnsupportedAlgorithm` | Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels. | | `helpers.FromContext` | Go context.Context accessor; py/ts thread verified-request state explicitly. | +| `helpers.HostAnchored` | Go label-boundary same-host-and-port predicate, folding a scheme's default port into an omitted one. Python and TS carry a PRIVATE near-namesake in their WBA modules, neither of which is a counterpart: TS's compares URL.host, which normalizes the default port but is not exported; Python's compares netloc, which keeps an explicit :443 AND includes userinfo. Exporting an aligned predicate in all three is tracked separately. | +| `helpers.HostOf` | Go host-extraction helper behind the two routing predicates. No Python or TS equivalent exists today, private or otherwise; it is a prerequisite of the tracked cross-language endpoint-rule work. | +| `helpers.IsBareHost` | Go plain-hostname predicate for the report leg's first check. No Python or TS equivalent exists today; exporting the pair in both is part of the tracked cross-language endpoint-rule work. | | `helpers.NewContext` | Go context.Context accessor; py/ts thread verified-request state explicitly. | | `helpers.NewEd25519Signer` | Go Ed25519 Signer constructor; py/ts inject a sign function rather than constructing a named signer. | | `helpers.NewEd25519SignerFromSeed` | Go Ed25519 Signer-from-seed constructor; py/ts inject a sign function rather than constructing a named signer. | | `helpers.NewMultisigContext` | Go context.Context accessor; py/ts thread multisig state explicitly. | +| `helpers.PoPOptions` | Go options struct for the delivery-proof signer; Python takes the same values as keyword arguments and TS as an options object. | +| `helpers.RedactURL` | Go query-stripping helper for a signed URL headed to a log; py/ts redact inline at the log site. | +| `helpers.RetrievalAuthFailureReasonFromToken` | Go lookup from the delivery edge's refusal token to the typed enum; py/ts branch on the token string directly. | | `helpers.SharedValidator` | Go protovalidate validator singleton; TS/Python ship no protovalidate face. | +| `helpers.SignOfferAcceptanceWith` | Go Signer-custody variant of SignOfferAcceptance so the SDK never holds the key; py/ts pass key material directly to their single acceptance signer. | | `helpers.SignOptions` | Go options struct for SignRequest; py/ts pass options via kwargs/options objects. | | `helpers.SignatureAgentFromContext` | Go context.Context accessor; py/ts thread signature-agent state explicitly. | | `helpers.SignedURL` | Go signed-URL result value type; py/ts return language-native result objects. | @@ -252,6 +286,17 @@ Go constructs (functional-option builders, `errors.Is` sentinels, value types, c | `helpers.VerifyRequestResolved` | Go resolver-injected VerifyRequest overload; py/ts expose a single verify entry point. | | `helpers.WithSignatureAgent` | Go functional-option builder; py/ts pass options via kwargs/options objects. | | `resolvers.ActiveKeyScanOptions` | Go scan-options struct; py/ts pass scan options inline. | +| `resolvers.Content` | Go value struct for one fetched resource; py/ts return their runtime-native body/type pair. | +| `resolvers.ContentFetchOptions` | Go options struct for the content-download leg; py/ts pass the same values as kwargs/an options object. | +| `resolvers.ContentFetcher` | Part of the Go content-download leg; the TS/Python download verbs are tracked with their unary client work. | +| `resolvers.DefaultContentTimeout` | Go default bound for one content fetch. Neither Python nor TS ships a content fetcher, so no counterpart holds this default today; it arrives with the download verbs tracked alongside their unary client work. | +| `resolvers.DefaultMaxContentBytes` | Go default body cap for one content fetch. Neither Python nor TS ships a content fetcher, so no counterpart holds this cap today; it arrives with the download verbs tracked alongside their unary client work. | +| `resolvers.ErrEndpointRefused` | Go errors.Is sentinel for a manifest-advertised endpoint the resolver will not return (wrong host, or userinfo). Python and TS DO ship endpoint resolvers, and neither enforces this rule yet; closing that gap is tracked separately and will bring a mapped counterpart. | +| `resolvers.FetchError` | Go typed error for the content leg; py/ts raise/throw a runtime-native error carrying the same class and reason. | +| `resolvers.FetchFailure` | Go failure-class enum for the content leg; py/ts express the same classes as string literals. | +| `resolvers.NewContentFetcher` | Go constructor for the content-download leg; py/ts fold construction into their client. | +| `resolvers.NewGuardedTransport` | Go constructor composing the SSRF guard over a caller's base transport; py/ts expose their guarded fetch as a single factory with no separable base. | +| `resolvers.ProofSigner` | Go interface seam that keeps key custody out of the dialing tier; py/ts inject a signing callable instead of a named interface. | | `resolvers.SSRFCheckRedirect` | Go redirect-policy hook; py/ts fold redirect checks into the guard (async_ssrf_guard / the guarded fetch). | ## Cross-language conformance-vector replay diff --git a/gen/descriptor.binpb b/gen/descriptor.binpb index 31e82678..71cee430 100644 Binary files a/gen/descriptor.binpb and b/gen/descriptor.binpb differ diff --git a/gen/go/ramp/v1/ramp.pb.go b/gen/go/ramp/v1/ramp.pb.go index 62454d55..c8d8d5b2 100644 --- a/gen/go/ramp/v1/ramp.pb.go +++ b/gen/go/ramp/v1/ramp.pb.go @@ -6689,7 +6689,19 @@ type WellKnownManifest struct { Operator *string `protobuf:"bytes,10,opt,name=operator,proto3,oneof" json:"operator,omitempty"` // Exchange-only. Operator's corporate domain (may differ from domain). OperatorDomain *string `protobuf:"bytes,11,opt,name=operator_domain,json=operatorDomain,proto3,oneof" json:"operator_domain,omitempty"` - // Exchange-only. ExchangeService endpoint URL. + // Exchange-only. ExchangeService endpoint URL. MUST be on the same host AND + // PORT that serve this manifest, or on a subdomain of that host on that port, + // and MUST NOT carry userinfo. A consumer refuses an endpoint anywhere else: + // this document is only as trustworthy as the host that served it, so an + // endpoint naming an unrelated host would let whoever answers for the manifest + // redirect a signed call to a party the signature never covered, and another + // port is another service the publisher of the manifest need not control. The + // host match is on a full dot-delimited label boundary, so evil-a.com is not a + // subdomain of a.com. A port equal to the scheme's default and an omitted port + // are the SAME port, so https://x, https://x:443 and x all match. An Exchange + // reachable on a non-default port names that port on both sides. (One + // paragraph deliberately: a blank line here routes the first paragraph into + // the generated types' JSON-Schema title, which the Pydantic/Zod export drops.) Endpoint *string `protobuf:"bytes,12,opt,name=endpoint,proto3,oneof" json:"endpoint,omitempty"` // Exchange-only. Health check endpoint URL. HealthEndpoint *string `protobuf:"bytes,13,opt,name=health_endpoint,json=healthEndpoint,proto3,oneof" json:"health_endpoint,omitempty"` diff --git a/gen/python/wire/models.py b/gen/python/wire/models.py index 742de083..13ada668 100644 --- a/gen/python/wire/models.py +++ b/gen/python/wire/models.py @@ -1493,7 +1493,8 @@ class WellKnownManifest(WireModel): '', description='Canonical domain serving this manifest.' ) endpoint: str | None = Field( - None, description='Exchange-only. ExchangeService endpoint URL.' + None, + description="Exchange-only. ExchangeService endpoint URL. MUST be on the same host AND\n PORT that serve this manifest, or on a subdomain of that host on that port,\n and MUST NOT carry userinfo. A consumer refuses an endpoint anywhere else:\n this document is only as trustworthy as the host that served it, so an\n endpoint naming an unrelated host would let whoever answers for the manifest\n redirect a signed call to a party the signature never covered, and another\n port is another service the publisher of the manifest need not control. The\n host match is on a full dot-delimited label boundary, so evil-a.com is not a\n subdomain of a.com. A port equal to the scheme's default and an omitted port\n are the SAME port, so https://x, https://x:443 and x all match. An Exchange\n reachable on a non-default port names that port on both sides. (One\n paragraph deliberately: a blank line here routes the first paragraph into\n the generated types' JSON-Schema title, which the Pydantic/Zod export drops.)", ) exchanges: list[AuthorizedExchange] | None = Field( None, diff --git a/gen/ts/wire/schemas.ts b/gen/ts/wire/schemas.ts index 4bc9ada0..844d7cdb 100644 --- a/gen/ts/wire/schemas.ts +++ b/gen/ts/wire/schemas.ts @@ -204,5 +204,5 @@ export const UsageReportResponseSchema = wire(z.object({ "ext": z.record(z.strin export const WBAFileSchema = wire(z.object({ "keys": z.array(z.object({ "alg": z.string().describe("Signing algorithm. RAMP v1.0: MUST be \"EdDSA\".").default(""), "crv": z.string().describe("Curve. RAMP v1.0: MUST be \"Ed25519\".").default(""), "kty": z.string().describe("Key type. RAMP v1.0: MUST be \"OKP\".").default(""), "not_after": z.string().describe("RFC3339 timestamp. Key is invalid at and after this instant\n (strict upper bound).").default(""), "not_before": z.string().describe("RFC3339 timestamp. Key is invalid before this instant.").default(""), "use": z.string().describe("Intended key use. RAMP v1.0: MUST be \"sig\".").default(""), "x": z.string().describe("base64url-encoded 32-byte Ed25519 public key.").default("") }).describe("RAMP v1.0 supports Ed25519 only: kty=\"OKP\", crv=\"Ed25519\", alg=\"EdDSA\".\n Additional curves are a later concern.\n\n Time bounds are RFC3339 strings (sortable, ops-debuggable, avoids the\n JWT nbf/exp collision). At least one key in the served key set (WBAFile.keys)\n MUST have `not_before <= now < not_after`. Verification MUST reject\n signatures whose key falls outside its window.\n\n Keys carry no `kid`: the RFC 9421 keyid is the RFC 7638 JWK Thumbprint,\n computed locally by the verifier. Carrying a kid alongside the thumbprint\n created a drift surface and is removed.")).describe("RFC 7517 JWK Set \"keys\" member. RAMP v1: Ed25519 (OKP) keys, each with\n not_before/not_after RAMP extension members.").optional(), "revocation_url": z.string().describe("Directory-level emergency revocation channel. One per directory; the list\n it points to enumerates revoked key thumbprints. Consumers poll on a 300s\n cadence (±10% jitter) and replace their local revoked set with the response.").optional() }).describe("WBAFile — Pure Web Bot Auth directory served at the WBA-canonical well-known\n path (/.well-known/http-message-signatures-directory). A JOSE JWK Set per\n RFC 7517 §5 plus a directory-level revocation pointer. JWKs carry no kid; the\n RFC 9421 keyid is the RFC 7638 JWK Thumbprint. Off-the-shelf WBA verifiers\n read the `keys` array and ignore RAMP's extra members (per-key\n not_before/not_after, and revocation_url) per RFC 7517 §5.")); -export const WellKnownManifestSchema = wire(z.object({ "accepted_verifiers": z.array(z.string()).describe("Exchange-only. Trusted attestation verification vendors (domains).").optional(), "base_currency": z.string().describe("Exchange-only. Base currency for pricing (ISO 4217). All unit_cost\n values from this Exchange are denominated in this currency.").optional(), "catalog_contributors": z.array(z.object({ "domain": z.string().describe("Canonical domain of the authorized contributor (e.g., \"doubleverify.com\").").default(""), "relationship": z.string().describe("Relationship of this contributor to the provider.\n Examples: \"verifier\" (resource intelligence vendor that attests to resource\n properties), \"exchange\" (an Exchange that enriches catalog entries).").default("") }).describe("CatalogContributor — A third party authorized to push catalog metadata\n (including attestations) on behalf of a provider.")).describe("Publisher-only. Authorized third-party catalog contributors.\n MUST be empty for non-publisher roles.").optional(), "catalog_endpoint": z.string().describe("Exchange-only. CatalogService endpoint URL (if exposed).").optional(), "contact": z.string().describe("Contact email (licensing, integration, security).").optional(), "delivery_methods_supported": z.array(z.enum(["DELIVERY_METHOD_DIRECT","DELIVERY_METHOD_INSTRUCTIONS","DELIVERY_METHOD_STREAMING"])).describe("Exchange-only. Supported delivery methods.").optional(), "domain": z.string().describe("Canonical domain serving this manifest.").default(""), "endpoint": z.string().describe("Exchange-only. ExchangeService endpoint URL.").optional(), "exchanges": z.array(z.object({ "domain": z.string().describe("Canonical domain of the Exchange.").default(""), "endpoint": z.string().describe("RAMP ExchangeService endpoint URL.").default(""), "ext": z.record(z.string(), z.any()).describe("Extension point").optional(), "ext_critical": z.array(z.string()).describe("Critical extension keys (COSE crit pattern, RFC 9052).\n Lists keys within ext that the consumer MUST understand.\n Unknown keys in this list → reject with UNKNOWN_CRITICAL_EXTENSION.\n Empty (default) → all ext keys are safe to ignore.").optional(), "relationship": z.enum(["PROVIDER_RELATIONSHIP_DIRECT","PROVIDER_RELATIONSHIP_RESELLER"]).describe("Relationship type (mirrors ads.txt DIRECT/RESELLER).") }).describe("AuthorizedExchange — A Exchange authorized to sell this provider's resources.")).describe("Publisher-only. Authorized exchanges for this publisher's resources.\n Like ads.txt — declares who may sell. MUST be empty for non-publisher\n roles.").optional(), "ext": z.record(z.string(), z.any()).describe("Extension point").optional(), "ext_critical": z.array(z.string()).describe("Critical extension keys (COSE crit pattern, RFC 9052). Lists keys\n within ext that the consumer MUST understand. Unknown values reject\n with UNKNOWN_CRITICAL_EXTENSION. Empty (default) → ignore-unknown.").optional(), "gnap_grant_endpoint": z.string().describe("Exchange-only. GNAP grant endpoint when GNAP is supported.").optional(), "hash_methods_supported": z.array(z.string()).describe("Exchange-only. Accepted resource hash methods for attestation\n verification.").optional(), "health_endpoint": z.string().describe("Exchange-only. Health check endpoint URL.").optional(), "max_intermediary_hops": z.coerce.number().int().gte(-2147483648).lte(2147483647).describe("Exchange-only. Maximum forwarding hops this Exchange tolerates on an inbound\n request (Agent → Broker → … → Exchange), counted as RFC 9421 HTTP Message\n Signatures. A request carrying more SHOULD be rejected. Lets Exchanges\n publish their chain-depth tolerance so Brokers prune before forwarding.\n Absent = no published limit (Exchange applies its own default policy).").optional(), "name": z.string().describe("Exchange-only. Human-readable Exchange name.").optional(), "oidc_issuer": z.string().describe("Exchange-only. OIDC Discovery URL when OAuth methods are supported.").optional(), "operator": z.string().describe("Exchange-only. Organization operating this Exchange.").optional(), "operator_domain": z.string().describe("Exchange-only. Operator's corporate domain (may differ from domain).").optional(), "pricing_models_supported": z.array(z.enum(["PRICING_MODEL_FREE","PRICING_MODEL_PER_UNIT","PRICING_MODEL_FLAT"])).describe("Exchange-only. Supported pricing models.").optional(), "privacy_uri": z.string().describe("Exchange-only. Privacy policy URL.").optional(), "protocol_versions_supported": z.array(z.string()).describe("Exchange-only. Supported RAMP protocol versions (e.g. [\"1.0\"]).").optional(), "registration_schema": z.record(z.string(), z.any()).describe("Exchange-only. JSON Schema (draft 2020-12) describing the\n RegisterRequest.registration_data object this Exchange expects. This field\n is the single home of the enforce/pass-through contract, and publishing it\n IS the enforcement switch. Present: this Exchange validates\n registration_data against the schema and refuses a non-conforming payload\n with REGISTRATION_FAILURE_REASON_INVALID_REGISTRATION_DATA, naming the\n offending members in RegistrationFailure.field_errors. Absent:\n registration_data is passed through to the system of record uninspected,\n so an Exchange that publishes no schema needs no change to stay\n conformant. Safety rules, because a consumer reads this schema out of a\n third party's manifest: it MUST be self-contained, and a consumer MUST NOT\n resolve a remote $ref out of it — doing so turns every reader into an SSRF\n vector aimed at a URL the schema's author chose. A consumer SHOULD bound\n validation time and recursion depth; draft 2020-12 `pattern` admits\n regexes with catastrophic backtracking. Size is capped at 16KB, measured\n as the UTF-8 bytes of this member as served in ramp.json; a consumer\n SHOULD reject an oversized schema and skip its local pre-check rather than\n truncate it, which leaves the Exchange's own enforcement the deciding\n check exactly as when no schema is published.").optional(), "role": z.enum(["ROLE_AGENT","ROLE_EXCHANGE","ROLE_BROKER","ROLE_PUBLISHER"]).describe("Role this manifest describes."), "supported_auth_methods": z.array(z.enum(["AUTH_METHOD_GNAP","AUTH_METHOD_OAUTH_DPOP","AUTH_METHOD_OAUTH_BEARER","AUTH_METHOD_OAUTH_MTLS"])).describe("Exchange-only. Authorization methods this Exchange supports\n (ordered by preference).").optional(), "supported_profiles": z.array(z.string()).describe("Exchange-only. Domain extension profiles this Exchange conforms to.\n See standards-layering docs.").optional(), "terms_uri": z.string().describe("Exchange-only. Terms of service URL.").optional(), "ver": z.string().describe("RAMP protocol version of THIS MANIFEST DOCUMENT's schema — a namespace\n separate from the RPC envelope `ver`, deliberately not coupled to it.\n MUST equal \"1.0\"; consumers REJECT unrecognised major versions.").default("") }).describe("Commercial graph only: role, authorized exchanges/contributors, and exchange\n capability fields. Identity keys are NOT here — they live in the WBA directory\n (WBAFile) served at /.well-known/http-message-signatures-directory and are\n referenced by RFC 7638 thumbprint, never republished here.\n Per-role fields are populated only when that role applies; consumers\n MUST ignore non-applicable fields based on `role`.")); +export const WellKnownManifestSchema = wire(z.object({ "accepted_verifiers": z.array(z.string()).describe("Exchange-only. Trusted attestation verification vendors (domains).").optional(), "base_currency": z.string().describe("Exchange-only. Base currency for pricing (ISO 4217). All unit_cost\n values from this Exchange are denominated in this currency.").optional(), "catalog_contributors": z.array(z.object({ "domain": z.string().describe("Canonical domain of the authorized contributor (e.g., \"doubleverify.com\").").default(""), "relationship": z.string().describe("Relationship of this contributor to the provider.\n Examples: \"verifier\" (resource intelligence vendor that attests to resource\n properties), \"exchange\" (an Exchange that enriches catalog entries).").default("") }).describe("CatalogContributor — A third party authorized to push catalog metadata\n (including attestations) on behalf of a provider.")).describe("Publisher-only. Authorized third-party catalog contributors.\n MUST be empty for non-publisher roles.").optional(), "catalog_endpoint": z.string().describe("Exchange-only. CatalogService endpoint URL (if exposed).").optional(), "contact": z.string().describe("Contact email (licensing, integration, security).").optional(), "delivery_methods_supported": z.array(z.enum(["DELIVERY_METHOD_DIRECT","DELIVERY_METHOD_INSTRUCTIONS","DELIVERY_METHOD_STREAMING"])).describe("Exchange-only. Supported delivery methods.").optional(), "domain": z.string().describe("Canonical domain serving this manifest.").default(""), "endpoint": z.string().describe("Exchange-only. ExchangeService endpoint URL. MUST be on the same host AND\n PORT that serve this manifest, or on a subdomain of that host on that port,\n and MUST NOT carry userinfo. A consumer refuses an endpoint anywhere else:\n this document is only as trustworthy as the host that served it, so an\n endpoint naming an unrelated host would let whoever answers for the manifest\n redirect a signed call to a party the signature never covered, and another\n port is another service the publisher of the manifest need not control. The\n host match is on a full dot-delimited label boundary, so evil-a.com is not a\n subdomain of a.com. A port equal to the scheme's default and an omitted port\n are the SAME port, so https://x, https://x:443 and x all match. An Exchange\n reachable on a non-default port names that port on both sides. (One\n paragraph deliberately: a blank line here routes the first paragraph into\n the generated types' JSON-Schema title, which the Pydantic/Zod export drops.)").optional(), "exchanges": z.array(z.object({ "domain": z.string().describe("Canonical domain of the Exchange.").default(""), "endpoint": z.string().describe("RAMP ExchangeService endpoint URL.").default(""), "ext": z.record(z.string(), z.any()).describe("Extension point").optional(), "ext_critical": z.array(z.string()).describe("Critical extension keys (COSE crit pattern, RFC 9052).\n Lists keys within ext that the consumer MUST understand.\n Unknown keys in this list → reject with UNKNOWN_CRITICAL_EXTENSION.\n Empty (default) → all ext keys are safe to ignore.").optional(), "relationship": z.enum(["PROVIDER_RELATIONSHIP_DIRECT","PROVIDER_RELATIONSHIP_RESELLER"]).describe("Relationship type (mirrors ads.txt DIRECT/RESELLER).") }).describe("AuthorizedExchange — A Exchange authorized to sell this provider's resources.")).describe("Publisher-only. Authorized exchanges for this publisher's resources.\n Like ads.txt — declares who may sell. MUST be empty for non-publisher\n roles.").optional(), "ext": z.record(z.string(), z.any()).describe("Extension point").optional(), "ext_critical": z.array(z.string()).describe("Critical extension keys (COSE crit pattern, RFC 9052). Lists keys\n within ext that the consumer MUST understand. Unknown values reject\n with UNKNOWN_CRITICAL_EXTENSION. Empty (default) → ignore-unknown.").optional(), "gnap_grant_endpoint": z.string().describe("Exchange-only. GNAP grant endpoint when GNAP is supported.").optional(), "hash_methods_supported": z.array(z.string()).describe("Exchange-only. Accepted resource hash methods for attestation\n verification.").optional(), "health_endpoint": z.string().describe("Exchange-only. Health check endpoint URL.").optional(), "max_intermediary_hops": z.coerce.number().int().gte(-2147483648).lte(2147483647).describe("Exchange-only. Maximum forwarding hops this Exchange tolerates on an inbound\n request (Agent → Broker → … → Exchange), counted as RFC 9421 HTTP Message\n Signatures. A request carrying more SHOULD be rejected. Lets Exchanges\n publish their chain-depth tolerance so Brokers prune before forwarding.\n Absent = no published limit (Exchange applies its own default policy).").optional(), "name": z.string().describe("Exchange-only. Human-readable Exchange name.").optional(), "oidc_issuer": z.string().describe("Exchange-only. OIDC Discovery URL when OAuth methods are supported.").optional(), "operator": z.string().describe("Exchange-only. Organization operating this Exchange.").optional(), "operator_domain": z.string().describe("Exchange-only. Operator's corporate domain (may differ from domain).").optional(), "pricing_models_supported": z.array(z.enum(["PRICING_MODEL_FREE","PRICING_MODEL_PER_UNIT","PRICING_MODEL_FLAT"])).describe("Exchange-only. Supported pricing models.").optional(), "privacy_uri": z.string().describe("Exchange-only. Privacy policy URL.").optional(), "protocol_versions_supported": z.array(z.string()).describe("Exchange-only. Supported RAMP protocol versions (e.g. [\"1.0\"]).").optional(), "registration_schema": z.record(z.string(), z.any()).describe("Exchange-only. JSON Schema (draft 2020-12) describing the\n RegisterRequest.registration_data object this Exchange expects. This field\n is the single home of the enforce/pass-through contract, and publishing it\n IS the enforcement switch. Present: this Exchange validates\n registration_data against the schema and refuses a non-conforming payload\n with REGISTRATION_FAILURE_REASON_INVALID_REGISTRATION_DATA, naming the\n offending members in RegistrationFailure.field_errors. Absent:\n registration_data is passed through to the system of record uninspected,\n so an Exchange that publishes no schema needs no change to stay\n conformant. Safety rules, because a consumer reads this schema out of a\n third party's manifest: it MUST be self-contained, and a consumer MUST NOT\n resolve a remote $ref out of it — doing so turns every reader into an SSRF\n vector aimed at a URL the schema's author chose. A consumer SHOULD bound\n validation time and recursion depth; draft 2020-12 `pattern` admits\n regexes with catastrophic backtracking. Size is capped at 16KB, measured\n as the UTF-8 bytes of this member as served in ramp.json; a consumer\n SHOULD reject an oversized schema and skip its local pre-check rather than\n truncate it, which leaves the Exchange's own enforcement the deciding\n check exactly as when no schema is published.").optional(), "role": z.enum(["ROLE_AGENT","ROLE_EXCHANGE","ROLE_BROKER","ROLE_PUBLISHER"]).describe("Role this manifest describes."), "supported_auth_methods": z.array(z.enum(["AUTH_METHOD_GNAP","AUTH_METHOD_OAUTH_DPOP","AUTH_METHOD_OAUTH_BEARER","AUTH_METHOD_OAUTH_MTLS"])).describe("Exchange-only. Authorization methods this Exchange supports\n (ordered by preference).").optional(), "supported_profiles": z.array(z.string()).describe("Exchange-only. Domain extension profiles this Exchange conforms to.\n See standards-layering docs.").optional(), "terms_uri": z.string().describe("Exchange-only. Terms of service URL.").optional(), "ver": z.string().describe("RAMP protocol version of THIS MANIFEST DOCUMENT's schema — a namespace\n separate from the RPC envelope `ver`, deliberately not coupled to it.\n MUST equal \"1.0\"; consumers REJECT unrecognised major versions.").default("") }).describe("Commercial graph only: role, authorized exchanges/contributors, and exchange\n capability fields. Identity keys are NOT here — they live in the WBA directory\n (WBAFile) served at /.well-known/http-message-signatures-directory and are\n referenced by RFC 7638 thumbprint, never republished here.\n Per-role fields are populated only when that role applies; consumers\n MUST ignore non-applicable fields based on `role`.")); diff --git a/proto/CHANGELOG.md b/proto/CHANGELOG.md index e335c003..44bead94 100644 --- a/proto/CHANGELOG.md +++ b/proto/CHANGELOG.md @@ -2,6 +2,97 @@ ## Unreleased +**`WellKnownManifest.endpoint` states its host binding (no wire change; conformance-affecting).** +The field said only "Exchange-only. ExchangeService endpoint URL", so nothing told an Exchange +operator that the address it advertises must stay on its own domain. It now does: the endpoint +MUST be on the host AND PORT that SERVE the manifest — not the self-asserted `domain` member +inside it — or on a subdomain of that host on that port, and MUST NOT carry userinfo. The manifest +is only as trustworthy as the host that served it, so an endpoint naming an unrelated host would +let whoever answers for the manifest redirect a signed call to a party the offer's signature never +covered — and a dial-time address guard has no objection to an unrelated PUBLIC host. Another port +is another service, which the party publishing the manifest need not control. The host match is on +a full dot-delimited label boundary, so `evil-a.com` is not a subdomain of `a.com`. A port equal to +the scheme's default and an omitted port are the SAME port, so `https://x`, `https://x:443` and `x` +all match; the scheme itself is not compared, and the default-port folding is scheme-relative so +that it cannot become a scheme check by accident. + +**This is the first entry in this changelog that changes what conforms without changing the +wire.** The classifier is deliberately not `(breaking)`: this change moves no field, message, or +encoding, and `buf breaking` reports nothing — while the bare `(breaking)` entries below all mark +a descriptor delta, and the one qualified use ("breaking for the generated clients") names the +audience it breaks. What this change does instead is narrow what a conformant manifest may say. + +**Two shapes that are conformant today will be refused after this.** The first is an Exchange +serving its API from a separate DOMAIN — a CDN, a hosting provider. The second is an Exchange on +a separate PORT: a single-domain deployment serving `/.well-known/ramp.json` on its default port +and advertising `"endpoint": "https://exchange.example:8443/v1"` is refused, as is the mirror +image (a portless endpoint under a manifest served on `:8443`) and a subdomain reached across +ports. A single domain is therefore no longer sufficient on its own — the authority must match on +both halves. + +Remedies, by shape. For a separate domain, front the API under a subdomain of the domain serving +the `ramp.json`. For a separate port, either move the API onto the port the manifest is served +from, or serve the manifest from the API's own authority — `https://exchange.example:8443/.well-known/ramp.json` +alongside `https://exchange.example:8443/v1`. Writing a scheme's default port out in full is NOT +a mismatch and needs no change. + +Both are refused as `ErrEndpointRefused`, which classifies as a FINAL verdict rather than a +transport failure — so a client will not retry its way out of a misconfiguration, and the symptom +is a usage report that never lands rather than one that is slow. + +Enforcement moved with the rule: it now runs in the SDK's shared endpoint resolver rather than +in one client, so every consumer of that resolver inherits it without changing a line. Two +consequences for anyone re-pinning. Resolution can now fail with a new `ErrEndpointRefused` +sentinel, which is a VERDICT — the Exchange answered and the answer is unusable — and a +classifier that branches only on the older `ErrNoEndpoint` will drop it into its +transport-failure bucket and retry something that will never succeed; add the new sentinel +alongside. And a Broker that resolves endpoints through this package inherits the rule for the +paths that use it. `gen/` and the website mirror are regenerated; proto comments only. + +**Go SDK: the delivery fetch correlates, and the offer-key cache is bounded (additive, no wire +change).** `resolvers.ContentFetchOptions` gained a `RequestID` hook, and `connect.NewClient` +feeds it the same mint the RPC legs read — so `WithRequestIDFunc` now reaches all three legs and +a delivery GET carries `X-Request-ID`. It did not before, and could not: the RPC legs correlate +through a Connect interceptor, which a plain GET never traverses, and there was no seam to add +one. **This changes what arrives at a delivery edge.** An edge that mints its own id when the +header is absent will now see the caller's instead, which is the point — a refused delivery used +to produce two log records under two ids with nothing joining them, on the one leg where +delivery failures are diagnosed. A fetcher built directly with no `RequestID` still sends no +header: this tier mints nothing of its own. + +`resolvers.CachedOfferKeyResolver`'s per-domain cache now evicts least-recently-used at a fixed +cap, like the endpoint cache and the per-origin client pool. Its key is a domain off +`Offer.exchange`, so which entries appear is driven by incoming offers, and an entry's expiry is +a freshness check rather than a removal — a stale entry held its slot indefinitely. Reaching it +needed a resolvable host serving a valid directory per domain, so the case was narrow rather +than open, but two sibling structures over the same key space were already bounded and this one +was not. + +**Go SDK: the Connect client covers the agent verb set, and its signing knobs are reachable +(additive, no wire change).** `connect.Client` gained `ReportUsage`, `Dispute` and `Fetch`, and +`connect.NewBrokerClient` gained `Resolve` — the client previously exposed `Discover` and +`Execute` alone, so a caller needing any of the rest had to assemble its own from +`rampv1connect` plus `core.NewSigningTransport`, which is the duplication the SDK exists to +remove. `Resolve` returns the same fail-closed `{verified, rejected}` split `Discover` does, +through the same `core.Verifier`; `Fetch` performs proof-of-possession on an agent-bound URL and +dials only through the SSRF-guarded client. + +Five client options join them, each because a value the tier below already accepted had no way +in: `WithSignWindow` (the RFC 9421 request freshness window — pair it with +`core.MonotonicWindow` when the peer screens replays on `(key id, signature)`, since one-second +timestamp resolution makes two identical requests inside a second sign to the same bytes), +`WithSignatureAgent` (the WBA directory origin the client signs as), `WithProofWindow`, +`WithContentTimeout` and `WithMaxContentBytes`. + +`WithSignatureAgent` is worth reading twice if you verify signatures. `signature-agent` is one +of the five REQUIRED covered components, so the header is signed whether or not a value was +supplied — a client that does not set it signs an EMPTY one. A peer that resolves the caller's +key by fetching the WBA directory at that origin then has nothing to resolve and refuses the +call at verification, which surfaces as a 401 from an otherwise healthy Exchange rather than as +anything the routing checks would catch. The value is stamped set-if-absent, so a relay +forwarding an originating agent's request does not overwrite the value that agent's own +signature covers. See `docs/sdk-parity-matrix.md` for the per-language surface. + **SDK (all 3 languages): the registration-failure builder can carry the field errors (additive, no wire change).** `helpers.RegistrationFailureDetail` (Go), `registration_failure_detail` (Python) and `registrationFailureDetail` (TS) now accept the diff --git a/proto/ramp/v1/ramp.proto b/proto/ramp/v1/ramp.proto index 4044f268..bce82993 100644 --- a/proto/ramp/v1/ramp.proto +++ b/proto/ramp/v1/ramp.proto @@ -2560,7 +2560,19 @@ message WellKnownManifest { // Exchange-only. Operator's corporate domain (may differ from domain). optional string operator_domain = 11; - // Exchange-only. ExchangeService endpoint URL. + // Exchange-only. ExchangeService endpoint URL. MUST be on the same host AND + // PORT that serve this manifest, or on a subdomain of that host on that port, + // and MUST NOT carry userinfo. A consumer refuses an endpoint anywhere else: + // this document is only as trustworthy as the host that served it, so an + // endpoint naming an unrelated host would let whoever answers for the manifest + // redirect a signed call to a party the signature never covered, and another + // port is another service the publisher of the manifest need not control. The + // host match is on a full dot-delimited label boundary, so evil-a.com is not a + // subdomain of a.com. A port equal to the scheme's default and an omitted port + // are the SAME port, so https://x, https://x:443 and x all match. An Exchange + // reachable on a non-default port names that port on both sides. (One + // paragraph deliberately: a blank line here routes the first paragraph into + // the generated types' JSON-Schema title, which the Pydantic/Zod export drops.) optional string endpoint = 12; // Exchange-only. Health check endpoint URL. diff --git a/sdk/go/README.md b/sdk/go/README.md index d230117e..69d154c6 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -9,14 +9,14 @@ directly with no `replace` directive. | **L0** | `gen/go/ramp/v1`, `gen/go/vocab/*` | generated wire types (consumed, never rebuilt) | | **L1** | **`sdk/go/helpers`** | stateless, **IO-free** protocol helpers — RFC 9421/7638 crypto, offer/acceptance verify, static key resolution, validation | | L2 · I/O | **`sdk/go/resolvers`** | the network-fetching tier: well-known JWKS / WBA directory / `ramp.json` endpoint / offer-key resolvers + the SSRF-guarded HTTP client. Runs on a maintained `net/http` client behind the SSRF guard; composes L1, never the reverse | -| L2 · transport | `sdk/go/core` (transport-neutral: Verifier, {verified,rejected}, VerifiedOffer guard, signing RoundTripper, ReplayStore — zero Connect) · `sdk/go/connect` (Connect **client** binding: `NewClient` + client options + `ErrorDetailFrom`) · `sdk/go/connectserver` (Connect **server** binding: `NewExchangeServiceHandler` + server options + `AsConnectError` + `AttachErrorDetail`/`AttachDetail` + reject→code) | transport-neutral core + Connect client/server bindings (state injected) | +| L2 · transport | `sdk/go/core` (transport-neutral: Verifier, {verified,rejected}, `DiscoveryResult` per-URI groups, VerifiedOffer guard, signing RoundTripper, ReplayStore — zero Connect) · `sdk/go/connect` (Connect **client** binding: `NewClient` + `NewBrokerClient` + the agent verbs **`Discover` · `Resolve` · `Execute` · `ReportUsage` · `Dispute` · `Fetch`** + client options + the `CallError` taxonomy + `ErrorDetailFrom`) · `sdk/go/connectserver` (Connect **server** binding: `NewExchangeServiceHandler` + server options + `AsConnectError` + `AttachErrorDetail`/`AttachDetail` + reject→code) | transport-neutral core + Connect client/server bindings (state injected) | | L3 | separate packages | framework adapters (convert, never replace) — later | The `L2` tier is split by kind: the **I/O** package (`resolvers`) is the only tier that dials the network (it holds the maintained HTTP client and the SSRF guard), -while the **transport** packages (`core` / `connect` / `connectserver`) are -transport-neutral composition and Connect bindings with no network fetch of their -own. An io-leaf guard (`helpers/io_leaf_guard_test.go`) fails the build if any pure +while the **transport** packages (`core` / `connect` / `connectserver`) hold no +dialing surface of their own — `connect` composes the I/O tier's guarded +transport and content fetcher rather than opening a second one. An io-leaf guard (`helpers/io_leaf_guard_test.go`) fails the build if any pure L1 file drags in a dialing surface, so the fetch surface cannot leak back down. ## L1 — `helpers` @@ -74,8 +74,33 @@ wire, _ := helpers.FormatMoney(rate.Mul(decimal.NewFromInt(qty))) if err := helpers.Validate(req); err != nil { /* helpers.ValidationRuleIDs(err) */ } ``` +**Agent-binding proof of possession** (ADR-013) — the SIGN face of the header pair +a bound delivery fetch presents. The covered set is exactly `@method` + +`@target-uri`: a GET has no body to digest, and the signed URL is itself the +credential. The key arrives as a `Signer` plus the public half, so custody never +moves into the SDK: + +```go +binding, _ := helpers.SignAgentBinding(ctx, signer, agentPub, helpers.PoPOptions{ + URL: signedURL, Created: created, Expires: expires, // keep the window short +}) +binding.Apply(req.Header) // X-RAMP-Agent-Key + Signature-Input + Signature +``` + +**Routing predicates** — the two pure checks that precede a signed call to an +address a network party named (a manifest may point at itself or a subdomain of +itself, and nothing else): + +```go +bare, _ := helpers.IsBareHost(offer.GetExchange()) // no scheme/path/query +ok, _ := helpers.HostAnchored(exchangeDomain, endpoint) // label-boundary match +``` + **Also:** RFC 7638 `Thumbprint`, ADR-019 `ErrorDetail` constructors + -`AsConnectError`/`ErrorDetailFrom`/`Reason`, `NewIdempotencyKey`, scope helpers. +`AsConnectError`/`ErrorDetailFrom`/`Reason`, `NewIdempotencyKey`, scope helpers, +`RedactURL` (a signed URL carries its credential in the query — never log it raw), +and `RetrievalAuthFailureReasonFromToken` (the delivery edge's refusal vocabulary, +mapped onto the typed enum). ## L2 · I/O — `resolvers` @@ -90,14 +115,36 @@ import "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" - **Key resolvers** — `NewWellKnownKeyResolver` (well-known JWKS, TTL cache), `NewWBAKeyResolver` (WBA directory, revocation/expiry-aware, with a `Run` poller). - **Endpoint resolver** — `NewWellKnownEndpointResolver` discovers an Exchange's - `retrieval_endpoint` from `/.well-known/ramp.json` (`ErrNoEndpoint` when absent). + own service endpoint (`WellKnownManifest.endpoint`) from `/.well-known/ramp.json`, + host-keyed and cached per host. Two sentinels, and the difference decides whether + a caller should retry: `ErrNoEndpoint` when the manifest was read and advertises + none, `ErrEndpointRefused` when it advertises one this resolver will not hand back + — on a host unrelated to the one that served the manifest, or carrying userinfo. + Both are verdicts, so both are final; anything else is a transport failure and + worth retrying. The key is an offer-supplied host, so the cache evicts + least-recently-used at a fixed cap and concurrent lookups for one host coalesce to + a single fetch. - **Active-key selection** — `ActiveEd25519Key` / `ActiveEd25519KeyWithExpiry` pick an identity's window-active key by document order; the `…Screened` variants fold in a revoked-thumbprint screen. `NewCachedOfferKeyResolver` caches the selected offer key with an expiry clamped to the key's `not_after`. - **SSRF-guarded client** — `NewGuardedClientFromEnv` is the single construction path - every fetch uses; `SSRFGuard` / `SSRFCheckRedirect` are the injectable dial-time + every fetch uses; `NewGuardedTransport` composes the same guard over a caller's + own base transport, so application settings ride UNDER the guard rather than + replacing it; `SSRFGuard` / `SSRFCheckRedirect` are the injectable dial-time address guard and redirect re-vet for callers wiring their own `http.Client`. +- **Content fetch** — `NewContentFetcher` retrieves the bytes a signed delivery URL + names, presenting a proof of possession through the injected `ProofSigner` seam + (so this tier holds no key material). Bounded body, bounded error body, media + type reported rather than sniffed, and a typed `FetchError` class. + +**Redirects: the guarded client follows, a signed leg refuses.** Following five +hops is right for a public well-known document — the address is re-pinned and the +scheme re-vetted on each. It is wrong for anything carrying a credential, so the +content fetch and the RPC legs take only the guarded `.Transport` and install +their own refusal: following a redirect either replays a proof bound to the old +URL, or hands a fresh proof of possession of the agent's key to whatever host the +first hop named. The guard is driven by exactly two orthogonal env flags — `SKIP_SSRF` (drop the dial-time address guard) and `ALLOW_INSECURE` (permit plaintext http) — both diff --git a/sdk/go/connect/broker.go b/sdk/go/connect/broker.go new file mode 100644 index 00000000..a6a06da0 --- /dev/null +++ b/sdk/go/connect/broker.go @@ -0,0 +1,122 @@ +package connect + +import ( + "context" + "errors" + + connectrpc "connectrpc.com/connect" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1/rampv1connect" + "github.com/RAMP-Protocol/protocol/sdk/go/core" +) + +// BrokerClient is the Connect client for BrokerService. +// +// It is a SEPARATE constructor rather than a second surface on the exchange +// client because the two speak to different parties. A Broker is not an +// Exchange: it fans a query out across Exchanges it knows and relays back what +// they offered, so its address is the Broker's, not any Exchange's. Hanging both +// off one base URL would mean one of the two was always pointed at the wrong +// party. +// +// It shares the exchange client's plumbing — the same signing transport, the same +// cross-cutting interceptors, the same fail-closed offer Verifier — so the two +// faces cannot drift in how they sign, correlate, validate or verify. +type BrokerClient struct { + rpc rampv1connect.BrokerServiceClient + verifier core.Verifier + // requester is the agent identity a Broker resolves the caller from. Held + // here rather than demanded on every request for the same reason the exchange + // client holds it: one client speaks for one agent. + requester *rampv1.Requester +} + +// NewBrokerClient builds a BrokerClient against a Broker's base URL. It accepts +// the same option type as NewClient, but only the options a discovery call has +// any use for actually do anything, and two of those need care: +// +// - WithOfferKey pins a SINGLE offer-verifying key for every exchange, which is +// the wrong shape here. Broker fan-out returns offers minted by different +// Exchanges, so anything not signed by that one key lands in Rejected. Inject +// WithKeyResolver instead — the resolvers tier ships one that resolves each +// issuing Exchange's own key. +// - WithRequester is REQUIRED, not optional: a Broker resolves the calling agent +// from it and declines a request that names none, so Resolve refuses locally +// rather than spending a round trip to be told. +// +// The options that do nothing here are the ones belonging to legs a Broker client +// does not have: WithAgentKey, WithProofWindow and WithContentTimeout / +// WithMaxContentBytes configure the delivery fetch; WithEndpointResolver and +// WithGuardedBaseTransport configure the offer-derived leg. Both legs live on the +// exchange client. Passing them here is silently inert rather than an error, so +// one shared option set can build both faces. +// +// BrokerService carries exactly one method today. The purchase path through a +// Broker is still a relay route rather than an RPC; when it becomes one, this +// type gains one method and nothing else here changes. +func NewBrokerClient(baseURL string, opts ...ClientOption) *BrokerClient { + cfg := resolvedConfig(opts...) + httpClient, connectOpts, verifier := plumbing(cfg) + return &BrokerClient{ + rpc: rampv1connect.NewBrokerServiceClient(httpClient, baseURL, connectOpts...), + verifier: verifier, + requester: cfg.requester, + } +} + +// Resolve runs discovery through the Broker, which fans out to the Exchanges it +// knows and returns one group per requested URI. +// +// Every returned offer is verified through the SAME fail-closed Verifier +// Discover uses — not a second verification path. Broker-relayed offers are +// precisely the case that rule exists for: the Broker forwards offers it did not +// mint, and an unverified relay can steer an agent's selection with doctored +// terms that only fail later, at the purchase. +// +// A resolve that finds nothing is a SUCCESSFUL answer carrying a typed reason, +// not an error: the whole-call reason lands on DiscoveryResult.AbsenceReason and +// the per-URI ones on each group. Only a genuine fault returns an error. +// +// Resolve carries no idempotency key. Pure discovery buys nothing and changes +// nothing, so there is nothing for a server to deduplicate — the request message +// has no such field. +// +// The request is CLONED before ver and the requester are filled in, so the message +// the caller built stays untouched. Both are filled only when EMPTY: a value the +// caller set is theirs. A Broker resolves the calling agent from requester.id and +// refuses a request that names none, so leaving it to every caller to remember +// would make the identity the client already holds useless exactly where it is +// needed. +func (b *BrokerClient) Resolve(ctx context.Context, req *rampv1.DiscoveryRequest) (core.DiscoveryResult, error) { + const op = "resolve" + if req == nil { + return core.DiscoveryResult{}, malformed(op, errors.New("request is nil")) + } + sent, err := cloneRequest(req, op) + if err != nil { + return core.DiscoveryResult{}, err + } + stampDiscovery(&sent.Ver, &sent.Requester, b.requester) + // Refused locally rather than sent: a Broker resolves the calling agent from + // the requester and declines a request that names none, so this is a verdict + // the client already knows, and naming the remedy beats relaying "requester + // required" from a round trip away. Execute refuses the same way. + if sent.Requester == nil { + return core.DiscoveryResult{}, malformed(op, errors.New( + "no requester configured; a Broker resolves who is asking (see WithRequester)")) + } + resp, err := b.rpc.Resolve(ctx, connectrpc.NewRequest(sent)) + if err != nil { + return core.DiscoveryResult{}, sendError(op, err) + } + msg := resp.Msg + return core.DiscoveryResult{ + Groups: b.verifier.SortGroups(ctx, msg.GetOfferGroups()), + // The raw field, not the getter: an absent optional enum and the + // unspecified value are different answers, and a getter collapses them. + AbsenceReason: msg.AbsenceReason, + // A DiscoveryResponse names no single Exchange and carries no rate-limit + // signal — each offer carries its own issuing domain instead. + }, nil +} diff --git a/sdk/go/connect/callerror.go b/sdk/go/connect/callerror.go new file mode 100644 index 00000000..85f01e04 --- /dev/null +++ b/sdk/go/connect/callerror.go @@ -0,0 +1,155 @@ +package connect + +import ( + "errors" + + connectrpc "connectrpc.com/connect" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/failure" +) + +// A refusal and a failure are different things, and a caller handles four +// situations differently. It cannot tell them apart from a message, so the class +// is carried as a value. +// +// The fourth is the one worth naming: WE refused to send. The routing checks that +// precede a call to an offer-derived address can decline before anything leaves +// the process, and folding that into "unreachable" — as a network timeout — hides +// the difference between "the network is bad" and "the address failed a security +// check". They call for opposite responses. + +// CallErrorKind classifies why a client call did not produce an answer. +type CallErrorKind int + +const ( + // CallUnknown is the zero value; it carries no classification. + CallUnknown CallErrorKind = iota + // CallRefused is a server that answered and said no, with a status and, + // usually, a typed reason. + CallRefused + // CallUnreachable is a server that did not answer: dial failure, timeout, or + // a redirect this SDK refused to follow. + CallUnreachable + // CallNotSent is THIS SDK declining to send. The address failed the + // plain-hostname, same-host or dial-time guard, so nothing left the process + // and no signature was exposed. + CallNotSent + // CallMalformed is a request that could not be built or signed faithfully. + // Nothing left the process. + CallMalformed + // CallTooLarge is a response body past the configured cap. + CallTooLarge + // CallNotSignable is a signature or proof that could not be produced — + // typically custody declining or timing out. Nothing left the process. + CallNotSignable +) + +var callErrorKindNames = map[CallErrorKind]string{ + CallRefused: "refused", + CallUnreachable: "unreachable", + CallNotSent: "not_sent", + CallMalformed: "malformed", + CallTooLarge: "too_large", + CallNotSignable: "not_signable", +} + +// String renders the kind for logging and for the reason a caller sees when the +// peer supplied none. +func (k CallErrorKind) String() string { return failure.Name(callErrorKindNames, k) } + +// CallError is the client's typed failure for the verbs that are not plain +// Connect round trips — the ones that vet an address before sending, or that +// speak HTTP rather than an RPC. +// +// Detail carries the typed protocol reason when there is one. On an RPC path it +// is the ErrorDetail the peer emitted; on the content path it is SYNTHESIZED +// locally from the edge's refusal token, because the edge answers a small JSON +// object rather than a protobuf. ErrorDetailFrom reads both, so a caller branches +// on one vocabulary either way. +type CallError struct { + Kind CallErrorKind + Op string + Status int // HTTP status when the peer answered; 0 otherwise + Reason string // the peer's own refusal token when it sent one + Detail *rampv1.ErrorDetail + Err error +} + +func (e *CallError) Error() string { + return failure.Render("connect", e.Op, e.Kind.String(), e.Status, e.Reason, e.Err) +} + +// Unwrap keeps the cause matchable, so errors.Is still reaches a custody or +// resolver sentinel after the failure has been classified here. +func (e *CallError) Unwrap() error { return e.Err } + +// ReasonOf returns the most specific machine-readable reason available: the +// peer's own token when it sent one, otherwise the failure class. +func (e *CallError) ReasonOf() string { return failure.ReasonOr(e.Reason, e.Kind.String()) } + +// notSent builds the refusal for an address that failed a routing check. It is +// its own constructor because every such refusal must state which check declined +// and must never carry a status: nothing was sent, so there is nothing to report +// a status for. +func notSent(op string, err error) *CallError { + return &CallError{Kind: CallNotSent, Op: op, Err: err} +} + +// malformed builds the refusal for a request that could not be assembled. +func malformed(op string, err error) *CallError { + return &CallError{Kind: CallMalformed, Op: op, Err: err} +} + +// asCallError extracts a *CallError from err's chain. +func asCallError(err error) (*CallError, bool) { + var cerr *CallError + if errors.As(err, &cerr) { + return cerr, true + } + return nil, false +} + +// sendError classifies a failure the transport returned AFTER the routing checks +// passed, so one verb answers with one error type however it failed. +// +// Without this a single method yields a *CallError when it declines to send and a +// bare transport error when the peer refuses, which makes errors.As(&CallError{}) +// a coin flip on the very type callers are told to branch on. +// +// The Connect error is kept in the chain with %w, so errors.As still reaches it +// and ErrorDetailFrom still finds the typed detail the peer attached. +func sendError(op string, err error) error { + out := &CallError{Kind: CallUnreachable, Op: op, Err: err} + var cerr *connectrpc.Error + if !errors.As(err, &cerr) { + return out + } + // A peer that answered is a refusal, whatever it said. The distinction a caller + // needs is "it said no" versus "it never answered", and only the first is worth + // surfacing a reason for. + // + // Three codes land on the second side. Unavailable is the transport failure + // proper. The two context codes are LOCAL outcomes wearing a Connect code: + // connect-go stamps CodeDeadlineExceeded on a context that ran out and + // CodeCanceled on one the caller cancelled, so neither means the peer reached a + // verdict. Classifying a caller's own cancellation as CallRefused would tell it + // the Exchange declined a call the Exchange may never have seen. + switch cerr.Code() { + case connectrpc.CodeUnavailable, + connectrpc.CodeDeadlineExceeded, + connectrpc.CodeCanceled: + out.Kind = CallUnreachable + case connectrpc.CodeResourceExhausted: + // The read cap, seen from this side: the peer's answer was larger than the + // client agreed to read. + out.Kind = CallTooLarge + default: + out.Kind = CallRefused + } + out.Reason = cerr.Code().String() + if detail, ok := errorDetailFromConnect(cerr); ok { + out.Detail = detail + } + return out +} diff --git a/sdk/go/connect/client.go b/sdk/go/connect/client.go index 0a7ccc50..df6f84be 100644 --- a/sdk/go/connect/client.go +++ b/sdk/go/connect/client.go @@ -2,15 +2,20 @@ package connect import ( "context" + "errors" + "fmt" "net/http" "time" connectrpc "connectrpc.com/connect" + "google.golang.org/protobuf/proto" rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1/rampv1connect" "github.com/RAMP-Protocol/protocol/sdk/go/core" "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/failure" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" ) // Client is the L2 low-tier RAMP Connect client: a configurable ExchangeService @@ -22,42 +27,160 @@ import ( type Client struct { rpc rampv1connect.ExchangeServiceClient verifier core.Verifier + cfg clientConfig + + // exchanges caches one client per OFFER-DERIVED Exchange origin. The home + // client above is the configured one; a usage report or a dispute goes to + // whichever Exchange issued the offer, which is discovered at runtime. + exchanges *exchangePool + // endpoints resolves an offer's exchange domain to that Exchange's own + // advertised origin. Never configuration. + endpoints EndpointResolver + // fetcher is the content leg. It dials, so it lives one tier down. + fetcher *resolvers.ContentFetcher } -// NewClient builds a Client against baseURL. The sign face is composed onto the -// HTTP client's transport BEFORE the Connect client is built (Content-Digest needs -// the marshaled body bytes), then the cross-cutting interceptors are wired in the -// ADR order: sign(RoundTripper) · request-id · validate · (app extras). Offer -// verification is Strict by default. -func NewClient(baseURL string, opts ...ClientOption) *Client { +// resolvedConfig applies opts over the defaults every client shares. +func resolvedConfig(opts ...ClientOption) clientConfig { cfg := clientConfig{httpClient: &http.Client{}, mode: core.Strict} for _, o := range opts { o(&cfg) } - httpClient := signedHTTPClient(cfg) - interceptors := clientInterceptors(cfg) - rpc := rampv1connect.NewExchangeServiceClient( - httpClient, baseURL, connectrpc.WithInterceptors(interceptors...), - ) + return cfg +} + +// DefaultMaxRPCReadBytes caps the response body a single RAMP call will read. +// Connect +// treats an unset cap as "any size" and compresses every exchange, so without one +// a hostile or misconfigured peer can decompress an unbounded body into the +// caller's memory. A RAMP response for a realistic batch is small; the bound is +// what stops a peer — including one an offer named — spending the caller's memory +// on its behalf. Override it per client with WithClientOptions. +const DefaultMaxRPCReadBytes = 1 << 20 // 1 MiB + +// plumbing assembles what every client face is built from: the signing HTTP +// client, the Connect options, and the offer Verifier. Extracted so the exchange +// and broker faces cannot drift in how they sign, correlate, validate, verify, or +// bound what they read. +func plumbing(cfg clientConfig) (*http.Client, []connectrpc.ClientOption, core.Verifier) { + // The caller's options come last so an application can tighten (or widen) a + // default the SDK chose. + opts := append([]connectrpc.ClientOption{ + connectrpc.WithInterceptors(clientInterceptors(cfg)...), + connectrpc.WithReadMaxBytes(DefaultMaxRPCReadBytes), + }, cfg.connectOpts...) + return signedHTTPClient(cfg, cfg.httpClient.Transport), + opts, + core.NewVerifier(cfg.mode, cfg.resolveOfferResolver(), time.Now) +} + +// NewClient builds a Client against baseURL — the agent's HOME Exchange, the one +// its account lives on. The sign face is composed onto the HTTP client's +// transport BEFORE the Connect client is built (Content-Digest needs the +// marshaled body bytes), then the cross-cutting interceptors are wired in the +// ADR order: sign(RoundTripper) · request-id · validate · (app extras). Offer +// verification is Strict by default. +// +// Discovery and purchase go to baseURL. A usage report or a dispute does NOT: +// those reach the Exchange that issued the offer, resolved per call from that +// Exchange's own manifest, over a separately guarded transport. +func NewClient(baseURL string, opts ...ClientOption) *Client { + cfg := resolvedConfig(opts...) + httpClient, connectOpts, verifier := plumbing(cfg) return &Client{ - rpc: rpc, - verifier: core.NewVerifier(cfg.mode, cfg.resolveOfferResolver(), time.Now), + rpc: rampv1connect.NewExchangeServiceClient(httpClient, baseURL, connectOpts...), + verifier: verifier, + cfg: cfg, + // A SECOND signing client for the offer-derived leg, over the guarded + // transport: the caller names a domain, the manifest it serves names an + // endpoint, and a signed call then goes there. Without the guard one hop + // down, that is a signed request aimed at an arbitrary internal address. + // Redirects are refused outright — following one would re-sign the call for + // a target the peer chose, after the endpoint check had already passed. And + // it carries its own deadline, because an offer-named Exchange that accepts + // a connection and then never answers would otherwise hold the call, a + // goroutine and a socket open indefinitely. + exchanges: newExchangePool( + offerDerivedClient(cfg, resolvers.NewGuardedTransport(cfg.guardedBase)), connectOpts...), + endpoints: cfg.resolveEndpointResolver(), + fetcher: resolvers.NewContentFetcher(resolvers.ContentFetchOptions{ + BaseTransport: cfg.guardedBase, + Timeout: cfg.fetchTimeout, + MaxBytes: cfg.fetchMaxByte, + // The same mint the RPC legs read, so WithRequestIDFunc reaches all + // three. Without it the delivery fetch is the one leg with no id, and + // an edge that mints its own logs a refusal under a value nothing here + // can join it to. + RequestID: requestIDMint(cfg.requestID), + }), } } // signedHTTPClient returns an *http.Client whose transport is the SDK signing -// RoundTripper wrapping the injected client's base transport (preserving a custom -// proxy/mTLS transport as the base). -func signedHTTPClient(cfg clientConfig) *http.Client { - base := cfg.httpClient.Transport +// RoundTripper wrapping base (preserving a custom proxy/mTLS transport +// underneath). Redirects are refused: a RAMP RPC has no legitimate reason to be +// redirected, and following one would re-sign the caller's request for a target +// the peer chose — which would also move the destination after the endpoint check +// had run. +func signedHTTPClient(cfg clientConfig, base http.RoundTripper) *http.Client { if base == nil { base = http.DefaultTransport } signed := *cfg.httpClient - signed.Transport = core.NewSigningTransport(cfg.signer, base) + signed.Transport = core.NewSigningTransport(cfg.signer, base, signingOptions(cfg)...) + signed.CheckRedirect = refuseRPCRedirect return &signed } +// signingOptions renders the signing knobs the client models onto the transport's +// own option type. It is the ONE place they meet, so every face the SDK builds — +// the home Exchange, the Broker, and the offer-derived pool, which all reach the +// wire through signedHTTPClient — signs on identical terms. +// It ACCUMULATES rather than returning on the first knob it finds. Written as an +// early return for a single option, the second one to arrive is silently dropped +// whenever the first is unset — which is how a client came to sign an empty +// Signature-Agent while the option to fill it already existed a tier down. +func signingOptions(cfg clientConfig) []core.SigningOption { + var opts []core.SigningOption + if cfg.signWindow != nil { + opts = append(opts, core.WithWindow(cfg.signWindow)) + } + if cfg.signatureAgent != "" { + opts = append(opts, core.WithSignatureAgent(cfg.signatureAgent)) + } + return opts +} + +// refuseRPCRedirect stops the client following any 3xx on an RPC leg. Following +// one would re-sign the caller's request for a target the peer chose, after the +// endpoint check had already passed. +var refuseRPCRedirect = failure.RefuseRedirect( + "connect", "a RAMP call is never redirected", helpers.RedactURL) + +// DefaultCallTimeout bounds one call on the offer-derived leg. A RAMP RPC is +// interactive — something is waiting on the other end — so a request that has not +// answered by now is more useful as an error than as a hang. +const DefaultCallTimeout = 30 * time.Second + +// offerDerivedClient is signedHTTPClient plus a deadline. The home Exchange and +// the Broker are operator-configured, so their timeout is the caller's to set +// through WithHTTPClient; an Exchange an offer named is not, and a client with no +// deadline against a host chosen by another party is a hang waiting to happen. +// An explicit timeout on the injected client is respected. +func offerDerivedClient(cfg clientConfig, base http.RoundTripper) *http.Client { + client := signedHTTPClient(cfg, base) + if client.Timeout <= 0 { + client.Timeout = DefaultCallTimeout + } + // The caller's cookie jar does not come along. This leg already drops the + // caller's proxy, TLS dialer and redirect policy, and a jar is the same kind of + // ambient state: http.CookieJar is an interface, so what an arbitrary + // implementation sends to a host an offer named is not this package's to + // assume. A RAMP call carries its identity in the signature, never in a cookie. + client.Jar = nil + return client +} + // clientInterceptors assembles the cross-cutting interceptor stack (request-id · // validate · app extras). Sign is NOT here — it is the RoundTripper. Panics only // if the shared validator fails to build, which is a programmer/config error, not @@ -73,39 +196,113 @@ func clientInterceptors(cfg clientConfig) []connectrpc.Interceptor { return out } -// Discover issues DiscoverResources and returns the fail-closed {verified, -// rejected} split: EVERY returned offer is verified against the exchange -// offer-signing key (resolved through the injected resolver) before it is handed -// back. Neither an unverifiable nor a doctored offer is silently dropped — it lands -// in Rejected with a reason. Round-trip: client sign → HTTP → server verify → -// origin → response, then the offer Verifier over the response. +// Discover issues DiscoverResources and returns one group per requested URI, +// each carrying the fail-closed {verified, rejected} split: EVERY returned offer +// is verified against the exchange offer-signing key (resolved through the +// injected resolver) before it is handed back. Neither an unverifiable nor a +// doctored offer is silently dropped — it lands in Rejected with a reason. A URI +// that the responder GROUPED and left empty keeps its group, carrying the typed +// reason, so a refusal is an answer rather than an absence. (A response carrying +// no groups at all yields none — there is nothing to keep.) Round-trip: client +// sign → HTTP → server verify → origin → response, then the offer Verifier over +// the response. // -// The query is the CALLER's message and is sent unmodified — unlike Execute, -// which builds its own request, Discover cannot stamp fields without mutating -// what it was handed. The caller is therefore the sender for ver purposes and -// MUST set query.Ver = helpers.ProtocolVersion (see "Protocol version" in -// ramp.proto: senders stamp it from one constant, never a literal). -func (c *Client) Discover(ctx context.Context, query *rampv1.ResourceQuery) (core.Result, error) { - resp, err := c.rpc.DiscoverResources(ctx, connectrpc.NewRequest(query)) +// The query is CLONED before ver and the requester are filled in, so the message +// the caller built stays untouched — it crossed a package boundary as an +// argument, not as a buffer. Both fields are filled only when EMPTY: a value the +// caller set is theirs. +func (c *Client) Discover(ctx context.Context, query *rampv1.ResourceQuery) (core.DiscoveryResult, error) { + const op = "discover" + if query == nil { + return core.DiscoveryResult{}, malformed(op, errors.New("query is nil")) + } + sent, err := cloneRequest(query, op) if err != nil { - return core.Result{}, err + return core.DiscoveryResult{}, err + } + stampDiscovery(&sent.Ver, &sent.Requester, c.cfg.requester) + resp, err := c.rpc.DiscoverResources(ctx, connectrpc.NewRequest(sent)) + if err != nil { + return core.DiscoveryResult{}, sendError(op, err) + } + msg := resp.Msg + return core.DiscoveryResult{ + Groups: c.discoveredGroups(ctx, sent, msg), + Exchange: msg.GetExchange(), + RateLimit: msg.GetRateLimit(), + }, nil +} + +// discoveredGroups folds a ResourceResponse's two offer representations into the +// per-URI form. +// +// The message carries a grouped list AND a flat one, and the contract says a +// responder populating groups SHOULD leave the flat list empty "to avoid +// ambiguity" — but a real Exchange populates both, the flat list mirroring the +// grouped offers as a single-URI convenience. So the two are read as ALTERNATIVES, +// never concatenated: concatenating would double every offer against such a +// server, and deduplicating would silently accept a responder whose two lists +// disagree, which is precisely the ambiguity the contract forbids. +// +// Groups win when present. The flat fallback becomes a single group; it carries +// no URI of its own, so it takes the query's only URI when the query named +// exactly one, and none otherwise — the SDK does not invent an attribution the +// wire did not make. +func (c *Client) discoveredGroups(ctx context.Context, query *rampv1.ResourceQuery, msg *rampv1.ResourceResponse) []core.OfferGroupResult { + if groups := msg.GetOfferGroups(); len(groups) > 0 { + return c.verifier.SortGroups(ctx, groups) + } + flat := msg.GetOffers() + if len(flat) == 0 { + return nil } - return c.verifier.Sort(ctx, resp.Msg.GetOffers()), nil + var uri string + if uris := query.GetUris(); len(uris) == 1 { + uri = uris[0] + } + return []core.OfferGroupResult{{URI: uri, Result: c.verifier.Sort(ctx, flat)}} } -// ExecuteOption tunes a single Execute call. -type ExecuteOption func(*executeConfig) +// CallOption tunes a single state-mutating call. +type CallOption func(*callConfig) + +// ExecuteOption is the original name for CallOption, kept because the option set +// is identical across execute, report and dispute — the three RPCs the protocol +// requires an idempotency key on. +type ExecuteOption = CallOption -type executeConfig struct { +type callConfig struct { idempotencyKey string } -// WithIdempotencyKey pins the idempotency key for this Execute call. Reusing a key -// makes the call a deliberate replay: the server dedupes on it (a fresh key is -// minted per call by default). The SDK never tracks keys — the server owns dedup +// WithIdempotencyKey pins the idempotency key for this call. Reusing a key makes +// the call a deliberate replay: the server dedupes on it (a fresh key is minted +// per call by default). The SDK never tracks keys — the server owns dedup // (ADR-019 §4, ADR-020 §3). -func WithIdempotencyKey(key string) ExecuteOption { - return func(e *executeConfig) { e.idempotencyKey = key } +// +// Hold the key and pass the same one back when retrying, on every verb that takes +// this option. The key identifies the ACTION, not the attempt: a fresh key on a +// retry reads to the server as a second purchase, a second report, a second +// dispute. +func WithIdempotencyKey(key string) CallOption { + return func(e *callConfig) { e.idempotencyKey = key } +} + +// idempotencyKeyFor resolves the key for one call, in precedence order: the key +// pinned for this call, then whatever the caller already put on the message, then +// a freshly minted one. +func idempotencyKeyFor(opts []CallOption, onMessage string) (string, error) { + var cc callConfig + for _, o := range opts { + o(&cc) + } + if cc.idempotencyKey != "" { + return cc.idempotencyKey, nil + } + if onMessage != "" { + return onMessage, nil + } + return helpers.NewIdempotencyKey() } // Execute commits to a VERIFIED offer and returns the transaction response. It @@ -114,34 +311,115 @@ func WithIdempotencyKey(key string) ExecuteOption { // key is minted fresh unless WithIdempotencyKey pins one. Execute builds the whole // TransactionRequest, so it also stamps ver from helpers.ProtocolVersion — the // caller neither supplies nor overrides it. -func (c *Client) Execute(ctx context.Context, offer core.VerifiedOffer, opts ...ExecuteOption) (*rampv1.TransactionResponse, error) { - var ec executeConfig - for _, o := range opts { - o(&ec) +func (c *Client) Execute(ctx context.Context, offer core.VerifiedOffer, opts ...CallOption) (*rampv1.TransactionResponse, error) { + const op = "execute" + if c.cfg.requester == nil { + return nil, malformed(op, errors.New( + "no requester configured; an Exchange resolves who is buying from it (see WithRequester)")) } - key := ec.idempotencyKey - if key == "" { - minted, err := helpers.NewIdempotencyKey() - if err != nil { - return nil, err - } - key = minted + signer := c.cfg.signer + if signer == nil { + // CallNotSignable, matching what Fetch answers for the same missing + // holder: a caller branching on the kind sees one condition under one + // class, whichever verb met it first. + return nil, &CallError{Kind: CallNotSignable, Op: op, Err: errors.New( + "no signer configured; a purchase carries a detached acceptance signed with the agent's own key (see WithSigner)")} + } + // An acceptance floating free of a concrete offer is meaningless, and an + // unsigned offer is reachable here: WithVerification(Off) and + // RejectedOffer.Unsafe() both mint a VerifiedOffer without a signature check. + if offer.Offer().GetSignature() == "" { + return nil, malformed(op, errors.New("cannot accept an unsigned offer")) + } + key, err := idempotencyKeyFor(opts, "") + if err != nil { + return nil, malformed(op, err) + } + // The acceptance covers the offer, the requester, and the idempotency key, so + // a retry that pins the same key reproduces byte-identical acceptance bytes. + // That is the deliberate-replay semantic, not an accident. + acceptance, err := helpers.SignOfferAcceptanceWith(ctx, signer, offer.Offer(), c.cfg.requester, key) + if err != nil { + return nil, &CallError{Kind: CallNotSignable, Op: op, Err: err} } - // Items-only wire shape: a single offer is the degenerate - // 1-element items list, each item reflecting its signed Offer back exactly as - // received at discovery. The authoritative identity is the reflected offer; the - // optional top-level offer_id correlation scalar is left unset. + // Items-only wire shape: a single offer is the degenerate 1-element items + // list, each item reflecting its signed Offer back exactly as received at + // discovery. The authoritative identity is the reflected offer; the optional + // top-level offer_id correlation scalar is left unset. // // ver comes from helpers.ProtocolVersion — the single owner of the protocol // version across all three SDKs — never a literal, so a bump is one edit. req := &rampv1.TransactionRequest{ Ver: helpers.ProtocolVersion, IdempotencyKey: key, - Items: []*rampv1.TransactionItem{{Offer: offer.Offer()}}, + Requester: c.cfg.requester, + Items: []*rampv1.TransactionItem{{ + Offer: offer.Offer(), + AgentAcceptance: &rampv1.AgentAcceptance{ + Signature: acceptance, + SignatureAlgorithm: helpers.AcceptanceSignatureAlgorithm, + }, + }}, } resp, err := c.rpc.ExecuteTransaction(ctx, connectrpc.NewRequest(req)) if err != nil { - return nil, err + return nil, sendError(op, err) } return resp.Msg, nil } + +// stampEnvelope fills the two envelope fields the protocol requires on a +// state-mutating call, WITHOUT overwriting what the caller already set. +// +// Fill-when-empty is the whole rule. `ver` has a single owner, so the SDK supplies +// it rather than making every caller reach for the constant. The idempotency key +// is REQUIRED and identifies the action rather than the attempt, so a value the +// caller put there is theirs — discarding it would turn each of their retries into +// a fresh action, which is the double-counting the field exists to prevent. +// WithIdempotencyKey overrides both. +func stampEnvelope(ver, idempotencyKey *string, opts []CallOption) error { + if *ver == "" { + *ver = helpers.ProtocolVersion + } + key, err := idempotencyKeyFor(opts, *idempotencyKey) + if err != nil { + return err + } + *idempotencyKey = key + return nil +} + +// stampDiscovery fills the envelope a DISCOVERY call carries, which is the +// mutating envelope minus the idempotency key: pure discovery buys nothing and +// changes nothing, so there is no action for a key to identify. +// +// Both fills are only-when-empty. The caller's own value always wins — the +// message crossed a package boundary as an argument, not as a buffer to fill in — +// and the requester is filled because both reference services resolve the calling +// agent from it and refuse a request that names none, while the client already +// holds that identity. +func stampDiscovery(ver *string, requester **rampv1.Requester, configured *rampv1.Requester) { + if *ver == "" { + *ver = helpers.ProtocolVersion + } + if *requester == nil { + *requester = configured + } +} + +// cloneRequest copies a caller's message so the SDK can stamp its envelope +// without touching what the caller still holds. +// +// The type assertion cannot fail for a concrete message — proto.Clone returns the +// same dynamic type it was given — but it is checked rather than asserted blind, +// because a silent nil would reach the wire as an empty request. One helper +// rather than four copies of the same three lines. +func cloneRequest[T proto.Message](msg T, op string) (T, error) { + cloned, ok := proto.Clone(msg).(T) + if !ok { + var zero T + return zero, malformed(op, fmt.Errorf( + "cloned %T has the wrong type", msg)) + } + return cloned, nil +} diff --git a/sdk/go/connect/client_verify_test.go b/sdk/go/connect/client_verify_test.go index b39ab520..0334c6e0 100644 --- a/sdk/go/connect/client_verify_test.go +++ b/sdk/go/connect/client_verify_test.go @@ -44,6 +44,11 @@ type signingFixture struct { signer helpers.Signer resolver *helpers.StaticKeyResolver keyID string + // pub is the public half of the same key. The protocol carries ONE agent + // identity: the key that signs requests also signs the detached acceptance and + // is the key a delivery URL is bound to, so tests that verify an acceptance or + // present a fetch proof need it alongside the Signer. + pub ed25519.PublicKey } // newSigningFixture builds a request-signing keypair, an L1 Signer over it, and a @@ -55,7 +60,13 @@ func newSigningFixture(t *testing.T) signingFixture { if err != nil { t.Fatalf("generate request-signing key: %v", err) } - const keyID = "agent.test.v1" + // keyid IS the RFC 7638 thumbprint: it is the anchor of the three-way identity + // an edge checks on a bound fetch, and the value an Exchange binds a delivery + // URL to. A fixture that used an opaque label could never mint a fetch proof. + keyID, err := helpers.Thumbprint(pub) + if err != nil { + t.Fatalf("thumbprint request-signing key: %v", err) + } signer, err := helpers.NewEd25519Signer(keyID, priv) if err != nil { t.Fatalf("new signer: %v", err) @@ -64,6 +75,7 @@ func newSigningFixture(t *testing.T) signingFixture { signer: signer, resolver: helpers.NewStaticKeyResolver(map[string]ed25519.PublicKey{keyID: pub}), keyID: keyID, + pub: pub, } } @@ -215,7 +227,7 @@ func TestClientSign_RoundTripsThroughServerVerify(t *testing.T) { replay := newMemReplayStore() srv := newVerifyingServer(t, sig, replay, nil) - client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(sig.signer)) + client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester())) // Execute needs a VerifiedOffer; but this test only asserts transport // acceptance, so a discover round-trip (empty offer set) is the minimal @@ -263,7 +275,7 @@ func TestDiscover_SortsVerifiedAndRejected(t *testing.T) { srv := newVerifyingServer(t, sig, replay, []*rampv1.Offer{off.good, off.doctored}) client := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithOfferKey(off.exchangePub), // exchange offer-verifying key ) @@ -271,23 +283,23 @@ func TestDiscover_SortsVerifiedAndRejected(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("want exactly 1 verified offer, got %d", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("want exactly 1 verified offer, got %d", len(res.Verified())) } - if got := res.Verified[0].Offer().GetOfferId(); got != "offer-good" { + if got := res.Verified()[0].Offer().GetOfferId(); got != "offer-good" { t.Fatalf("verified offer id: want offer-good, got %q", got) } - if len(res.Rejected) != 1 { - t.Fatalf("want exactly 1 rejected offer, got %d", len(res.Rejected)) + if len(res.Rejected()) != 1 { + t.Fatalf("want exactly 1 rejected offer, got %d", len(res.Rejected())) } - if got := res.Rejected[0].Offer.GetOfferId(); got != "offer-doctored" { + if got := res.Rejected()[0].Offer.GetOfferId(); got != "offer-doctored" { t.Fatalf("rejected offer id: want offer-doctored, got %q", got) } - if res.Rejected[0].Reason == nil { + if res.Rejected()[0].Reason == nil { t.Fatal("rejected offer must carry a non-nil reason") } - if !errors.Is(res.Rejected[0].Reason, helpers.ErrOfferSignatureInvalid) { - t.Fatalf("rejected reason: want ErrOfferSignatureInvalid, got %v", res.Rejected[0].Reason) + if !errors.Is(res.Rejected()[0].Reason, helpers.ErrOfferSignatureInvalid) { + t.Fatalf("rejected reason: want ErrOfferSignatureInvalid, got %v", res.Rejected()[0].Reason) } } @@ -302,7 +314,7 @@ func TestExecute_AcceptsVerifiedOffer(t *testing.T) { srv := newVerifyingServer(t, sig, replay, []*rampv1.Offer{off.good}) client := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithOfferKey(off.exchangePub), ) @@ -310,13 +322,13 @@ func TestExecute_AcceptsVerifiedOffer(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("want 1 verified offer, got %d", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("want 1 verified offer, got %d", len(res.Verified())) } - // Execute takes ONLY a VerifiedOffer — this compiles because res.Verified[0] - // is one. Passing res.Rejected[0] (a RejectedOffer) or a raw *rampv1.Offer + // Execute takes ONLY a VerifiedOffer — this compiles because res.Verified()[0] + // is one. Passing res.Rejected()[0] (a RejectedOffer) or a raw *rampv1.Offer // here would NOT compile; that guard is documented in doc_compileguard_test.go. - if _, err := client.Execute(context.Background(), res.Verified[0]); err != nil { + if _, err := client.Execute(context.Background(), res.Verified()[0]); err != nil { t.Fatalf("Execute on a verified offer must succeed, got: %v", err) } } @@ -336,7 +348,7 @@ func TestExecute_StampsProtocolVersion(t *testing.T) { srv, origin := newVerifyingServerStub(t, sig, newMemReplayStore(), []*rampv1.Offer{off.good}) client := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithOfferKey(off.exchangePub), ) @@ -347,10 +359,10 @@ func TestExecute_StampsProtocolVersion(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("want 1 verified offer, got %d", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("want 1 verified offer, got %d", len(res.Verified())) } - if _, err := client.Execute(context.Background(), res.Verified[0]); err != nil { + if _, err := client.Execute(context.Background(), res.Verified()[0]); err != nil { t.Fatalf("Execute: %v", err) } @@ -380,7 +392,7 @@ func TestRejectedOffer_RequiresUnsafeToExecute(t *testing.T) { srv := newVerifyingServer(t, sig, replay, []*rampv1.Offer{off.doctored}) client := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithOfferKey(off.exchangePub), ) @@ -388,13 +400,13 @@ func TestRejectedOffer_RequiresUnsafeToExecute(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Rejected) != 1 { - t.Fatalf("want 1 rejected offer, got %d", len(res.Rejected)) + if len(res.Rejected()) != 1 { + t.Fatalf("want 1 rejected offer, got %d", len(res.Rejected())) } // The explicit escape: .Unsafe() converts a RejectedOffer into the executable // VerifiedOffer shape. This is the single documented bypass; without it // Execute(ctx, rejected) does not compile. - forced := res.Rejected[0].Unsafe() + forced := res.Rejected()[0].Unsafe() if _, err := client.Execute(context.Background(), forced); err != nil { t.Fatalf("Execute on an explicitly-unsafed offer must reach the server, got: %v", err) } @@ -416,17 +428,17 @@ func TestWithVerification_StrictRejectsUnverifiable(t *testing.T) { // No WithOfferKey → the client cannot resolve the exchange offer key, so even // the genuinely-signed offer is UNVERIFIABLE and must be rejected under Strict. - client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(sig.signer)) + client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester())) res, err := client.Discover(context.Background(), &rampv1.ResourceQuery{}) if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 0 { - t.Fatalf("Strict with no resolvable offer key must verify nothing, got %d verified", len(res.Verified)) + if len(res.Verified()) != 0 { + t.Fatalf("Strict with no resolvable offer key must verify nothing, got %d verified", len(res.Verified())) } - if len(res.Rejected) != 1 { - t.Fatalf("Strict must surface the unverifiable offer as rejected, got %d rejected", len(res.Rejected)) + if len(res.Rejected()) != 1 { + t.Fatalf("Strict must surface the unverifiable offer as rejected, got %d rejected", len(res.Rejected())) } } @@ -442,7 +454,7 @@ func TestWithVerification_OffSurfacesUnverified(t *testing.T) { srv := newVerifyingServer(t, sig, replay, []*rampv1.Offer{off.good}) client := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithVerification(core.Off), // loud, named opt-out ) @@ -450,10 +462,10 @@ func TestWithVerification_OffSurfacesUnverified(t *testing.T) { if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("WithVerification(Off) must surface the offer without verification, got %d verified", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("WithVerification(Off) must surface the offer without verification, got %d verified", len(res.Verified())) } - if len(res.Rejected) != 0 { - t.Fatalf("WithVerification(Off) must not reject, got %d rejected", len(res.Rejected)) + if len(res.Rejected()) != 0 { + t.Fatalf("WithVerification(Off) must not reject, got %d rejected", len(res.Rejected())) } } diff --git a/sdk/go/connect/doc.go b/sdk/go/connect/doc.go index f6a7c2f5..0315ec0c 100644 --- a/sdk/go/connect/doc.go +++ b/sdk/go/connect/doc.go @@ -1,13 +1,25 @@ // Package connect is the opt-in Connect-Go CLIENT binding over the transport-neutral -// sdk/go/core L2 substance: the configurable Connect client (NewClient — sign face -// as a signing RoundTripper, request-id/validate interceptors, the fail-closed offer -// Verifier), the shared bidirectional protovalidate interceptor -// (NewValidateInterceptor, the single definition the server binding also composes), -// and the READ direction of the ADR-019 ErrorDetail↔Connect bridge (ErrorDetailFrom -// — a client reads the typed error detail an upstream emitted). It depends -// one-directionally on core (Verifier, Result, VerifiedOffer guard, signing -// transport, ReplayStore) and imports connectrpc; core and helpers stay Connect-free -// (ADR-020 §2/§3). +// sdk/go/core L2 substance. It carries the agent verb set — Discover, Resolve, +// Execute, ReportUsage, Dispute and the low-tier content Fetch — across two +// constructors: NewClient for an Exchange and NewBrokerClient for a Broker, which +// are different parties and so cannot share one base URL. +// +// Around those sit the configurable client itself (sign face as a signing +// RoundTripper, request-id/validate interceptors, the fail-closed offer Verifier), +// the routing tier that resolves an offer's Exchange from that Exchange's own +// manifest and vets it before anything signed is sent, the CallError taxonomy that +// tells a refusal from a failure from a local decline, the shared bidirectional +// protovalidate interceptor (NewValidateInterceptor, the single definition the +// server binding also composes), and the READ direction of the ADR-019 +// ErrorDetail↔Connect bridge (ErrorDetailFrom — a client reads the typed error +// detail an upstream emitted, whether a peer sent it or the content leg +// synthesized it). +// +// It depends one-directionally on core (Verifier, DiscoveryResult, VerifiedOffer +// guard, signing transport, ReplayStore) and on resolvers for the one thing this +// package must not do itself — dial. The content fetch and the offer-derived RPC +// leg both run on the resolvers tier's guarded transport; core and helpers stay +// Connect-free (ADR-020 §2/§3). // // The SERVER binding is a SEPARATE package, sdk/go/connectserver // (NewExchangeServiceHandler + the verify http-seam + AsConnectError / reject→code). diff --git a/sdk/go/connect/errordetail.go b/sdk/go/connect/errordetail.go index c20495a4..2b19741e 100644 --- a/sdk/go/connect/errordetail.go +++ b/sdk/go/connect/errordetail.go @@ -16,10 +16,26 @@ import ( // already-extracted detail) stays in sdk/go/helpers; the emit direction // (AsConnectError) lives in the server binding sdk/go/connectserver. func ErrorDetailFrom(err error) (*rampv1.ErrorDetail, bool) { + // The SDK's own typed failure is checked first, so ONE accessor serves every + // verb. On an RPC path the detail below was emitted by the peer; on the + // content path it was synthesized locally from the edge's refusal token, + // because a delivery edge answers a small JSON object rather than a protobuf. + // What a synthesized detail names as its domain, and why, is recorded once on + // edgeErrorDomain rather than restated here. + if callErr, ok := asCallError(err); ok && callErr.Detail != nil { + return callErr.Detail, true + } var cerr *connectrpc.Error if !errors.As(err, &cerr) { return nil, false } + return errorDetailFromConnect(cerr) +} + +// errorDetailFromConnect reads the first RAMP ErrorDetail off an already-unwrapped +// Connect error. Split out so the CallError bridge can reuse the one extraction +// rather than restating it. +func errorDetailFromConnect(cerr *connectrpc.Error) (*rampv1.ErrorDetail, bool) { for _, d := range cerr.Details() { msg, verr := d.Value() if verr != nil { diff --git a/sdk/go/connect/guards_test.go b/sdk/go/connect/guards_test.go new file mode 100644 index 00000000..e1ed214e --- /dev/null +++ b/sdk/go/connect/guards_test.go @@ -0,0 +1,291 @@ +package connect_test + +import ( + "context" + "crypto/tls" + "errors" + "net" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + + "google.golang.org/protobuf/proto" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + rampconnect "github.com/RAMP-Protocol/protocol/sdk/go/connect" +) + +// The transport guarantees the offer-derived leg claims, driven end to end. +// +// These sit apart from the verb tests because the verb tests deliberately opt out +// of the production dial posture to reach a loopback server. The point here is the +// opposite: leave the guard installed and prove it bites. + +// fixedEndpoint is an EndpointResolver that answers with one endpoint, so a test +// can drive the dial without standing up a manifest server. It is the seam's +// stated purpose. +type fixedEndpoint struct{ endpoint string } + +func (f fixedEndpoint) ResolveEndpoint(_ context.Context, _ string) (string, error) { + return f.endpoint, nil +} + +// A private address is refused at DIAL time even when it passes every routing +// check — the endpoint is anchored to the domain that advertised it, so the +// same-host rule is satisfied and only the address guard stands between a signed +// report and a host inside the caller's own network. +// +// The guard is deliberately NOT disabled here: an option cannot remove it, and +// this test is what proves the leg is guarded at all. +func TestReportUsage_GuardRefusesAPrivateEndpoint(t *testing.T) { + sig := newSigningFixture(t) + client := rampconnect.NewClient("https://home.invalid", + rampconnect.WithSigner(sig.signer), + // Anchored to the domain, so the check passes: localhost:1 advertises + // localhost:1. The PORT is named on both sides deliberately — anchoring + // compares it, so an endpoint on a port the exchange value does not carry + // would be refused by the routing check and this test would prove nothing + // about the guard. Only the dial-time address guard can refuse this. + rampconnect.WithEndpointResolver(fixedEndpoint{endpoint: "https://localhost:1"}), + ) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String("localhost:1"), + TransactionId: "txn-1", + }) + if err == nil { + t.Fatal("a signed report to a private address must be refused") + } + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if !strings.Contains(err.Error(), "SSRF guard") { + t.Errorf("error = %v, want the dial-time address guard to be the refusal", err) + } +} + +// A caller-supplied transport goes UNDER the guard, never in place of it. Passing +// one must not reopen the address guard — that coupling is the whole reason the +// seam takes a base rather than a whole round-tripper. +// +// The TLS-dialer case is the one that matters: net/http prefers a transport's own +// TLS dialer over DialContext on https, which is every RAMP leg, so a base +// carrying one would take the dial past the address pin entirely. An empty base +// cannot express that, which is why it alone proved less than it appeared to. +func TestReportUsage_GuardSurvivesACallerSuppliedTransport(t *testing.T) { + bases := map[string]*http.Transport{ + "empty": {}, + "custom TLS dialer": { + DialTLSContext: func(_ context.Context, network, addr string) (net.Conn, error) { + return tls.Dial(network, addr, &tls.Config{InsecureSkipVerify: true}) //nolint:gosec // the point is that this dial must never happen + }, + }, + "legacy TLS dialer": { + DialTLS: func(network, addr string) (net.Conn, error) { + return tls.Dial(network, addr, &tls.Config{InsecureSkipVerify: true}) //nolint:gosec // the point is that this dial must never happen + }, + }, + } + for name, base := range bases { + t.Run(name, func(t *testing.T) { + sig := newSigningFixture(t) + client := rampconnect.NewClient("https://home.invalid", + rampconnect.WithSigner(sig.signer), + rampconnect.WithGuardedBaseTransport(base), + rampconnect.WithEndpointResolver(fixedEndpoint{endpoint: "https://localhost:1"}), + ) + + // Port named on both sides so the routing check passes and the dial is + // actually attempted; see the sibling test above. + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String("localhost:1"), + TransactionId: "txn-1", + }) + if err == nil || !strings.Contains(err.Error(), "SSRF guard") { + t.Fatalf("error = %v, want the guard still installed under the injected transport", err) + } + }) + } +} + +// The content fetch is the leg the SDK's own contract names — it "dials only +// through the SSRF-guarded client" — and it shares the same base-transport seam, +// so it inherits the same bypass. A delivery URL names a host another party +// chose, and the request carries a live proof of possession of the agent key, so +// this is the leg where a bypass costs the most. +func TestFetch_GuardSurvivesACallerSuppliedTLSDialer(t *testing.T) { + sig := newSigningFixture(t) + content := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("internal bytes")) + })) + defer content.Close() + tlsCfg := content.Client().Transport.(*http.Transport).TLSClientConfig + + client := rampconnect.NewClient("https://home.invalid", + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + rampconnect.WithGuardedBaseTransport(&http.Transport{ + TLSClientConfig: tlsCfg, + DialTLSContext: func(_ context.Context, network, addr string) (net.Conn, error) { + return tls.Dial(network, addr, tlsCfg) + }, + }), + ) + + _, err := client.Fetch(context.Background(), content.URL+"/doc") + if err == nil { + t.Fatal("a bound fetch reached a loopback delivery host — the dial guard was bypassed") + } + if !strings.Contains(err.Error(), "SSRF guard") { + t.Errorf("error = %v, want the dial-time address guard to be the refusal", err) + } +} + +// The client re-checks the endpoint an INJECTED resolver hands back, and it +// applies the whole rule — not the half of it that is about hosts. +// +// Credentials in the authority are the half that is easy to lose: the host +// comparison reads the host and ignores any user:password before it, so an +// endpoint carrying them passes an anchoring check and then has net/http stamp an +// Authorization header the SDK never chose, on a leg that already carries the +// agent's own signature. The resolver refuses this; so must the client, because +// the resolver is a seam a caller can replace. +func TestReportUsage_RefusesAnInjectedEndpointCarryingUserinfo(t *testing.T) { + sig := newSigningFixture(t) + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + // Anchored to the domain, so only the userinfo arm can refuse it. + rampconnect.WithEndpointResolver(fixedEndpoint{ + endpoint: "http://agent:s3cret@exchange.test", + }), + )...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String("exchange.test"), + TransactionId: "txn-1", + }) + if err == nil { + t.Fatal("a signed report to an endpoint carrying credentials must be refused") + } + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallNotSent { + t.Errorf("kind = %v, want CallNotSent — nothing left the process", cerr.Kind) + } + if !strings.Contains(err.Error(), "userinfo") { + t.Errorf("error = %v, want it to name the credential as the reason", err) + } + // The refusal must not echo the credential it refused. + if strings.Contains(err.Error(), "s3cret") { + t.Errorf("the refusal leaked the credential: %v", err) + } +} + +// A RAMP call is never legitimately redirected. Following one would re-sign the +// caller's request for a target the peer chose — after the endpoint check had +// already passed, which is the window that check exists to close. +func TestReportUsage_RefusesRedirectAndNeverContactsTheTarget(t *testing.T) { + sig := newSigningFixture(t) + + var targetHits atomic.Int64 + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + targetHits.Add(1) + })) + defer target.Close() + + // The Exchange answers the RPC with a redirect to somewhere else. + domain, _ := loopbackManifestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/stolen", http.StatusFound) + })) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + }) + if err == nil { + t.Fatal("expected the redirect to be refused") + } + if !strings.Contains(err.Error(), "never redirected") { + t.Errorf("error = %v, want the redirect refusal", err) + } + if n := targetHits.Load(); n != 0 { + t.Errorf("redirect target contacted %d times; a signed call must never follow one", n) + } +} + +// A peer that answers and says no is a REFUSAL, and it reaches the caller as the +// same typed error the pre-send checks produce — one verb, one error type, +// however it failed. Before this the same method returned two unrelated types +// depending on where it failed. +func TestReportUsage_PeerRefusalIsATypedCallError(t *testing.T) { + sig := newSigningFixture(t) + domain, _ := loopbackManifestServer(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"code":"permission_denied","message":"no"}`)) + })) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + }) + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError on the send path too", err) + } + if cerr.Kind != rampconnect.CallRefused { + t.Errorf("kind = %v, want CallRefused for a peer that answered", cerr.Kind) + } + if cerr.ReasonOf() == "" { + t.Error("a refusal must carry a machine-readable reason") + } +} + +// The accessors a caller is told to branch on, exercised directly: they are the +// public face of the type and were entirely uncovered. +func TestCallError_Accessors(t *testing.T) { + cause := errors.New("underlying") + full := &rampconnect.CallError{ + Kind: rampconnect.CallRefused, Op: "report usage", + Status: http.StatusForbidden, Reason: "pop_expired", Err: cause, + } + msg := full.Error() + for _, want := range []string{"report usage", "refused", "Forbidden", "pop_expired", "underlying"} { + if !strings.Contains(msg, want) { + t.Errorf("Error() = %q, want it to mention %q", msg, want) + } + } + if !errors.Is(full, cause) { + t.Error("Unwrap must keep the cause reachable") + } + if got := full.ReasonOf(); got != "pop_expired" { + t.Errorf("ReasonOf() = %q, want the peer's own token", got) + } + + // With no token from the peer, the class the SDK owns is the fallback. + bare := &rampconnect.CallError{Kind: rampconnect.CallNotSent, Op: "dispute"} + if got := bare.ReasonOf(); got != "not_sent" { + t.Errorf("ReasonOf() = %q, want the failure class as the fallback", got) + } + if got := rampconnect.CallErrorKind(99).String(); got != "unknown" { + t.Errorf("an unnamed kind renders as %q, want \"unknown\"", got) + } + // A status net/http does not know renders as the bare number rather than a + // truncated-looking "(HTTP 599 )". + odd := &rampconnect.CallError{Kind: rampconnect.CallRefused, Op: "fetch", Status: 599} + if !strings.Contains(odd.Error(), "(HTTP 599)") { + t.Errorf("Error() = %q, want the bare status number", odd.Error()) + } +} diff --git a/sdk/go/connect/interceptors.go b/sdk/go/connect/interceptors.go index 9910184f..09256690 100644 --- a/sdk/go/connect/interceptors.go +++ b/sdk/go/connect/interceptors.go @@ -19,11 +19,23 @@ type requestIDInterceptor struct { mint core.RequestIDFunc } -func newRequestIDInterceptor(mint core.RequestIDFunc) connectrpc.Interceptor { +// requestIDMint resolves the correlation-id source once, so every leg of a client +// reads the same one. +// +// It is a named function rather than a nil check at each site because the legs do +// not share a mechanism: the RPC legs correlate through the interceptor below, and +// the delivery fetch is a plain GET that never reaches an interceptor and takes its +// own hook. Two nil checks are two places for the default to drift, and the leg +// that would drift silently is the one carrying no id at all. +func requestIDMint(mint core.RequestIDFunc) core.RequestIDFunc { if mint == nil { - mint = core.DefaultRequestID + return core.DefaultRequestID } - return &requestIDInterceptor{mint: mint} + return mint +} + +func newRequestIDInterceptor(mint core.RequestIDFunc) connectrpc.Interceptor { + return &requestIDInterceptor{mint: requestIDMint(mint)} } func (i *requestIDInterceptor) WrapUnary(next connectrpc.UnaryFunc) connectrpc.UnaryFunc { diff --git a/sdk/go/connect/options.go b/sdk/go/connect/options.go index 4213fa16..c31846df 100644 --- a/sdk/go/connect/options.go +++ b/sdk/go/connect/options.go @@ -4,11 +4,15 @@ import ( "context" "crypto/ed25519" "net/http" + "time" connectrpc "connectrpc.com/connect" + "google.golang.org/protobuf/proto" + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" "github.com/RAMP-Protocol/protocol/sdk/go/core" "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" ) // clientConfig is the resolved set of injected holders a Client is built from. @@ -24,6 +28,17 @@ type clientConfig struct { validation Validation requestID core.RequestIDFunc extra []connectrpc.Interceptor + + requester *rampv1.Requester + agentKey ed25519.PublicKey + proofWindow core.Window + signWindow core.Window + signatureAgent string + endpoints EndpointResolver + guardedBase *http.Transport + connectOpts []connectrpc.ClientOption + fetchTimeout time.Duration + fetchMaxByte int64 } // ClientOption configures a Client. Options are the ONLY way to inject the @@ -88,6 +103,181 @@ func WithInterceptors(is ...connectrpc.Interceptor) ClientOption { return func(c *clientConfig) { c.extra = append(c.extra, is...) } } +// WithRequester injects the agent's own identity, forwarded on a purchase for +// authorization and audit and covered by the detached offer acceptance. +// +// It is client-level rather than per-call on purpose: the requester IS the +// identity the injected Signer already fixes for the transport signature, so a +// per-call requester would let the two disagree about who is buying — the exact +// ambiguity the acceptance exists to remove. A verifying Broker refuses a +// requester id that does not normalise to the signer's own directory host. +// +// The message is cloned here, so a later mutation by the caller cannot reach a +// request already in flight. +func WithRequester(r *rampv1.Requester) ClientOption { + return func(c *clientConfig) { + if r == nil { + c.requester = nil + return + } + cloned, ok := proto.Clone(r).(*rampv1.Requester) + if !ok { + // Unreachable for a concrete message, but the fallback must be + // nil rather than whatever was configured before: keeping a stale + // identity would send one agent's requester on another's behalf, + // where nil fails closed into "no requester configured". + c.requester = nil + return + } + c.requester = cloned + } +} + +// WithAgentKey injects the PUBLIC half of the key WithSigner signs with. A +// delivery fetch presents it in a header, and a Signer cannot yield it — custody +// keeps the private half, so the public half has to be supplied alongside. +// Without it the client can buy but cannot fetch what it bought. +// +// There is deliberately no option for a SEPARATE acceptance key. The protocol +// carries one agent identity: agent_identity_hash is defined as the thumbprint of +// the agent's request-signing key, an Exchange verifies the detached acceptance +// against the key registered for the caller its request signature identified, and +// the delivery URL is bound to that same thumbprint. A second key would be +// refused at execute, and any URL it did produce could never be fetched — the +// presented key would not match the binding. +// The key is COPIED for the same reason WithRequester clones: the caller keeps +// the slice it passed, and a later append or overwrite there would otherwise +// change which key every subsequent fetch presents. +func WithAgentKey(pub ed25519.PublicKey) ClientOption { + return func(c *clientConfig) { + if pub == nil { + c.agentKey = nil + return + } + c.agentKey = append(ed25519.PublicKey(nil), pub...) + } +} + +// WithProofWindow overrides the freshness window stamped on a delivery-fetch +// proof. The default is 30 seconds from the wall clock. +// +// Deliberately NOT the signed URL's own expiry, which can be hours: the proof +// covers only the method and the URL, so anyone who observes the request can +// repeat it until the window closes. +func WithProofWindow(w core.Window) ClientOption { + return func(c *clientConfig) { c.proofWindow = w } +} + +// WithSignWindow overrides the freshness window stamped on every outbound RFC +// 9421 REQUEST signature — the home Exchange, the Broker, and the leg that routes +// to the Exchange an offer named. The default is five minutes from the wall clock. +// +// Two reasons an application supplies its own. A deployment with a shorter +// freshness policy sets its own TTL, and through this option the value it already +// reads from configuration keeps meaning something. And a peer that screens +// replays on (key id, signature) refuses a repeat: signature timestamps have +// one-second resolution, so two identical requests inside one second sign to the +// same bytes. core.MonotonicWindow keeps each signature unique for exactly that. +// +// Distinct from WithProofWindow, which stamps a delivery-fetch proof rather than a +// request signature. Both take a core.Window; neither substitutes for the other. +func WithSignWindow(w core.Window) ClientOption { + return func(c *clientConfig) { c.signWindow = w } +} + +// WithSignatureAgent names the WBA directory origin this client signs as — the +// place a peer fetches to find the key that signed the request. It is stamped into +// the Signature-Agent header of every outbound RFC 9421 request. +// +// Leaving it unset does not omit the header: signature-agent is one of the five +// REQUIRED covered components, so the signature covers it either way and an unset +// client signs an EMPTY value. A peer that resolves the caller's key from that +// origin then has nothing to resolve, and refuses the call at verification — after +// the request was routed, signed and sent, which is why the symptom is a 401 from +// a healthy Exchange rather than anything the routing checks would catch. +// +// One value per client, because one client speaks for one agent — the same reason +// WithRequester is held rather than passed per call. An application signing as +// several agents builds a client per agent. +// +// Stamped SET-IF-ABSENT. A request that already carries a Signature-Agent keeps +// it, so a relay forwarding an originating agent's call does not overwrite the +// value that agent's own signature covers. +func WithSignatureAgent(dir string) ClientOption { + return func(c *clientConfig) { c.signatureAgent = dir } +} + +// WithContentTimeout bounds one delivery fetch, proof minting included. The +// default is resolvers.DefaultContentTimeout. +// +// A delivery host is named by another party, so the bound is not optional — this +// option moves it, it does not remove it. A value <= 0 keeps the default. +func WithContentTimeout(d time.Duration) ClientOption { + return func(c *clientConfig) { c.fetchTimeout = d } +} + +// WithMaxContentBytes caps one fetched body. The default is +// resolvers.DefaultMaxContentBytes. +// +// Worth setting when the application carries its own per-item budget: a cap here +// that disagrees with the one the caller accounts against makes that accounting +// wrong, and an over-cap body is reported as CallTooLarge rather than truncated. +// A value <= 0 keeps the default. +// +// The two bounds are separate scalars rather than one options struct because +// ContentFetchOptions also carries the base transport, which arrives through +// WithGuardedBaseTransport — a second way to set it would be a field that had to +// be silently ignored. +func WithMaxContentBytes(n int64) ClientOption { + return func(c *clientConfig) { c.fetchMaxByte = n } +} + +// WithEndpointResolver injects the resolver that turns an offer's exchange domain +// into the origin that Exchange advertises for itself. It defaults to the +// SSRF-guarded well-known resolver. +// +// There is deliberately no option to supply an endpoint directly. A usage report +// must reach the Exchange that issued the offer, and that address comes from the +// Exchange's own manifest — never from configuration. Leaving no configuration +// slot for it is what makes that structural rather than a convention. +func WithEndpointResolver(r EndpointResolver) ClientOption { + return func(c *clientConfig) { c.endpoints = r } +} + +// WithGuardedBaseTransport carries the caller's own transport settings — a tuned +// connection pool, client certificates via TLSClientConfig — UNDERNEATH the SSRF +// guard on both legs that dial an address another party named: the content fetch, +// and the RPCs that route to the Exchange an offer identified. +// +// It is not a way to replace the guard. Those two legs dial hosts the client did +// not configure, so the dial-time address pin and the https-only scheme check are +// applied in every case; a caller supplies what sits under them. The only way to +// reach a private or plaintext endpoint is the deliberate, deployment-level +// SKIP_SSRF / ALLOW_INSECURE opt-out. +// +// One setting is dropped rather than carried: a custom TLS dialer. net/http +// prefers a transport's own TLS dialer over the pinned one on https, so honouring +// it would take every signed call around the address check. TLS itself is +// configured through TLSClientConfig, which is kept. +// +// The home Exchange and the Broker are operator-configured origins and are +// trusted as far as that configuration is, so they dial through WithHTTPClient's +// transport instead. +func WithGuardedBaseTransport(base *http.Transport) ClientOption { + return func(c *clientConfig) { c.guardedBase = base } +} + +// WithClientOptions appends raw Connect client options (a codec, a read cap +// tighter than the SDK default) to every Connect client the SDK builds. +// Interceptors belong in WithInterceptors; this is the escape hatch for the +// remaining client-level knobs the SDK does not model, mirroring +// connectserver.WithHandlerOptions on the server face. +// +// Options are appended AFTER the SDK's own, so a caller-supplied value wins. +func WithClientOptions(opts ...connectrpc.ClientOption) ClientOption { + return func(c *clientConfig) { c.connectOpts = append(c.connectOpts, opts...) } +} + // resolveOfferResolver returns the KeyResolver the offer Verifier uses. A custom // resolver (WithKeyResolver) wins; otherwise WithOfferKey produces a fixed-key // resolver that returns the injected key for any exchange id (the offer signature @@ -121,3 +311,14 @@ type emptyResolver struct{} func (emptyResolver) Resolve(_ context.Context, keyID string) (ed25519.PublicKey, error) { return nil, helpers.ErrUnknownKey } + +// resolveEndpointResolver returns the resolver an offer-derived call routes +// through: the injected one, or the SSRF-guarded well-known resolver. The +// endpoint is always read from the Exchange's own manifest — there is no +// configuration path to it. +func (c clientConfig) resolveEndpointResolver() EndpointResolver { + if c.endpoints != nil { + return c.endpoints + } + return resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{}) +} diff --git a/sdk/go/connect/options_wire_test.go b/sdk/go/connect/options_wire_test.go new file mode 100644 index 00000000..fcb627ad --- /dev/null +++ b/sdk/go/connect/options_wire_test.go @@ -0,0 +1,237 @@ +package connect_test + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "google.golang.org/protobuf/proto" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + rampconnect "github.com/RAMP-Protocol/protocol/sdk/go/connect" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// The knobs the tier below `connect` already had and the client had no way to +// reach. Each is asserted on the WIRE — the bytes the peer sees, or the outcome a +// bound produces — because an assertion that an option set a field would pass +// against a client that then dropped it, which is the defect these close. + +// A supplied signing window reaches the emitted Signature-Input, so a deployment +// with its own freshness policy gets the TTL it configured rather than the SDK's +// five-minute default. The response is deliberately not a valid Connect reply: +// the request has already been signed and sent by the time it is read, which is +// the only thing under test. +func TestWithSignWindow_ReachesTheEmittedSignature(t *testing.T) { + const created, expires = 1_700_000_000, 1_700_000_030 + + var got string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get("Signature-Input") + w.WriteHeader(http.StatusInternalServerError) + })) + defer srv.Close() + + sig := newSigningFixture(t) + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), + rampconnect.WithSignWindow(func() (int64, int64) { return created, expires }), + ) + // The call fails at the response; the signature was already on the wire. + _, _ = client.Discover(context.Background(), &rampv1.ResourceQuery{Uris: []string{"https://a.test/x"}}) + + if got == "" { + t.Fatal("no Signature-Input reached the peer; the request was not signed") + } + for _, want := range []string{"created=1700000000", "expires=1700000030"} { + if !strings.Contains(got, want) { + t.Errorf("Signature-Input = %q, want it to carry %s", got, want) + } + } +} + +// Without the option the SDK's own default still applies, so the window is a +// default and not a requirement. +func TestSignWindow_DefaultsWhenUnset(t *testing.T) { + var got string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get("Signature-Input") + w.WriteHeader(http.StatusInternalServerError) + })) + defer srv.Close() + + sig := newSigningFixture(t) + client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(sig.signer)) + _, _ = client.Discover(context.Background(), &rampv1.ResourceQuery{Uris: []string{"https://a.test/x"}}) + + if !strings.Contains(got, "created=") || !strings.Contains(got, "expires=") { + t.Errorf("Signature-Input = %q, want a stamped window from the default", got) + } +} + +// A supplied body cap is what the fetch enforces. Asserted through the refusal a +// too-large body produces, because the fetcher exposes no getter — deliberately, +// so a test cannot pass by reading back what it just set. +func TestWithMaxContentBytes_BoundsTheFetchedBody(t *testing.T) { + body := strings.Repeat("x", 512) + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(body)) + })) + defer content.Close() + + sig := newSigningFixture(t) + opts := append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + rampconnect.WithMaxContentBytes(16), + ) + client := rampconnect.NewClient("http://home.invalid", opts...) + + _, err := client.Fetch(context.Background(), content.URL+"/doc") + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallTooLarge { + t.Errorf("kind = %v, want CallTooLarge under a 16-byte cap", cerr.Kind) + } + + // The same body under the default cap succeeds, so the refusal above is the + // supplied bound rather than something else about the response. + relaxed := append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + ) + if _, err := rampconnect.NewClient("http://home.invalid", relaxed...). + Fetch(context.Background(), content.URL+"/doc"); err != nil { + t.Fatalf("the same body under the default cap must succeed: %v", err) + } +} + +// A supplied fetch deadline is what bounds the call. The handler outlives it, so +// a client that ignored the option would block for the 30-second default and the +// test would time out rather than fail. +func TestWithContentTimeout_BoundsTheFetch(t *testing.T) { + release := make(chan struct{}) + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + })) + defer content.Close() + defer close(release) + + sig := newSigningFixture(t) + opts := append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + rampconnect.WithContentTimeout(50*time.Millisecond), + ) + client := rampconnect.NewClient("http://home.invalid", opts...) + + start := time.Now() + _, err := client.Fetch(context.Background(), content.URL+"/doc") + if err == nil { + t.Fatal("a fetch past its deadline must fail") + } + if elapsed := time.Since(start); elapsed > 5*time.Second { + t.Errorf("fetch took %v; the supplied deadline was not applied", elapsed) + } +} + +// An idempotency key the caller put ON THE MESSAGE survives to the wire. +// +// The middle tier of the precedence rule, and the one with consequences: minting +// a default is intended, but overwriting a value the caller chose turns each of +// their retries into a fresh action, which is the double-counting the field +// exists to prevent. Fresh-mint and the per-call option are covered elsewhere; +// this is the branch a regression to "always mint" would slip past. +func TestReportUsage_KeepsAKeyTheCallerPutOnTheMessage(t *testing.T) { + sig := newSigningFixture(t) + origin := &groupExchange{} + domain, _ := selfAdvertisingExchange(t, sig, origin) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + const own = "app-owned-key-1" + report := &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + IdempotencyKey: own, + } + if _, err := client.ReportUsage(context.Background(), report); err != nil { + t.Fatalf("ReportUsage: %v", err) + } + if got := origin.gotReport.GetIdempotencyKey(); got != own { + t.Errorf("idempotency key = %q, want the caller's own %q", got, own) + } +} + +// A supplied proof window reaches the delivery request's signature, so the +// freshness of a bound fetch is the caller's policy rather than the SDK's +// 30-second default. +func TestWithProofWindow_ReachesTheFetchSignature(t *testing.T) { + const created, expires = 1_700_000_000, 1_700_000_045 + + var got string + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get("Signature-Input") + _, _ = w.Write([]byte("bytes")) + })) + defer content.Close() + + sig := newSigningFixture(t) + opts := append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + rampconnect.WithProofWindow(func() (int64, int64) { return created, expires }), + ) + if _, err := rampconnect.NewClient("http://home.invalid", opts...). + Fetch(context.Background(), content.URL+"/doc"); err != nil { + t.Fatalf("Fetch: %v", err) + } + for _, want := range []string{"created=1700000000", "expires=1700000045"} { + if !strings.Contains(got, want) { + t.Errorf("proof Signature-Input = %q, want it to carry %s", got, want) + } + } +} + +// Without the option the proof still carries a window, so it is a default rather +// than something a caller must supply to fetch at all. +func TestProofWindow_DefaultsWhenUnset(t *testing.T) { + var got string + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get("Signature-Input") + _, _ = w.Write([]byte("bytes")) + })) + defer content.Close() + + sig := newSigningFixture(t) + opts := append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), rampconnect.WithAgentKey(sig.pub)) + if _, err := rampconnect.NewClient("http://home.invalid", opts...). + Fetch(context.Background(), content.URL+"/doc"); err != nil { + t.Fatalf("Fetch: %v", err) + } + if !strings.Contains(got, "created=") || !strings.Contains(got, "expires=") { + t.Errorf("proof Signature-Input = %q, want a stamped window from the default", got) + } +} + +// The defaults the two options move are the resolvers tier's own, so a caller +// reading either constant gets the value the client actually runs with. +func TestContentBounds_DefaultToTheResolversTierValues(t *testing.T) { + if resolvers.DefaultContentTimeout != 30*time.Second { + t.Errorf("DefaultContentTimeout = %v", resolvers.DefaultContentTimeout) + } + if resolvers.DefaultMaxContentBytes != 8<<20 { + t.Errorf("DefaultMaxContentBytes = %d", resolvers.DefaultMaxContentBytes) + } +} diff --git a/sdk/go/connect/route.go b/sdk/go/connect/route.go new file mode 100644 index 00000000..839579bd --- /dev/null +++ b/sdk/go/connect/route.go @@ -0,0 +1,151 @@ +package connect + +import ( + "context" + "errors" + "fmt" + "net/http" + + connectrpc "connectrpc.com/connect" + + "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1/rampv1connect" + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/endpointrule" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/lrucache" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// Routing a call to the Exchange that issued an offer. +// +// There are three kinds of destination, and the difference decides how much +// checking a call needs. The Broker and the home Exchange come from the client's +// own configuration and are trusted as far as that configuration is. The Exchange +// a usage report goes to is named inside an OFFER, which arrived over the +// network — so an actor who can influence an offer can influence that address. +// +// A signature covers the DOMAIN; it says nothing about where that domain's +// endpoint lives, or where its DNS points. That is why the address is resolved +// from the Exchange's own manifest and then checked, rather than taken on trust +// or, worse, read from configuration. + +// EndpointResolver turns a signed exchange domain into the origin that Exchange +// advertises for itself. It is an interface so a test can drive reporting without +// standing up a manifest server — and, more to the point, so this package has no +// way to accept a report endpoint from configuration. +// +// An implementation's ERROR decides how a caller is told to react, so it is part +// of the contract rather than an implementation detail. A failure that is a +// VERDICT — the host is unusable, the host is not allowed, the manifest advertises +// no endpoint, or it advertises one that must not be used — MUST wrap +// helpers.ErrInvalidHost, resolvers.ErrNoEndpoint or resolvers.ErrEndpointRefused; +// those three surface as CallNotSent, which tells the caller not to retry. +// Anything else is read as a transport failure and reported as CallUnreachable, +// i.e. worth retrying. An implementation that returns a bare error for a refusal +// therefore has its final answer retried indefinitely. +type EndpointResolver interface { + ResolveEndpoint(ctx context.Context, host string) (string, error) +} + +// vetExchangeEndpoint resolves exchangeDomain to an origin a signed call may be +// sent to, or refuses — naming the check that declined, and classifying it by +// CAUSE. "The Exchange said no", "we could not reach it" and "we refused to dial +// it" are three different outcomes calling for three different responses, and +// only the class tells them apart: a verdict is final, a transport failure is +// worth retrying. +// +// It is one function so the vetting reads in one place: the checks are the part +// that grows, and seeing them together is what makes it evident that no branch +// falls through to the send. +func vetExchangeEndpoint(ctx context.Context, resolver EndpointResolver, exchangeDomain, op string) (string, error) { + if resolver == nil { + return "", notSent(op, errors.New("no endpoint resolver configured")) + } + if exchangeDomain == "" { + return "", notSent(op, errors.New("no exchange domain to route to; it comes from the signed offer")) + } + // A plain hostname, checked here even if the caller checked it. The resolver + // builds its URL by concatenating this value, so a path or query smuggled + // through would choose what gets fetched; this package owns the call and + // cannot rely on every present and future caller having vetted it. + bare, err := helpers.IsBareHost(exchangeDomain) + if err != nil { + return "", notSent(op, fmt.Errorf("exchange %q is not a usable domain: %w", exchangeDomain, err)) + } + if !bare { + return "", notSent(op, fmt.Errorf( + "exchange %q is not a bare domain, refusing to resolve it", exchangeDomain)) + } + endpoint, err := resolver.ResolveEndpoint(ctx, exchangeDomain) + if err != nil { + // Classified by CAUSE, not by position. Reaching the manifest is a network + // operation, and a DNS blip or a 500 from an otherwise healthy Exchange is + // TRANSIENT — reporting it as a refusal would tell a caller "we declined to + // send this, do not retry" and permanently drop a usage report over a + // momentary outage. Only a verdict is a refusal: the value was not a usable + // host, the host was not allowed, the manifest was read and advertises no + // endpoint at all, or it advertises one the resolver will not hand back. + // + // ErrInvalidHost is in the set because the resolver checks the host itself + // too, and a value that is not a host will not become one on a later attempt. + // This package checks it before resolving, so the SDK's own resolver never + // reaches here that way — an injected one can. + kind := CallUnreachable + if errors.Is(err, helpers.ErrInvalidHost) || + errors.Is(err, resolvers.ErrNoEndpoint) || + errors.Is(err, resolvers.ErrEndpointRefused) { + kind = CallNotSent + } + return "", &CallError{ + Kind: kind, Op: op, + Err: fmt.Errorf("resolve exchange %q: %w", exchangeDomain, err), + } + } + // Re-checked here even though the SDK's own resolver already refuses such an + // endpoint. The resolver is an injectable seam: a caller may supply one, and + // this package cannot make a signed call conditional on a stranger's + // implementation having remembered the rule. The cost is string work on a path + // that just did a network fetch. + // + // The SAME predicate both times, deliberately. Stated twice it drifts, and a + // half-mirrored version of this rule is how a signed call ends up carrying + // credentials the SDK never chose. + if err := endpointrule.Vet(exchangeDomain, endpoint); err != nil { + return "", notSent(op, fmt.Errorf( + "refusing to send a signed call to the endpoint exchange %q advertises: %w", + exchangeDomain, err)) + } + return endpoint, nil +} + +// maxPooledExchanges bounds the per-origin client pool. Which Exchanges appear is +// driven by incoming offers, so the key space is open-ended and caller-influenced +// — an unbounded map is somewhere an authenticated caller can make the process +// grow without limit. A real deployment talks to a handful of Exchanges. +const maxPooledExchanges = 256 + +// exchangePool caches one Connect client per vetted origin over the SDK's shared +// bounded map, which carries the eviction policy and the reason for it. +// +// Every pooled client shares the ONE signing HTTP client, so a cached client +// still signs as the agent of the CURRENT request. The pool caches transport +// plumbing, never identity. +type exchangePool struct { + http connectrpc.HTTPClient + opts []connectrpc.ClientOption + clients *lrucache.Cache[string, rampv1connect.ExchangeServiceClient] +} + +func newExchangePool(httpClient *http.Client, opts ...connectrpc.ClientOption) *exchangePool { + return &exchangePool{ + http: httpClient, + opts: opts, + clients: lrucache.New[string, rampv1connect.ExchangeServiceClient](maxPooledExchanges), + } +} + +// clientFor returns the cached client for origin, creating it on first use. +func (p *exchangePool) clientFor(origin string) rampv1connect.ExchangeServiceClient { + return p.clients.GetOrCreate(origin, func(o string) rampv1connect.ExchangeServiceClient { + return rampv1connect.NewExchangeServiceClient(p.http, o, p.opts...) + }) +} diff --git a/sdk/go/connect/route_internal_test.go b/sdk/go/connect/route_internal_test.go new file mode 100644 index 00000000..e81f19db --- /dev/null +++ b/sdk/go/connect/route_internal_test.go @@ -0,0 +1,40 @@ +package connect + +import ( + "net/http" + "testing" +) + +// The per-origin client pool, tested from inside the package. +// +// Eviction itself is pinned once on the shared bounded map this pool is built +// from; what is left to prove here is the wiring — that the pool is bounded at +// all, and that a known origin reuses its client rather than rebuilding the +// plumbing on every call. + +func TestExchangePool_ReusesTheClientForAKnownOrigin(t *testing.T) { + pool := newExchangePool(&http.Client{}) + first := pool.clientFor("https://ex.test") + second := pool.clientFor("https://ex.test") + if first != second { + t.Error("a known origin must reuse its cached client") + } + if got := pool.clients.Len(); got != 1 { + t.Errorf("pool size = %d, want 1", got) + } +} + +// The bound is the security property: which Exchanges appear is driven by +// incoming offers, so the key space is open-ended and caller-influenced. +func TestExchangePool_IsBoundedAtTheCap(t *testing.T) { + pool := newExchangePool(&http.Client{}) + for i := range maxPooledExchanges + 10 { + pool.clientFor(string(rune('a'+i%26)) + string(rune('0'+i/26))) + } + // EXACTLY the cap, not merely at-or-under it: this is what pins that the + // pool passed its own constant to the shared cache rather than some other + // bound. Ordering is pinned once, on the shared type. + if got := pool.clients.Len(); got != maxPooledExchanges { + t.Errorf("pool size = %d, want exactly %d", got, maxPooledExchanges) + } +} diff --git a/sdk/go/connect/taxonomy_test.go b/sdk/go/connect/taxonomy_test.go new file mode 100644 index 00000000..1f9f2f74 --- /dev/null +++ b/sdk/go/connect/taxonomy_test.go @@ -0,0 +1,186 @@ +package connect_test + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + + connectrpc "connectrpc.com/connect" + "google.golang.org/protobuf/proto" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + rampconnect "github.com/RAMP-Protocol/protocol/sdk/go/connect" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// The failure taxonomy, and the branches of it a caller is told to act on. +// +// The two vocabularies are separate types over separate tiers, bridged in one +// place. Nothing enforced that the tokens they share stay the same word, and the +// classifications a caller is told to branch on had no test at all — which is how +// a transient failure could have been reported as a permanent verdict without +// anything going red. + +// The shared tokens must stay identical across the two tiers. A caller that logs +// a reason and greps for it should not have to know which tier produced it, and +// the bridge between them maps the classes 1:1 with nothing else holding the +// words together. +func TestFailureTokens_AgreeAcrossTheTwoTiers(t *testing.T) { + pairs := []struct { + call rampconnect.CallErrorKind + fetch resolvers.FetchFailure + }{ + {rampconnect.CallRefused, resolvers.FetchRefused}, + {rampconnect.CallUnreachable, resolvers.FetchUnreachable}, + {rampconnect.CallTooLarge, resolvers.FetchTooLarge}, + {rampconnect.CallNotSignable, resolvers.FetchNotSignable}, + {rampconnect.CallMalformed, resolvers.FetchMalformed}, + } + for _, p := range pairs { + if got, want := p.call.String(), p.fetch.String(); got != want { + t.Errorf("token drift: CallErrorKind %q vs FetchFailure %q", got, want) + } + } + // The value outside either set renders the same way too, so an unmapped + // classification never prints a bare integer. + if got := rampconnect.CallErrorKind(99).String(); got != "unknown" { + t.Errorf("unnamed kind = %q, want \"unknown\"", got) + } + if got := resolvers.FetchFailure(99).String(); got != "unknown" { + t.Errorf("unnamed failure = %q, want \"unknown\"", got) + } +} + +// A transient failure to reach the manifest is UNREACHABLE, not a refusal to +// send. The distinction is the whole point of the classification: a caller +// following the documented handling for a refusal would drop a usage report for +// good over a momentary outage. +func TestReportUsage_TransientResolveFailureIsUnreachable(t *testing.T) { + // A closed port: the manifest fetch fails at the transport, which is exactly + // the shape of a DNS blip or a restarting Exchange. + dead := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + domain := strings.TrimPrefix(dead.URL, "http://") + dead.Close() + + sig := newSigningFixture(t) + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + }) + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallUnreachable { + t.Errorf("kind = %v, want CallUnreachable — a transport failure is retryable, "+ + "and reporting it as a refusal tells a caller to give up", cerr.Kind) + } +} + +// An Exchange that answers the manifest but advertises no endpoint is a VERDICT, +// not a transport failure: it was reached, and it has nothing to offer. The +// opposite branch of the same classification. +func TestReportUsage_NoAdvertisedEndpointIsNotSent(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{}`)) // a manifest with no endpoint + })) + defer srv.Close() + + sig := newSigningFixture(t) + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String(strings.TrimPrefix(srv.URL, "http://")), + TransactionId: "txn-1", + }) + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallNotSent { + t.Errorf("kind = %v, want CallNotSent — the Exchange answered and advertises nothing", cerr.Kind) + } + if !errors.Is(err, resolvers.ErrNoEndpoint) { + t.Error("the resolver's sentinel must stay reachable through the classification") + } +} + +// A peer that refuses because the response exceeds the read cap surfaces as +// CallTooLarge rather than as a generic refusal, so a caller can tell "raise your +// budget" from "the Exchange said no". +func TestSendError_ResourceExhaustedIsTooLarge(t *testing.T) { + sig := newSigningFixture(t) + domain, _ := loopbackManifestServer(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusTooManyRequests) + _, _ = w.Write([]byte(`{"code":"resource_exhausted","message":"too big"}`)) + })) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.ReportUsage(context.Background(), &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + }) + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallTooLarge { + t.Errorf("kind = %v, want CallTooLarge for a resource-exhausted peer", cerr.Kind) + } + // The Connect error stays reachable underneath, so a caller that wants the + // code rather than the SDK's class can still get it. + var connErr *connectrpc.Error + if !errors.As(err, &connErr) { + t.Error("the Connect error must stay reachable through the wrapper") + } +} + +// A caller that cancels its own context has not been refused by anyone. +// connect-go stamps CodeCanceled on a locally cancelled call, and reporting that +// as CallRefused would tell the caller the Exchange declined a request the +// Exchange may never have finished reading — a final verdict, invented locally, +// about a peer that said nothing. +func TestSendError_CallerCancellationIsNotARefusal(t *testing.T) { + sig := newSigningFixture(t) + release, reached := make(chan struct{}), make(chan struct{}) + var once sync.Once + domain, _ := loopbackManifestServer(t, http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { + once.Do(func() { close(reached) }) + <-release + })) + // Registered AFTER the server's own cleanup so it runs BEFORE it: Close blocks + // on the in-flight handler, which is parked on release until this fires. + t.Cleanup(func() { close(release) }) + + ctx, cancel := context.WithCancel(context.Background()) + go func() { + <-reached // the RPC is on the wire; nothing has answered it + cancel() + }() + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + _, err := client.ReportUsage(ctx, &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + }) + + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallUnreachable { + t.Errorf("kind = %v, want CallUnreachable — the caller gave up; the peer did not refuse", cerr.Kind) + } +} diff --git a/sdk/go/connect/testsupport_test.go b/sdk/go/connect/testsupport_test.go index 6bac722c..83121228 100644 --- a/sdk/go/connect/testsupport_test.go +++ b/sdk/go/connect/testsupport_test.go @@ -10,13 +10,51 @@ package connect_test import ( "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" "sync" + "sync/atomic" + "testing" "time" + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" "github.com/RAMP-Protocol/protocol/sdk/go/core" "google.golang.org/protobuf/types/known/timestamppb" ) +// loopbackManifestServer stands up an Exchange that advertises ITSELF in its +// well-known manifest and serves everything else from rest. +// +// The manifest route and the late-bound origin are the same five lines wherever a +// test drives the offer-derived leg; only the catch-all differs — a redirect, a +// refusal, a real RPC handler. Parameterising the catch-all is what keeps the +// self-advertising part from being retyped per test, where it can quietly drift +// into advertising something else. +// +// Returns the BARE domain, which is what a UsageReport carries: the exchange +// field names a domain, never an origin. +// It also returns the well-known FETCH COUNT, which is how the caching tests tell +// a second resolve that hit the cache from one that went back to the network. +func loopbackManifestServer(t *testing.T, rest http.Handler) (string, *atomic.Int64) { + t.Helper() + var ( + hits atomic.Int64 + origin string + ) + mux := http.NewServeMux() + mux.HandleFunc("/.well-known/ramp.json", func(w http.ResponseWriter, _ *http.Request) { + hits.Add(1) + _ = json.NewEncoder(w).Encode(map[string]any{"endpoint": origin}) + }) + mux.Handle("/", rest) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + origin = srv.URL + return strings.TrimPrefix(srv.URL, "http://"), &hits +} + // memReplayStore is a minimal in-memory nonce store: SeenOrAdd reports whether a // nonce has been observed and records it if not. It ignores TTL expiry (a test // double); the SDK owns WHEN SeenOrAdd is called in fail-closed verification. @@ -54,3 +92,15 @@ var _ core.ReplayStore = (*memReplayStore)(nil) func timestampProto(t time.Time) *timestamppb.Timestamp { return timestamppb.New(t) } + +// testRequester is the agent identity the execute path requires. A purchase +// carries a detached acceptance covering the requester, so a client that has not +// been told who it is cannot buy — these tests supply one the way an application +// would. +func testRequester() *rampv1.Requester { + return &rampv1.Requester{ + Id: "https://agent.test", + Domain: "agent.test", + Type: rampv1.RequesterType_REQUESTER_TYPE_AGENT, + } +} diff --git a/sdk/go/connect/validation_test.go b/sdk/go/connect/validation_test.go index 196edc58..1dc35152 100644 --- a/sdk/go/connect/validation_test.go +++ b/sdk/go/connect/validation_test.go @@ -36,25 +36,25 @@ func TestWithValidation_StrictRejectsInvalidRequest(t *testing.T) { // Client A (validation Off) surfaces the offer so we can obtain a VerifiedOffer // wrapping the proto-minimal fixture (no pricing.model). surfacer := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithVerification(core.Off), ) res, err := surfacer.Discover(context.Background(), &rampv1.ResourceQuery{}) if err != nil { t.Fatalf("Discover under Off verification must surface the offer: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("want 1 surfaced offer, got %d", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("want 1 surfaced offer, got %d", len(res.Verified())) } // Client B opts into strict validation: its outbound Execute request reflects // the model-less offer, which the bidirectional validate interceptor must reject // with CodeInvalidArgument BEFORE the round-trip. strict := rampconnect.NewClient(srv.URL, - rampconnect.WithSigner(sig.signer), + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), rampconnect.WithValidation(rampconnect.ValidationStrict), ) - _, err = strict.Execute(context.Background(), res.Verified[0]) + _, err = strict.Execute(context.Background(), res.Verified()[0]) if err == nil { t.Fatal("strict validation must reject an offer with no pricing.model") } diff --git a/sdk/go/connect/verbs.go b/sdk/go/connect/verbs.go new file mode 100644 index 00000000..367c4564 --- /dev/null +++ b/sdk/go/connect/verbs.go @@ -0,0 +1,220 @@ +package connect + +import ( + "context" + "errors" + "fmt" + "time" + + connectrpc "connectrpc.com/connect" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + "github.com/RAMP-Protocol/protocol/sdk/go/core" + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// defaultProofWindow is how long a delivery-fetch proof stays valid. +// +// Short on purpose, and deliberately NOT the signed URL's own expiry, which can +// be hours: the proof covers only the method and the URL, so for as long as the +// window is open anyone who observes the request can repeat it. +const defaultProofWindow = 30 * time.Second + +// edgeErrorDomain is the ErrorDetail domain for a refusal by a delivery edge. It +// is the failing surface, not the fetched resource: the field is a stable +// grouping key for tooling, so it names the tier that refused. +const edgeErrorDomain = "ramp.v1.Edge" + +// ReportUsage files a usage report with the Exchange that ISSUED the offer — +// never through a Broker, and never to an address from configuration. +// +// The destination comes off the report itself: UsageReport.exchange carries the +// offer's signed exchange domain, and the endpoint is then resolved from that +// Exchange's own well-known manifest. Reading it off the message rather than +// taking it as an argument is what makes the rule structural — there is no +// parameter a configured origin could be passed as, so it cannot become the +// default by anyone's convenience. Set it from the offer being reported: +// +// report.Exchange = proto.String(verified.Offer().GetExchange()) +// +// The report is cloned before ver and the idempotency key are stamped, so the +// message the caller built stays untouched — it crossed a package boundary as an +// argument, not as a buffer to fill in. +// +// The idempotency key identifies the REPORT, not the attempt. A fresh one is +// minted only when the caller supplied none — a value already on the message, or +// one pinned with WithIdempotencyKey, is left alone. That distinction matters: an +// application that mints its own key for its own dedup would otherwise have it +// silently discarded and see every retry counted as a second report. +func (c *Client) ReportUsage(ctx context.Context, report *rampv1.UsageReport, opts ...CallOption) (*rampv1.UsageReportResponse, error) { + const op = "report usage" + if report == nil { + return nil, malformed(op, errors.New("report is nil")) + } + endpoint, err := vetExchangeEndpoint(ctx, c.endpoints, report.GetExchange(), op) + if err != nil { + return nil, err + } + stamped, err := cloneRequest(report, op) + if err != nil { + return nil, err + } + if err = stampEnvelope(&stamped.Ver, &stamped.IdempotencyKey, opts); err != nil { + return nil, malformed(op, err) + } + resp, err := c.exchanges.clientFor(endpoint).ReportUsage(ctx, connectrpc.NewRequest(stamped)) + if err != nil { + return nil, sendError(op, err) + } + return resp.Msg, nil +} + +// Dispute files a dispute with the Exchange that issued the offer, over the same +// vetted routing a usage report takes. +// +// The exchange domain is an ARGUMENT here rather than a field, because +// DisputeRequest carries no exchange field to read it from — an asymmetry forced +// by the message shape, not chosen. It is still the offer's signed domain and +// still goes through the identical checks, so a configured value cannot reach the +// wire unvetted. +// +// The dispute chain is a structural invariant: an agent must have filed a usage +// report and received a report_id before it can dispute, so req.ReportId and +// req.TransactionId both name links the Exchange already holds. +func (c *Client) Dispute(ctx context.Context, exchangeDomain string, req *rampv1.DisputeRequest, opts ...CallOption) (*rampv1.DisputeResponse, error) { + const op = "dispute" + if req == nil { + return nil, malformed(op, errors.New("request is nil")) + } + endpoint, err := vetExchangeEndpoint(ctx, c.endpoints, exchangeDomain, op) + if err != nil { + return nil, err + } + stamped, err := cloneRequest(req, op) + if err != nil { + return nil, err + } + if err = stampEnvelope(&stamped.Ver, &stamped.IdempotencyKey, opts); err != nil { + return nil, malformed(op, err) + } + resp, err := c.exchanges.clientFor(endpoint).DisputeTransaction(ctx, connectrpc.NewRequest(stamped)) + if err != nil { + return nil, sendError(op, err) + } + return resp.Msg, nil +} + +// Fetch retrieves the content a signed delivery URL names, presenting proof of +// possession of the agent key that URL is bound to. +// +// This is the LOW-TIER fetch: follow one signed URL, present the key, return the +// bytes. It does not discover, select, buy or report — that orchestration is a +// separate, higher tier. +// +// The transport rules are the report leg's, for the same reason: the retrieval +// host is chosen by a party on the network. It dials through the SSRF guard and +// REFUSES redirects. Following one would either replay a proof bound to the old +// URL, which the edge's own check rejects, or hand a fresh proof of possession of +// the agent's key to whatever host the first hop named. +// +// A refusal from the edge arrives as a typed reason where the edge's vocabulary +// maps onto the protocol's, so ErrorDetailFrom reads a fetch failure and an RPC +// failure through the same accessor. +// +// It takes no CallOption: a fetch is a GET against an already-issued URL, so +// there is no idempotency key to pin — nothing on this path mutates state. +// +// The URL is taken as given. Whether it is one this agent bought, and whether its +// agent_id matches this agent's key, are the CALLER's checks to make — the SDK +// exports helpers.VerifyURLEd25519 and VerifiedURL.CheckProofOfPossession for +// exactly that, and running them first turns an edge 403 into a local answer. +// Worth doing when the URL reached the caller from anywhere but its own execute +// response: a proof of possession is minted for whatever URL is passed in. +func (c *Client) Fetch(ctx context.Context, signedURL string) (resolvers.Content, error) { + const op = "fetch content" + signer, err := c.proofSigner() + if err != nil { + return resolvers.Content{}, &CallError{Kind: CallNotSignable, Op: op, Err: err} + } + content, err := c.fetcher.Fetch(ctx, signedURL, signer) + if err != nil { + return resolvers.Content{}, fetchCallError(op, signedURL, err) + } + return content, nil +} + +// proofSigner composes the injected custody into the seam the content tier asks +// for. Both halves are required and neither can be derived from the other: the +// Signer keeps the private half, and the header presents the public one. +func (c *Client) proofSigner() (resolvers.ProofSigner, error) { + signer := c.cfg.signer + if signer == nil { + return nil, errors.New("no signer configured; a bound fetch proves possession of the agent key (see WithSigner)") + } + if len(c.cfg.agentKey) == 0 { + return nil, errors.New("no agent public key configured; a bound fetch presents it alongside the proof (see WithAgentKey)") + } + window := c.cfg.proofWindow + if window == nil { + window = core.ClockWindow(time.Now, defaultProofWindow) + } + return proofSigner{signer: signer, pub: c.cfg.agentKey, window: window}, nil +} + +// proofSigner mints one agent binding per fetch. +type proofSigner struct { + signer helpers.Signer + pub []byte + window core.Window +} + +func (p proofSigner) SignFetch(ctx context.Context, target string) (helpers.AgentBinding, error) { + created, expires := p.window() + return helpers.SignAgentBinding(ctx, p.signer, p.pub, helpers.PoPOptions{ + URL: target, Created: created, Expires: expires, + }) +} + +// fetchCallError translates the content tier's failure into the client's own +// taxonomy, and promotes the edge's refusal token to a typed protocol reason when +// the vocabularies line up. +// +// The detail is SYNTHESIZED here rather than received: a delivery edge answers a +// small JSON object, not a protobuf. Its domain names the delivery edge as the +// failing SURFACE — a stable grouping, matching the value the cross-language +// error-detail corpus already uses for this reason block. Deliberately not the +// fetched URL: domain mirrors google.rpc.ErrorInfo.domain so generic tooling can +// group errors, and a per-URL value has unbounded cardinality and groups nothing. +func fetchCallError(op, signedURL string, err error) error { + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) { + return &CallError{Kind: CallUnknown, Op: op, Err: err} + } + out := &CallError{ + Kind: fetchKinds[ferr.Failure], + Op: op, + Status: ferr.Status, + Reason: ferr.Reason, + Err: err, + } + if reason, ok := helpers.RetrievalAuthFailureReasonFromToken(ferr.Reason); ok { + out.Detail = helpers.RetrievalAuthFailureDetail( + edgeErrorDomain, + fmt.Sprintf("delivery refused: %s", ferr.Reason), + reason, + ) + } + return out +} + +// fetchKinds maps the content tier's failure classes onto the client's. The two +// vocabularies are deliberately separate — the content tier knows nothing about +// RPCs — and this is the single place they meet. +var fetchKinds = map[resolvers.FetchFailure]CallErrorKind{ + resolvers.FetchRefused: CallRefused, + resolvers.FetchUnreachable: CallUnreachable, + resolvers.FetchTooLarge: CallTooLarge, + resolvers.FetchNotSignable: CallNotSignable, + resolvers.FetchMalformed: CallMalformed, +} diff --git a/sdk/go/connect/verbs_test.go b/sdk/go/connect/verbs_test.go new file mode 100644 index 00000000..12ad4c57 --- /dev/null +++ b/sdk/go/connect/verbs_test.go @@ -0,0 +1,809 @@ +package connect_test + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + + connectrpc "connectrpc.com/connect" + "google.golang.org/protobuf/proto" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1/rampv1connect" + rampconnect "github.com/RAMP-Protocol/protocol/sdk/go/connect" + rampserver "github.com/RAMP-Protocol/protocol/sdk/go/connectserver" + "github.com/RAMP-Protocol/protocol/sdk/go/core" + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// The four verbs this SDK was missing, plus the two shipped ones it had to fix, +// driven through the outermost public surface: an SDK-built client over real HTTP +// to a real Connect handler running the SDK's own server verify face. + +// --------------------------------------------------------------------------- +// Origins +// --------------------------------------------------------------------------- + +// groupExchange serves a ResourceResponse in whichever offer representation a +// test needs, and records the report and dispute it received. +type groupExchange struct { + rampv1connect.UnimplementedExchangeServiceHandler + groups []*rampv1.OfferGroup + flat []*rampv1.Offer + + gotReport *rampv1.UsageReport + gotDispute *rampv1.DisputeRequest +} + +func (g *groupExchange) DiscoverResources( + _ context.Context, _ *connectrpc.Request[rampv1.ResourceQuery], +) (*connectrpc.Response[rampv1.ResourceResponse], error) { + return connectrpc.NewResponse(&rampv1.ResourceResponse{ + Exchange: "exchange.test", + Offers: g.flat, + OfferGroups: g.groups, + }), nil +} + +func (g *groupExchange) ReportUsage( + _ context.Context, req *connectrpc.Request[rampv1.UsageReport], +) (*connectrpc.Response[rampv1.UsageReportResponse], error) { + g.gotReport = req.Msg + return connectrpc.NewResponse(&rampv1.UsageReportResponse{ + Ver: helpers.ProtocolVersion, ReportId: "report-1", + }), nil +} + +func (g *groupExchange) DisputeTransaction( + _ context.Context, req *connectrpc.Request[rampv1.DisputeRequest], +) (*connectrpc.Response[rampv1.DisputeResponse], error) { + g.gotDispute = req.Msg + return connectrpc.NewResponse(&rampv1.DisputeResponse{ + Ver: helpers.ProtocolVersion, DisputeId: proto.String("dispute-1"), + }), nil +} + +// stubBroker serves one DiscoveryResponse. +type stubBroker struct { + rampv1connect.UnimplementedBrokerServiceHandler + groups []*rampv1.OfferGroup + absence *rampv1.OfferAbsenceReason +} + +func (b *stubBroker) Resolve( + _ context.Context, _ *connectrpc.Request[rampv1.DiscoveryRequest], +) (*connectrpc.Response[rampv1.DiscoveryResponse], error) { + return connectrpc.NewResponse(&rampv1.DiscoveryResponse{ + Ver: helpers.ProtocolVersion, OfferGroups: b.groups, AbsenceReason: b.absence, + }), nil +} + +func serveExchange(t *testing.T, sig signingFixture, svc rampv1connect.ExchangeServiceHandler) *httptest.Server { + t.Helper() + path, h := rampserver.NewExchangeServiceHandler(svc, rampserver.WithKeyResolver(sig.resolver)) + mux := http.NewServeMux() + mux.Handle(path, h) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + return srv +} + +func absenceReason(r rampv1.OfferAbsenceReason) *rampv1.OfferAbsenceReason { return &r } + +// --------------------------------------------------------------------------- +// Discover: the two offer representations +// --------------------------------------------------------------------------- + +// A grouped answer keeps every URI, including the ones that yielded nothing — +// which is the whole reason the result is grouped. A refused URI has no offer to +// carry it back, so flattening would erase it entirely. +func TestDiscover_KeepsPerURIGroupsAndReasons(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + srv := serveExchange(t, sig, &groupExchange{groups: []*rampv1.OfferGroup{ + {Uri: "https://site.test/a", Offers: []*rampv1.Offer{offers.good}}, + { + Uri: "https://site.test/b", + AbsenceReason: absenceReason(rampv1.OfferAbsenceReason_OFFER_ABSENCE_REASON_SCOPE_INSUFFICIENT), + }, + { + Uri: "https://site.test/c", + AbsenceReason: absenceReason(rampv1.OfferAbsenceReason_OFFER_ABSENCE_REASON_RESTRICTION_FILTERED), + RestrictionFilters: []rampv1.RestrictionKind{rampv1.RestrictionKind_RESTRICTION_KIND_GEOGRAPHY}, + }, + }}) + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), rampconnect.WithOfferKey(offers.exchangePub)) + + res, err := client.Discover(context.Background(), &rampv1.ResourceQuery{}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if len(res.Groups) != 3 { + t.Fatalf("want a group per requested URI, got %d", len(res.Groups)) + } + if got := res.Groups[0].URI; got != "https://site.test/a" { + t.Errorf("group 0 URI = %q", got) + } + if len(res.Groups[0].Verified) != 1 { + t.Errorf("group 0 must carry its verified offer, got %d", len(res.Groups[0].Verified)) + } + // The refusal is an ANSWER: the agent can tell "acquire an entitlement and + // retry" from "give up" only because the reason survived. + if res.Groups[1].AbsenceReason == nil || + *res.Groups[1].AbsenceReason != rampv1.OfferAbsenceReason_OFFER_ABSENCE_REASON_SCOPE_INSUFFICIENT { + t.Errorf("group 1 absence reason = %v, want SCOPE_INSUFFICIENT", res.Groups[1].AbsenceReason) + } + if len(res.Groups[2].RestrictionFilters) != 1 { + t.Errorf("group 2 must carry the filtered axis, got %v", res.Groups[2].RestrictionFilters) + } + if res.Exchange != "exchange.test" { + t.Errorf("Exchange = %q, want the responding Exchange", res.Exchange) + } +} + +// A responder that populates BOTH representations must not have its offers +// counted twice: the flat list mirrors the grouped one. +func TestDiscover_GroupsWinOverTheFlatMirrorWithoutDoubleCounting(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + srv := serveExchange(t, sig, &groupExchange{ + groups: []*rampv1.OfferGroup{{Uri: "https://site.test/a", Offers: []*rampv1.Offer{offers.good}}}, + flat: []*rampv1.Offer{offers.good}, // the same offer, mirrored + }) + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), rampconnect.WithOfferKey(offers.exchangePub)) + + res, err := client.Discover(context.Background(), &rampv1.ResourceQuery{}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if n := len(res.Verified()); n != 1 { + t.Fatalf("the mirrored offer must be counted once, got %d", n) + } + if len(res.Groups) != 1 || res.Groups[0].URI != "https://site.test/a" { + t.Errorf("groups must win over the flat mirror, got %+v", res.Groups) + } +} + +// A responder that sends only the flat list still works, and a single-URI query +// lets the SDK attribute it. A multi-URI query does not, and the SDK does not +// invent an attribution the wire never made. +func TestDiscover_FlatFallback(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + srv := serveExchange(t, sig, &groupExchange{flat: []*rampv1.Offer{offers.good}}) + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), rampconnect.WithOfferKey(offers.exchangePub)) + + single, err := client.Discover(context.Background(), + &rampv1.ResourceQuery{Uris: []string{"https://site.test/a"}}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if len(single.Groups) != 1 || single.Groups[0].URI != "https://site.test/a" { + t.Errorf("a single-URI flat answer takes that URI, got %+v", single.Groups) + } + multi, err := client.Discover(context.Background(), + &rampv1.ResourceQuery{Uris: []string{"https://site.test/a", "https://site.test/b"}}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + if len(multi.Groups) != 1 || multi.Groups[0].URI != "" { + t.Errorf("a multi-URI flat answer carries no attribution, got %+v", multi.Groups) + } +} + +// --------------------------------------------------------------------------- +// Execute: the acceptance the shipped verb never sent +// --------------------------------------------------------------------------- + +// A purchase must carry the requester and a detached acceptance that VERIFIES — +// an Exchange checks it, so a request without one can only ever be refused. +func TestExecute_SendsRequesterAndAVerifyingAcceptance(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + origin := &groupExchange{groups: []*rampv1.OfferGroup{ + {Uri: "https://site.test/a", Offers: []*rampv1.Offer{offers.good}}, + }} + srv := serveExchange(t, sig, origin) + + // One key signs the transport, the acceptance and any later fetch proof — the + // protocol carries a single agent identity, so the test uses a single key and + // verifies the acceptance against its public half. + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), + rampconnect.WithOfferKey(offers.exchangePub), + rampconnect.WithRequester(testRequester()), + ) + res, err := client.Discover(context.Background(), &rampv1.ResourceQuery{}) + if err != nil { + t.Fatalf("Discover: %v", err) + } + verified := res.Verified()[0] + + execOrigin := &recordingExecute{} + execSrv := serveExchange(t, sig, execOrigin) + execClient := rampconnect.NewClient(execSrv.URL, + rampconnect.WithSigner(sig.signer), + rampconnect.WithRequester(testRequester()), + ) + if _, err = execClient.Execute(context.Background(), verified, + rampconnect.WithIdempotencyKey("pinned-key")); err != nil { + t.Fatalf("Execute: %v", err) + } + + got := execOrigin.req + if got.GetRequester().GetId() != testRequester().GetId() { + t.Errorf("requester = %+v, want the configured identity", got.GetRequester()) + } + item := got.GetItems()[0] + if item.GetAgentAcceptance().GetSignatureAlgorithm() != helpers.AcceptanceSignatureAlgorithm { + t.Errorf("acceptance algorithm = %q", item.GetAgentAcceptance().GetSignatureAlgorithm()) + } + if err = helpers.VerifyOfferAcceptance( + item.GetOffer(), got.GetRequester(), got.GetIdempotencyKey(), + item.GetAgentAcceptance().GetSignature(), sig.pub, + ); err != nil { + t.Errorf("the acceptance an Exchange would check does not verify: %v", err) + } +} + +// Every precondition refuses BEFORE anything leaves the process — an unsigned +// offer is reachable through the two named opt-outs, and an acceptance floating +// free of a concrete offer is meaningless. +func TestExecute_FailsClosedWithoutSendingAnything(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + origin := &recordingExecute{} + srv := serveExchange(t, sig, origin) + + unsigned := core.RejectedOffer{Offer: sampleOffer("offer-unsigned")}.Unsafe() + signedOffer := core.RejectedOffer{Offer: offers.good}.Unsafe() + + tests := map[string]struct { + opts []rampconnect.ClientOption + offer core.VerifiedOffer + }{ + "no requester": { + []rampconnect.ClientOption{rampconnect.WithSigner(sig.signer)}, signedOffer, + }, + "unsigned offer": { + []rampconnect.ClientOption{ + rampconnect.WithSigner(sig.signer), rampconnect.WithRequester(testRequester()), + }, unsigned, + }, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + client := rampconnect.NewClient(srv.URL, tc.opts...) + if _, err := client.Execute(context.Background(), tc.offer); err == nil { + t.Fatal("expected a refusal") + } + if origin.req != nil { + t.Error("the origin was contacted; the refusal must be local") + } + }) + } +} + +type recordingExecute struct { + rampv1connect.UnimplementedExchangeServiceHandler + req *rampv1.TransactionRequest +} + +func (r *recordingExecute) ExecuteTransaction( + _ context.Context, req *connectrpc.Request[rampv1.TransactionRequest], +) (*connectrpc.Response[rampv1.TransactionResponse], error) { + r.req = req.Msg + return connectrpc.NewResponse(&rampv1.TransactionResponse{Ver: helpers.ProtocolVersion}), nil +} + +// --------------------------------------------------------------------------- +// Resolve: the broker face +// --------------------------------------------------------------------------- + +func TestBrokerResolve_SplitsThroughTheSameVerifier(t *testing.T) { + sig := newSigningFixture(t) + offers := newOfferFixture(t) + path, h := rampserver.NewBrokerServiceHandler( + &stubBroker{groups: []*rampv1.OfferGroup{{ + Uri: "https://site.test/a", + Offers: []*rampv1.Offer{offers.good, offers.doctored}, + }}}, + rampserver.WithKeyResolver(sig.resolver), + ) + mux := http.NewServeMux() + mux.Handle(path, h) + srv := httptest.NewServer(mux) + defer srv.Close() + + broker := rampconnect.NewBrokerClient(srv.URL, + rampconnect.WithSigner(sig.signer), rampconnect.WithOfferKey(offers.exchangePub), + rampconnect.WithRequester(testRequester())) + res, err := broker.Resolve(context.Background(), + &rampv1.DiscoveryRequest{Ver: helpers.ProtocolVersion}) + if err != nil { + t.Fatalf("Resolve: %v", err) + } + // A relayed offer is exactly the case the fail-closed rule exists for: the + // Broker forwards offers it did not mint. + if len(res.Groups) != 1 { + t.Fatalf("want one group, got %d", len(res.Groups)) + } + if len(res.Groups[0].Verified) != 1 || len(res.Groups[0].Rejected) != 1 { + t.Errorf("want the doctored offer rejected in its own group, got %d verified / %d rejected", + len(res.Groups[0].Verified), len(res.Groups[0].Rejected)) + } +} + +// A resolve that finds nothing is a successful answer carrying a typed reason, +// never an error. +func TestBrokerResolve_WholeCallRefusalIsAnAnswer(t *testing.T) { + sig := newSigningFixture(t) + path, h := rampserver.NewBrokerServiceHandler( + &stubBroker{absence: absenceReason(rampv1.OfferAbsenceReason_OFFER_ABSENCE_REASON_NOT_AUTHORIZED)}, + rampserver.WithKeyResolver(sig.resolver), + ) + mux := http.NewServeMux() + mux.Handle(path, h) + srv := httptest.NewServer(mux) + defer srv.Close() + + broker := rampconnect.NewBrokerClient(srv.URL, rampconnect.WithSigner(sig.signer), + rampconnect.WithRequester(testRequester())) + res, err := broker.Resolve(context.Background(), + &rampv1.DiscoveryRequest{Ver: helpers.ProtocolVersion}) + if err != nil { + t.Fatalf("a refusal must not be raised as an error: %v", err) + } + if res.AbsenceReason == nil || + *res.AbsenceReason != rampv1.OfferAbsenceReason_OFFER_ABSENCE_REASON_NOT_AUTHORIZED { + t.Errorf("whole-call absence reason = %v, want NOT_AUTHORIZED", res.AbsenceReason) + } + if len(res.Groups) != 0 { + t.Errorf("a refusal carries no groups, got %d", len(res.Groups)) + } +} + +// A Broker resolves who is asking from the requester and declines a request that +// names none, so the client refuses it HERE rather than spending a round trip to +// be told — and names the remedy, which a relayed "requester required" cannot. +// Execute refuses the same way and that arm is covered; without this one the +// whole check could be deleted with every test still green. +func TestBrokerResolve_RefusesARequesterlessRequestLocally(t *testing.T) { + sig := newSigningFixture(t) + var hits atomic.Int64 + path, h := rampserver.NewBrokerServiceHandler( + &stubBroker{}, rampserver.WithKeyResolver(sig.resolver)) + mux := http.NewServeMux() + mux.Handle(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + h.ServeHTTP(w, r) + })) + srv := httptest.NewServer(mux) + defer srv.Close() + + // Every option the face uses EXCEPT WithRequester. + broker := rampconnect.NewBrokerClient(srv.URL, rampconnect.WithSigner(sig.signer)) + _, err := broker.Resolve(context.Background(), + &rampv1.DiscoveryRequest{Ver: helpers.ProtocolVersion}) + + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("error = %v, want a CallError", err) + } + if cerr.Kind != rampconnect.CallMalformed { + t.Errorf("kind = %v, want CallMalformed — the request is unsendable, not refused", cerr.Kind) + } + if !strings.Contains(err.Error(), "WithRequester") { + t.Errorf("the refusal must name the remedy, got %q", err) + } + if n := hits.Load(); n != 0 { + t.Errorf("the Broker was contacted %d time(s); this refusal must be local", n) + } +} + +// --------------------------------------------------------------------------- +// ReportUsage: offer-driven routing and the checks before the send +// --------------------------------------------------------------------------- + +// selfAdvertisingExchange stands up ONE host serving both the Exchange's RPC +// endpoint and its own /.well-known/ramp.json — which is what a real Exchange +// does, and what the same-host check requires. The manifest advertises the +// server's own origin, so the endpoint is anchored to the domain it was resolved +// from. It returns the bare domain a report routes on, plus the well-known fetch +// counter. +func selfAdvertisingExchange(t *testing.T, sig signingFixture, svc rampv1connect.ExchangeServiceHandler) (string, *atomic.Int64) { + t.Helper() + path, h := rampserver.NewExchangeServiceHandler(svc, rampserver.WithKeyResolver(sig.resolver)) + rpc := http.NewServeMux() + rpc.Handle(path, h) + // The manifest half is loopbackManifestServer's, so the self-advertising part + // is written once. This is the "a real RPC handler" case its catch-all takes. + return loopbackManifestServer(t, rpc) +} + +// crossHost serves a manifest advertising SOMEONE ELSE's origin — the case the +// same-host check exists to refuse. +// +// The advertised name is a foreign hostname rather than another loopback server: +// anchoring compares hostnames, and every httptest server shares 127.0.0.1, so +// two local servers are the same host by the rule under test. A refusal must be +// driven by a genuinely different NAME, which is also the production shape. +func crossHost(t *testing.T, endpoint string) string { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/.well-known/ramp.json" { + w.WriteHeader(http.StatusNotFound) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{"endpoint": endpoint}) + })) + t.Cleanup(srv.Close) + return strings.TrimPrefix(srv.URL, "http://") +} + +// allowLoopback opts this test out of the production dial posture the way a +// deployment does — through the two documented env flags, read when the client is +// built. There is deliberately no option that removes the guard: injecting a +// transport puts it UNDER the guard, never in place of it, so a test that wants a +// loopback Exchange has to say so the same way an on-prem deployment would. +// +// The well-known resolver is still injected, because reading a manifest over http +// is a scheme choice rather than a guard opt-out. +func allowLoopback(t *testing.T) []rampconnect.ClientOption { + t.Helper() + t.Setenv("SKIP_SSRF", "1") + t.Setenv("ALLOW_INSECURE", "1") + return []rampconnect.ClientOption{ + rampconnect.WithEndpointResolver(resolvers.NewWellKnownEndpointResolver( + resolvers.WellKnownOptions{Scheme: "http", HTTP: http.DefaultClient})), + } +} + +func TestReportUsage_RoutesThroughTheIssuingExchangesOwnManifest(t *testing.T) { + sig := newSigningFixture(t) + origin := &groupExchange{} + domain, wkHits := selfAdvertisingExchange(t, sig, origin) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + report := &rampv1.UsageReport{ + Exchange: proto.String(domain), + TransactionId: "txn-1", + Usage: &rampv1.Usage{Function: []string{"ai-input"}}, + } + resp, err := client.ReportUsage(context.Background(), report) + if err != nil { + t.Fatalf("ReportUsage: %v", err) + } + if resp.GetReportId() != "report-1" { + t.Errorf("report id = %q", resp.GetReportId()) + } + if origin.gotReport.GetVer() != helpers.ProtocolVersion { + t.Errorf("ver = %q, want it stamped from the one constant", origin.gotReport.GetVer()) + } + if origin.gotReport.GetIdempotencyKey() == "" { + t.Error("a fresh idempotency key must be minted by default") + } + // The caller's message crossed a package boundary as an argument, not a buffer. + if report.GetVer() != "" || report.GetIdempotencyKey() != "" { + t.Errorf("the caller's report was mutated: %+v", report) + } + // A second report to the same Exchange reuses the cached manifest. + if _, err = client.ReportUsage(context.Background(), report); err != nil { + t.Fatalf("second ReportUsage: %v", err) + } + if n := wkHits.Load(); n != 1 { + t.Errorf("well-known fetched %d times, want it cached per host", n) + } +} + +// Every routing refusal happens before anything is sent, and says so: a caller +// must be able to tell "we refused to dial it" from "it did not answer". +func TestReportUsage_RefusesUnroutableAddressesWithoutSending(t *testing.T) { + sig := newSigningFixture(t) + + tests := map[string]string{ + "no exchange on the report": "", + "scheme is not a bare host": "https://exchange.test", + "path is not a bare host": "exchange.test/reports", + "query is not a bare host": "exchange.test?x=1", + "trailing colon": "exchange.test:", + // A manifest advertising an unrelated host — the case anchoring exists for. + "endpoint on another host": crossHost(t, "http://evil.invalid/v1"), + } + for name, domain := range tests { + t.Run(name, func(t *testing.T) { + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + report := &rampv1.UsageReport{TransactionId: "txn-1"} + if domain != "" { + report.Exchange = proto.String(domain) + } + _, err := client.ReportUsage(context.Background(), report) + if err == nil { + t.Fatal("expected a refusal") + } + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) || cerr.Kind != rampconnect.CallNotSent { + t.Fatalf("error = %v, want a CallNotSent CallError", err) + } + }) + } +} + +// --------------------------------------------------------------------------- +// Dispute +// --------------------------------------------------------------------------- + +func TestDispute_RoutesLikeAReportAndStampsTheEnvelope(t *testing.T) { + sig := newSigningFixture(t) + origin := &groupExchange{} + domain, _ := selfAdvertisingExchange(t, sig, origin) + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + req := &rampv1.DisputeRequest{ + TransactionId: "txn-1", + ReportId: "report-1", + Reason: rampv1.DisputeReason_DISPUTE_REASON_DELIVERY_FAILED, + } + resp, err := client.Dispute(context.Background(), domain, req, + rampconnect.WithIdempotencyKey("pinned")) + if err != nil { + t.Fatalf("Dispute: %v", err) + } + if resp.GetDisputeId() != "dispute-1" { + t.Errorf("dispute id = %q", resp.GetDisputeId()) + } + if origin.gotDispute.GetIdempotencyKey() != "pinned" { + t.Errorf("idempotency key = %q, want the pinned one", origin.gotDispute.GetIdempotencyKey()) + } + if origin.gotDispute.GetVer() != helpers.ProtocolVersion { + t.Errorf("ver = %q", origin.gotDispute.GetVer()) + } + if req.GetVer() != "" || req.GetIdempotencyKey() != "" { + t.Errorf("the caller's request was mutated: %+v", req) + } +} + +// The same vetting guards both verbs, so a dispute cannot be aimed at an +// unroutable address either. +func TestDispute_SharesTheRoutingRefusals(t *testing.T) { + sig := newSigningFixture(t) + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.Dispute(context.Background(), "https://exchange.test", + &rampv1.DisputeRequest{TransactionId: "txn-1"}) + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) || cerr.Kind != rampconnect.CallNotSent { + t.Fatalf("error = %v, want a CallNotSent CallError", err) + } +} + +// --------------------------------------------------------------------------- +// Fetch +// --------------------------------------------------------------------------- + +func TestFetch_PresentsTheProofAndSurfacesATypedRefusal(t *testing.T) { + sig := newSigningFixture(t) + + var sawProof atomic.Bool + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(helpers.AgentKeyHeader) != "" && r.Header.Get("Signature") != "" { + sawProof.Store(true) + } + if r.URL.Query().Get("refuse") == "1" { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"reason":"pop_expired"}`)) + return + } + w.Header().Set("Content-Type", "text/plain") + _, _ = w.Write([]byte("licensed bytes")) + })) + defer content.Close() + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + )...) + + got, err := client.Fetch(context.Background(), content.URL+"/doc?agent_id=tp") + if err != nil { + t.Fatalf("Fetch: %v", err) + } + if string(got.Body) != "licensed bytes" || got.MIMEType != "text/plain" { + t.Errorf("content = %+v", got) + } + if !sawProof.Load() { + t.Error("the fetch presented no proof of possession") + } + + // A refusal the edge names in its own vocabulary reaches the caller as the + // SAME typed reason an RPC refusal would, through the same accessor. + _, err = client.Fetch(context.Background(), content.URL+"/doc?refuse=1") + if err == nil { + t.Fatal("expected the edge refusal to surface") + } + detail, ok := rampconnect.ErrorDetailFrom(err) + if !ok { + t.Fatalf("no typed detail on a refused fetch: %v", err) + } + if got := detail.GetRetrievalAuthFailure().GetReason(); got != + rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_PROOF_EXPIRED { + t.Errorf("typed reason = %v, want PROOF_EXPIRED", got) + } + // The domain names the failing SURFACE, not the fetched URL: it is a grouping + // key for tooling, and a per-URL value has unbounded cardinality and groups + // nothing. Asserted as the exact value the cross-language error-detail corpus + // uses, so a change here has to be a deliberate one. + if got := detail.GetDomain(); got != "ramp.v1.Edge" { + t.Errorf("detail domain = %q, want the delivery edge as the failing surface", got) + } +} + +// A client that can buy but was never given the public half of its agent key +// cannot fetch, and says so rather than presenting a proof the edge will refuse. +func TestFetch_RefusesWithoutTheAgentPublicKey(t *testing.T) { + sig := newSigningFixture(t) + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), rampconnect.WithSigner(sig.signer))...) + + _, err := client.Fetch(context.Background(), "http://cdn.invalid/doc") + var cerr *rampconnect.CallError + if !errors.As(err, &cerr) || cerr.Kind != rampconnect.CallNotSignable { + t.Fatalf("error = %v, want a CallNotSignable CallError", err) + } +} + +// --------------------------------------------------------------------------- +// Signature-Agent: the directory a peer resolves the caller's key from +// --------------------------------------------------------------------------- + +// The configured directory must reach the WIRE, covered by the signature. +// +// signature-agent is one of the five REQUIRED covered components, so the header is +// signed whether or not a value was supplied — an unset client signs an EMPTY one. +// A peer that resolves the caller's key by fetching the WBA directory at that +// origin then has nothing to resolve and refuses the call at verification, after +// it was routed, signed and sent. That failure mode is why asserting the option +// sets a field would prove nothing: what matters is the bytes that leave. +func TestWithSignatureAgent_ReachesTheWireCovered(t *testing.T) { + const dir = "https://agent.example" + sig := newSigningFixture(t) + + var gotAgent, gotSigInput string + path, h := rampserver.NewExchangeServiceHandler( + &groupExchange{}, rampserver.WithKeyResolver(sig.resolver)) + mux := http.NewServeMux() + mux.Handle(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAgent, gotSigInput = r.Header.Get("Signature-Agent"), r.Header.Get("Signature-Input") + h.ServeHTTP(w, r) + })) + srv := httptest.NewServer(mux) + defer srv.Close() + + client := rampconnect.NewClient(srv.URL, + rampconnect.WithSigner(sig.signer), + rampconnect.WithSignatureAgent(dir), + rampconnect.WithRequester(testRequester())) + + // The call must SUCCEED: the header participates in the signature, so a value + // that reached the wire without being covered correctly would fail here. + if _, err := client.Discover(context.Background(), &rampv1.ResourceQuery{ + Uris: []string{"https://site.test/a"}, Ver: helpers.ProtocolVersion, + }); err != nil { + t.Fatalf("Discover: %v", err) + } + if gotAgent != dir { + t.Errorf("Signature-Agent = %q, want %q", gotAgent, dir) + } + // Present is not enough — an uncovered header is one any intermediary may + // rewrite, which is the whole reason the component is in the required set. + if !strings.Contains(gotSigInput, `"signature-agent"`) { + t.Errorf("Signature-Input = %q; want it to cover signature-agent", gotSigInput) + } +} + +// The Broker face stamps it too. Both clients reach the wire through the same +// plumbing, and that is the property worth pinning rather than assuming — a +// second construction path is exactly where one knob gets dropped. +func TestWithSignatureAgent_BrokerClientStampsItToo(t *testing.T) { + const dir = "https://agent.example" + sig := newSigningFixture(t) + + var gotAgent string + path, h := rampserver.NewBrokerServiceHandler( + &stubBroker{}, rampserver.WithKeyResolver(sig.resolver)) + mux := http.NewServeMux() + mux.Handle(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAgent = r.Header.Get("Signature-Agent") + h.ServeHTTP(w, r) + })) + srv := httptest.NewServer(mux) + defer srv.Close() + + broker := rampconnect.NewBrokerClient(srv.URL, + rampconnect.WithSigner(sig.signer), + rampconnect.WithSignatureAgent(dir), + rampconnect.WithRequester(testRequester())) + if _, err := broker.Resolve(context.Background(), + &rampv1.DiscoveryRequest{Ver: helpers.ProtocolVersion}); err != nil { + t.Fatalf("Resolve: %v", err) + } + if gotAgent != dir { + t.Errorf("Signature-Agent = %q, want %q", gotAgent, dir) + } +} + +// WithRequestIDFunc reaches the DELIVERY leg, not only the RPC legs. +// +// The two RPC legs correlate through an interceptor; a delivery fetch is a plain +// GET that never reaches one, so the option had to be threaded to the fetcher +// separately. A caller that passes one mint reasonably expects one id across every +// leg of a call, and a delivery edge that mints its own when the header is absent +// is where the gap surfaces — as two log lines under two ids and nothing joining +// them. +func TestFetch_CarriesTheClientsCorrelationID(t *testing.T) { + sig := newSigningFixture(t) + + var got string + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get(helpers.RequestIDHeader) + w.Header().Set("Content-Type", "text/plain") + _, _ = w.Write([]byte("licensed bytes")) + })) + defer content.Close() + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), + rampconnect.WithAgentKey(sig.pub), + rampconnect.WithRequestIDFunc(func() string { return "req-from-the-caller" }), + )...) + + if _, err := client.Fetch(context.Background(), content.URL+"/doc?agent_id=tp"); err != nil { + t.Fatalf("Fetch: %v", err) + } + if got != "req-from-the-caller" { + t.Errorf("%s = %q, want the client's own mint", helpers.RequestIDHeader, got) + } +} + +// And with no mint configured the leg still correlates: the client falls back to +// the same default source the RPC legs use, so the id is present rather than left +// for the edge to invent. +func TestFetch_CorrelatesEvenWithNoMintConfigured(t *testing.T) { + sig := newSigningFixture(t) + + var got string + content := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get(helpers.RequestIDHeader) + _, _ = w.Write([]byte("bytes")) + })) + defer content.Close() + + client := rampconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + rampconnect.WithSigner(sig.signer), rampconnect.WithAgentKey(sig.pub), + )...) + + if _, err := client.Fetch(context.Background(), content.URL+"/doc?agent_id=tp"); err != nil { + t.Fatalf("Fetch: %v", err) + } + if got == "" { + t.Errorf("%s is empty; the client must fall back to the default mint", helpers.RequestIDHeader) + } +} diff --git a/sdk/go/connectserver/server_verify_test.go b/sdk/go/connectserver/server_verify_test.go index 08034380..4691bfac 100644 --- a/sdk/go/connectserver/server_verify_test.go +++ b/sdk/go/connectserver/server_verify_test.go @@ -227,15 +227,22 @@ func TestServerVerify_FirstRequestAcceptedReplayRejected(t *testing.T) { client := rampconnect.NewClient(srv.URL, rampconnect.WithSigner(f.signer), rampconnect.WithOfferKey(off.exchangePub), + // A purchase carries a detached acceptance covering the requester, so a + // client that has not been told who it is cannot buy. + rampconnect.WithRequester(&rampv1.Requester{ + Id: "https://agent.test", + Domain: "agent.test", + Type: rampv1.RequesterType_REQUESTER_TYPE_AGENT, + }), ) res, err := client.Discover(context.Background(), &rampv1.ResourceQuery{}) if err != nil { t.Fatalf("Discover: %v", err) } - if len(res.Verified) != 1 { - t.Fatalf("want 1 verified offer to Execute, got %d", len(res.Verified)) + if len(res.Verified()) != 1 { + t.Fatalf("want 1 verified offer to Execute, got %d", len(res.Verified())) } - verified := res.Verified[0] + verified := res.Verified()[0] // Fixed idempotency key pins the nonce across both Execute calls so the second // is a genuine replay of the first. diff --git a/sdk/go/core/discovery.go b/sdk/go/core/discovery.go new file mode 100644 index 00000000..28923c9b --- /dev/null +++ b/sdk/go/core/discovery.go @@ -0,0 +1,128 @@ +package core + +import ( + "context" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" +) + +// The shape both discovery verbs return. +// +// A discovery call is per-URI: an agent asks about several resources at once and +// the answer comes back grouped, one group per requested URI, each either +// carrying offers or carrying a typed reason it carries none. A flat list cannot +// express that. It has nowhere to put the reason, and a refused URI vanishes +// entirely — its group holds no offer, so nothing survives to say which resource +// was refused or why. +// +// That distinction is the point of the vocabulary: "not in the catalogue" means +// give up, "scope insufficient" means acquire an entitlement and retry, and +// "content blocked" means never retry. Flattened, all three read as "found +// nothing". +// +// The fail-closed {verified, rejected} split is preserved inside each group, +// through the same Verifier — not a second verification path. + +// OfferGroupResult is one requested URI's answer: the verified/rejected split for +// that URI, plus why it is empty when it is. +type OfferGroupResult struct { + // URI is the resource this group answers for, echoed by the responder. + URI string + // AbsenceReason says why this URI yielded no offers. Nil means the responder + // stated no reason — which is a legitimate answer, not an omission: where the + // existence of a resource must itself stay hidden, a responder MAY withhold + // the reason rather than confirm the resource exists. Distinguishing "absent" + // from the unspecified enum value is why this is a pointer. + AbsenceReason *rampv1.OfferAbsenceReason + // DiscoveryMethod is how the responder found this URI, when it said. + DiscoveryMethod *rampv1.DiscoveryMethod + // RestrictionFilters names the restriction axes that drove a convenience + // pre-filter, when the absence reason is a restriction filter. Advisory + // diagnostics, not an enforcement verdict — but they tell an agent which axis + // to vary on a retry. + RestrictionFilters []rampv1.RestrictionKind + // Result is the fail-closed split for this URI's offers. + Result +} + +// DiscoveryResult is what Discover and Resolve return: one group per requested +// URI, plus the whole-call refusal when the call as a whole yielded nothing. +type DiscoveryResult struct { + // Groups is one entry per requested URI, in the order the responder returned. + Groups []OfferGroupResult + // AbsenceReason says why the CALL as a whole yielded nothing. It is set only + // on that path — when any group carries offers it stays nil, and the per-URI + // causes ride on each group instead. + // + // Only a Broker resolve can set it: the Exchange's own discovery response has + // no whole-call reason field, so from Discover this is always nil and the + // per-URI groups carry everything the responder said. + AbsenceReason *rampv1.OfferAbsenceReason + // Exchange is the canonical domain of the responding Exchange. Empty from a + // Broker resolve, whose response names no single Exchange — each offer carries + // its own issuing domain. + Exchange string + // RateLimit is the caller's rate-limit standing, when the responder reported + // it, so an agent can throttle before a fan-out meets a hard limit. Nil from a + // Broker resolve, whose message has no such field. + RateLimit *rampv1.RateLimitInfo +} + +// Verified flattens every verified offer across all groups, for a caller that +// does not care which URI an offer answers. +// +// It is a convenience over Groups, never a substitute. A URI that was REFUSED +// contributes nothing here — it has no offer to contribute — so a caller that +// reads only this cannot tell a refusal from a resource it never asked about. +// That is exactly the information Groups exists to keep. +func (d DiscoveryResult) Verified() []VerifiedOffer { + var out []VerifiedOffer + for _, g := range d.Groups { + out = append(out, g.Result.Verified...) + } + return out +} + +// Rejected flattens every rejected offer across all groups, with the reason each +// failed. The same caveat as Verified applies: a URI that yielded no offers at +// all is not a rejection and appears only in Groups. +func (d DiscoveryResult) Rejected() []RejectedOffer { + var out []RejectedOffer + for _, g := range d.Groups { + out = append(out, g.Result.Rejected...) + } + return out +} + +// SortGroups verifies every group's offers through this one Verifier and returns +// the per-URI results, preserving each group's URI and its typed reasons. +// +// One Verifier sorts every group deliberately: it is stateless apart from the +// injected resolver and clock, so a fresh one per group would mean N resolver +// caches and N clock readings for a single logical answer. +func (v Verifier) SortGroups(ctx context.Context, groups []*rampv1.OfferGroup) []OfferGroupResult { + if len(groups) == 0 { + return nil + } + out := make([]OfferGroupResult, 0, len(groups)) + for _, g := range groups { + // A nil element is not a group with no offers — it is nothing at all, and + // surfacing it as an empty URI would invent an answer the responder never + // gave. Skipped rather than dereferenced, since an in-process caller can + // build the slice by hand. + if g == nil { + continue + } + out = append(out, OfferGroupResult{ + URI: g.GetUri(), + // The raw fields, not the getters: a getter collapses an absent + // optional enum to its zero value, which would make "the responder + // said nothing" indistinguishable from "unspecified". + AbsenceReason: g.AbsenceReason, + DiscoveryMethod: g.DiscoveryMethod, + RestrictionFilters: g.GetRestrictionFilters(), + Result: v.Sort(ctx, g.GetOffers()), + }) + } + return out +} diff --git a/sdk/go/core/doc.go b/sdk/go/core/doc.go index ace5c05c..7f2234e5 100644 --- a/sdk/go/core/doc.go +++ b/sdk/go/core/doc.go @@ -1,10 +1,16 @@ // Package core is the transport-neutral L2 substance of the RAMP SDK: the unified // offer Verifier, the fail-closed {verified, rejected} contract (Result), the -// unforgeable VerifiedOffer compile guard with the loud RejectedOffer.Unsafe -// escape, the client signing http.RoundTripper (NewSigningTransport), the injected -// ReplayStore interface, and the neutral request-id mint/middleware — all built on -// the sdk/go/helpers L1 primitives with net/http as the only transport dependency -// (ADR-020 §2/§3). +// per-URI discovery shape both discovery verbs return (DiscoveryResult / +// OfferGroupResult, sorted by Verifier.SortGroups), the unforgeable VerifiedOffer +// compile guard with the loud RejectedOffer.Unsafe escape, the client signing +// http.RoundTripper (NewSigningTransport), the injected ReplayStore interface, and +// the neutral request-id mint/middleware — all built on the sdk/go/helpers L1 +// primitives with net/http as the only transport dependency (ADR-020 §2/§3). +// +// The discovery shape is grouped rather than flat because a discovery call is +// per-URI: a URI that yielded nothing has no offer to carry its identity back, so +// flattening would erase both which resource was refused and the typed reason it +// was — and those reasons are different agent actions, not shades of "none". // // core imports NOTHING from connectrpc: RAMP is an HTTP protocol, and this package // only needs net/http, so a team on grpc-go / plain net/http / any transport can diff --git a/sdk/go/helpers/acceptance.go b/sdk/go/helpers/acceptance.go index 25de9dba..cba96cfe 100644 --- a/sdk/go/helpers/acceptance.go +++ b/sdk/go/helpers/acceptance.go @@ -1,6 +1,7 @@ package helpers import ( + "context" "crypto/ed25519" "encoding/hex" "errors" @@ -102,6 +103,35 @@ func SignOfferAcceptance(priv ed25519.PrivateKey, offer *rampv1.Offer, requester return hex.EncodeToString(ed25519.Sign(priv, payload)), nil } +// SignOfferAcceptanceWith signs the canonical acceptance payload through an +// injected Signer, so an application whose key lives in a KMS or an HSM can +// produce an acceptance without ever handing the key over. It is the form the +// SDK's own client uses; SignOfferAcceptance above is the direct-key form for a +// caller that already holds the bytes. Both cover the identical payload from +// CanonicalAcceptanceBytes, so the two are interchangeable on the wire. +// +// The acceptance is signed with the same key that signs the caller's requests: +// its thumbprint becomes the delivery URL's agent_id, which is what the edge +// later requires proof of possession of. +func SignOfferAcceptanceWith(ctx context.Context, signer Signer, offer *rampv1.Offer, requester *rampv1.Requester, idempotencyKey string) (string, error) { + if signer == nil { + return "", errors.New("helpers: acceptance signer is nil") + } + if signer.Algorithm() != AlgEd25519 { + return "", fmt.Errorf("%w: offer acceptance requires %q, signer offers %q", + ErrUnsupportedAlgorithm, AlgEd25519, signer.Algorithm()) + } + payload, err := CanonicalAcceptanceBytes(offer, requester, idempotencyKey) + if err != nil { + return "", err + } + sig, err := signer.Sign(ctx, payload) + if err != nil { + return "", fmt.Errorf("helpers: sign offer acceptance: %w", err) + } + return hex.EncodeToString(sig), nil +} + // VerifyOfferAcceptance verifies signatureHex (an AgentAcceptance.signature) // against the canonical acceptance payload for the offer, using pub. It returns // ErrAcceptanceSignatureInvalid on any mismatch (wrong key or tampered binding). diff --git a/sdk/go/helpers/gen_vectors_test.go b/sdk/go/helpers/gen_vectors_test.go index 0cb43a2a..6f6cf5b5 100644 --- a/sdk/go/helpers/gen_vectors_test.go +++ b/sdk/go/helpers/gen_vectors_test.go @@ -167,34 +167,39 @@ func replaceFirst(s, old, new string) string { return s } -// signEdgePoP produces an edge-form RFC 9421 GET PoP over exactly the two -// covered components the edge verifier (src/edge/src/pop.ts) checks: @method and -// @target-uri. It reuses the oracle's buildSignatureBase / signatureInputInner so -// the emitted Signature-Input header and the signed bytes are the canonical Go -// byte contract. keyid is the agent thumbprint (the 3-way identity anchor). +// signEdgePoP produces the positive PoP vector through the SHIPPED signer, so the +// goldens the other two languages replay are the bytes production emits rather +// than a second implementation living in a test. keyid is the agent thumbprint +// (the 3-way identity anchor). func signEdgePoP(t *testing.T, priv ed25519.PrivateKey, keyid, method, rawURL string, created, expires int64) (sigInput, sig string) { t.Helper() - req, err := http.NewRequest(method, rawURL, nil) + signer, err := NewEd25519Signer(keyid, priv) if err != nil { - t.Fatalf("new request: %v", err) + t.Fatalf("build agent-binding signer: %v", err) } - params := sigParams{ - Label: "sig1", - Covered: plainComponents("@method", "@target-uri"), - KeyID: keyid, - Alg: AlgEd25519, - Created: created, - Expires: expires, + pub, ok := priv.Public().(ed25519.PublicKey) + if !ok { + t.Fatal("signing key has no ed25519 public half") } - base, err := buildSignatureBase(req, params) + binding, err := SignAgentBinding(context.Background(), signer, pub, PoPOptions{ + URL: rawURL, KeyID: keyid, Created: created, Expires: expires, Method: method, + }) if err != nil { - t.Fatalf("build signature base: %v", err) + t.Fatalf("sign agent binding: %v", err) } - raw := ed25519.Sign(priv, []byte(base)) - inner := signatureInputInner(params) - // Edge Signature-Input header carries "sig1=" + inner list; Signature carries - // the std-base64 byte string (label=:...:), matching pop.ts parseSignature. - return params.Label + "=" + inner, params.Label + "=:" + base64.StdEncoding.EncodeToString(raw) + ":" + return binding.SignatureInput, binding.Signature +} + +// signEdgePoPMispairedKey hand-builds a proof that declares one keyid while +// signing with a DIFFERENT key — the input the shipped signer refuses outright, +// because refusing it is the whole point of the thumbprint check. The negative +// vector therefore cannot come from the signer, and building it here is what +// keeps it honest: it reproduces what a hostile fetcher would actually put on the +// wire, which is the thing the edge verifier has to reject. +func signEdgePoPMispairedKey(priv ed25519.PrivateKey, declaredKeyID, method, rawURL string, created, expires int64) (sigInput, sig string) { + params := popSignatureParams(declaredKeyID, created, expires) + raw := ed25519.Sign(priv, []byte(popSignatureBase(method, rawURL, params))) + return popLabel + "=" + params, popLabel + "=:" + base64.StdEncoding.EncodeToString(raw) + ":" } func buildPopVectors(t *testing.T) []popVector { @@ -229,7 +234,7 @@ func buildPopVectors(t *testing.T) []popVector { // thumbprint != agent_id, so the 3-way identity check rejects it. wrongPriv := ed25519.NewKeyFromSeed(fixedSeed(0x44)) wrongPub := wrongPriv.Public().(ed25519.PublicKey) - wrongInput, wrongSig := signEdgePoP(t, wrongPriv, agentTP, method, url, created, expires) + wrongInput, wrongSig := signEdgePoPMispairedKey(wrongPriv, agentTP, method, url, created, expires) wrongPresented := b64urlNoPad(wrongPub) return []popVector{ diff --git a/sdk/go/helpers/hosts.go b/sdk/go/helpers/hosts.go new file mode 100644 index 00000000..fdd4a444 --- /dev/null +++ b/sdk/go/helpers/hosts.go @@ -0,0 +1,175 @@ +package helpers + +import ( + "errors" + "fmt" + "net/url" + "strings" +) + +// Host predicates for the routing checks that precede a signed call to an +// address a network party named. +// +// Both exist for the same reason: a value that arrives inside an offer, or +// inside a manifest that offer pointed at, is about to be concatenated into a URL +// or dialed directly. Neither check is about the network — they are pure string +// work, which is why they sit in the IO-free tier and can run before anything is +// fetched. + +// ErrInvalidHost signals a reference that cannot be read as a host at all. +var ErrInvalidHost = errors.New("helpers: reference is not a usable host") + +// HostOf extracts the host (including any port) from a bare domain, a host:port +// pair, or a full URL. A ref with no scheme is parsed as though it carried https, +// since a bare domain is otherwise indistinguishable from a path. +func HostOf(ref string) (string, error) { + parsed, _, err := parseRef(ref) + if err != nil { + return "", err + } + return parsed.Host, nil +} + +// parseRef reads a bare domain, a host:port pair, or a full URL into a URL with a +// non-empty Host. A ref with no scheme is parsed as though it carried https, since +// a bare domain is otherwise indistinguishable from a path. One parse behind both +// host predicates, so neither can disagree with the other about what a reference +// even is. +// +// hadScheme reports whether the caller actually WROTE a scheme, which the assumed +// https above would otherwise hide. Anchoring needs that: a scheme decides which +// port counts as the default, so a value that named none must not be treated as +// having named https. +func parseRef(ref string) (parsed *url.URL, hadScheme bool, err error) { + if strings.TrimSpace(ref) == "" { + return nil, false, fmt.Errorf("%w: empty reference", ErrInvalidHost) + } + toParse := ref + hadScheme = strings.Contains(ref, "://") + if !hadScheme { + toParse = "https://" + ref + } + parsed, err = url.Parse(toParse) + if err != nil { + return nil, false, fmt.Errorf("%w: %q: %w", ErrInvalidHost, ref, err) + } + if parsed.Host == "" { + return nil, false, fmt.Errorf("%w: %q has no host", ErrInvalidHost, ref) + } + return parsed, hadScheme, nil +} + +// IsBareHost reports whether ref is EXACTLY a host — nothing a URL could carry +// besides the authority. It answers false for a ref with a scheme, userinfo, a +// path, a query, or a fragment, because HostOf had to strip something to reach +// the host. A port is NOT a strip: "exchange.example:8443" is a bare host, and +// the well-known resolver concatenates host-with-port unchanged. +// +// It exists for the callers that hand a network-supplied domain to code which +// builds a URL by concatenation. There, narrowing a rich reference to its host is +// the wrong repair: the value was never a domain, and accepting it silently means +// the far side chose the path that gets fetched, not just the host it is fetched +// from. Comparing against the extracted host is what makes the rejection +// structural, rather than a blocklist of the separators anyone thought to name. +func IsBareHost(ref string) (bool, error) { + host, err := HostOf(ref) + if err != nil { + return false, err + } + // A trailing colon parses as a host with an empty port and would otherwise + // compare equal to itself. It is not a domain anyone meant to write, and the + // callers here concatenate the value into a URL, so it is refused rather than + // quietly normalized away. + if strings.HasSuffix(host, ":") { + return false, nil + } + return host == ref, nil +} + +// HostAnchored reports whether candidate is anchored to anchor — the same host +// and port, or a subdomain of that host on that port. Either side may be a bare +// domain, a host:port pair, or a full URL; a reference that does not parse is +// returned as an error, which callers treat as "not anchored". +// +// The use is checking a value a remote document supplied against the host that +// served that document: it may point at itself or at one of its own subdomains, +// and nothing else. Without it, a host could redirect a signed request — or a +// revocation poll — to an unrelated third-party address that a dial-time address +// guard would happily allow, because the address is perfectly public. +// +// The PORT is part of the comparison. What is being anchored is a place a signed +// call is sent, and a different port is a different service — one the party that +// published the anchor need not control. An Exchange reachable on a non-default +// port says so on both sides: the port belongs in the value the offer names as +// much as in the endpoint the manifest advertises. +// +// A DEFAULT port and its omission are the same port, so https://x, https://x:443 +// and x all anchor to one another. url.Parse does not materialize an implicit +// port, and refusing an operator who merely wrote :443 out in full would be a +// spelling check wearing a security check's clothes. +// +// The SCHEME is still not compared. Whether a leg may run in the clear is the +// guarded transport's decision, made in one place from one flag. Its only job here +// is choosing which port counts as the default — and a side that NAMED no scheme +// borrows the other's for that purpose, rather than being assumed to mean https. +// +// That last part is load-bearing, not a nicety. Both anchors in this SDK arrive +// schemeless: a WBA directory's authority and an Offer.exchange host are bare +// host[:port] values. Assuming https for them meant an anchor of "a.example:80" +// kept its port (80 is not https's default) while the candidate +// "http://a.example:80" folded it away — the same authority reaching two answers, +// which silently un-anchored every plaintext directory that spelled :80 in full. +func HostAnchored(anchor, candidate string) (bool, error) { + anchorURL, anchorHadScheme, err := parseRef(anchor) + if err != nil { + return false, fmt.Errorf("anchor host: %w", err) + } + candidateURL, candidateHadScheme, err := parseRef(candidate) + if err != nil { + return false, fmt.Errorf("candidate host: %w", err) + } + anchorScheme, candidateScheme := anchorURL.Scheme, candidateURL.Scheme + if !anchorHadScheme { + anchorScheme = candidateScheme + } + if !candidateHadScheme { + candidateScheme = anchorScheme + } + // Compared as two values rather than one joined string. Joined, the label + // boundary below would have to find ".a.com" at the end of "sub.a.com:8443" + // and would refuse a subdomain for having a port — the right answer reached + // through the wrong comparison is still the wrong comparison. + return sameOrSubdomain(anchorURL.Hostname(), candidateURL.Hostname()) && + canonicalPort(anchorScheme, anchorURL.Port()) == + canonicalPort(candidateScheme, candidateURL.Port()), nil +} + +// defaultPorts is the port a scheme reaches when none is written. +var defaultPorts = map[string]string{"http": "80", "https": "443"} + +// canonicalPort renders "the same port" as one string, so a port written out in +// full and the same port left implicit compare equal. An unknown scheme has no +// default to fold, so its port is kept verbatim. +func canonicalPort(scheme, port string) string { + if port == "" { + return "" + } + if def, ok := defaultPorts[strings.ToLower(scheme)]; ok && port == def { + return "" + } + return port +} + +// sameOrSubdomain reports whether candidate equals anchor or is a subdomain of +// it. Comparison is case-insensitive and tolerant of a trailing root dot. A +// subdomain match requires a full dot-delimited label boundary, so "evil-a.com" +// is NOT treated as a subdomain of "a.com" — the check a bare suffix match gets +// wrong, and the one an attacker registers a domain to exploit. +func sameOrSubdomain(anchor, candidate string) bool { + a := strings.ToLower(strings.TrimSuffix(anchor, ".")) + c := strings.ToLower(strings.TrimSuffix(candidate, ".")) + if a == "" { + return false + } + return c == a || strings.HasSuffix(c, "."+a) +} diff --git a/sdk/go/helpers/hosts_test.go b/sdk/go/helpers/hosts_test.go new file mode 100644 index 00000000..d4600da9 --- /dev/null +++ b/sdk/go/helpers/hosts_test.go @@ -0,0 +1,210 @@ +package helpers_test + +import ( + "strings" + "testing" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" +) + +func TestIsBareHost(t *testing.T) { + tests := map[string]struct { + ref string + want bool + wantErr bool + }{ + "plain domain": {"exchange.example", true, false}, + "subdomain": {"eu.exchange.example", true, false}, + "host with port": {"exchange.example:8443", true, false}, + "trailing root dot": {"exchange.example.", true, false}, + "https scheme": {"https://exchange.example", false, false}, + "http scheme": {"http://exchange.example", false, false}, + "path": {"exchange.example/reports", false, false}, + "root path": {"exchange.example/", false, false}, + "query": {"exchange.example?x=1", false, false}, + "fragment": {"exchange.example#frag", false, false}, + "userinfo": {"agent@exchange.example", false, false}, + "scheme and path": {"https://exchange.example/v1", false, false}, + "trailing colon": {"exchange.example:", false, false}, + "empty": {"", false, true}, + "whitespace": {" ", false, true}, + "control character in ref": {"exchange.example\n", false, true}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + got, err := helpers.IsBareHost(tc.ref) + if tc.wantErr { + if err == nil { + t.Fatalf("IsBareHost(%q) = %v, want an error", tc.ref, got) + } + return + } + if err != nil { + t.Fatalf("IsBareHost(%q): %v", tc.ref, err) + } + if got != tc.want { + t.Errorf("IsBareHost(%q) = %v, want %v", tc.ref, got, tc.want) + } + }) + } +} + +func TestHostAnchored(t *testing.T) { + tests := map[string]struct { + anchor string + candidate string + want bool + wantErr bool + }{ + "same host": {"a.com", "a.com", true, false}, + "subdomain": {"a.com", "cdn.a.com", true, false}, + "deep subdomain": {"a.com", "x.y.a.com", true, false}, + "candidate is a url": {"a.com", "https://cdn.a.com/v1/reports", true, false}, + "anchor is a url": {"https://a.com", "cdn.a.com", true, false}, + "case insensitive": {"A.com", "CDN.a.COM", true, false}, + "trailing root dot": {"a.com.", "cdn.a.com.", true, false}, + "label boundary not a prefix": {"a.com", "evil-a.com", false, false}, + "suffix without boundary": {"a.com", "xa.com", false, false}, + "unrelated host": {"a.com", "b.com", false, false}, + "parent is not anchored": {"cdn.a.com", "a.com", false, false}, + "empty anchor": {"", "a.com", false, true}, + "empty candidate": {"a.com", "", false, true}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + got, err := helpers.HostAnchored(tc.anchor, tc.candidate) + if tc.wantErr { + if err == nil { + t.Fatalf("HostAnchored(%q, %q) = %v, want an error", tc.anchor, tc.candidate, got) + } + return + } + if err != nil { + t.Fatalf("HostAnchored(%q, %q): %v", tc.anchor, tc.candidate, err) + } + if got != tc.want { + t.Errorf("HostAnchored(%q, %q) = %v, want %v", tc.anchor, tc.candidate, got, tc.want) + } + }) + } +} + +// The port IS part of the comparison. What is being anchored is a place a signed +// call is sent, and a different port is a different service — one the party that +// published the anchor need not control. +// +// A DEFAULT port and its omission are the same port, which is the half worth +// pinning: url.Parse does not materialize an implicit port, so a naive comparison +// would refuse an operator who merely wrote :443 out in full. The SCHEME is still +// not compared, and the default-port folding is scheme-relative precisely so it +// cannot become a scheme check by accident. +func TestHostAnchored_ComparesThePort(t *testing.T) { + cases := map[string]struct { + anchor, candidate string + want bool + }{ + "same name, different ports": {"a.com:8443", "a.com:9000", false}, + "bare anchor, ported endpoint": {"a.com", "https://a.com:8443/v1", false}, + "subdomain on an unnamed port": {"a.com", "https://cdn.a.com:8443/v1", false}, + "default written out": {"a.com", "https://a.com:443/v1", true}, + "default written on the anchor": {"a.com:443", "https://a.com/v1", true}, + "http default written out": {"a.com", "http://a.com:80/v1", true}, + "scheme alone does not divide": {"a.com", "http://a.com/v1", true}, + "same non-default port": {"a.com:8443", "https://a.com:8443/v1", true}, + "subdomain on the same port": {"a.com:8443", "https://cdn.a.com:8443/v1", true}, + "label boundary still holds": {"a.com", "https://evil-a.com:8443", false}, + "port does not soften a bad label": {"a.com:8443", "https://evil-a.com:8443", false}, + // A schemeless anchor borrows the candidate's scheme to decide which port is + // the default. Both anchors in this SDK arrive schemeless — a WBA directory's + // authority, an Offer.exchange host — so assuming https for them made + // "a.example:80" keep its port while "http://a.example:80" folded the same + // port away, and the two stopped anchoring to each other. + "plaintext anchor, port spelled both sides": {"a.example:80", "http://a.example:80/rev", true}, + "plaintext anchor, port spelled once": {"a.example:80", "http://a.example/rev", true}, + // Borrowing decides only WHICH port is default; it never makes two different + // ports equal. 443 is not http's default, and 80 is not https's. + "default of one scheme is not the other's": {"a.com:443", "http://a.com:80/v1", false}, + "and not in the other direction": {"a.com:80", "https://a.com:443/v1", false}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + anchored, err := helpers.HostAnchored(tc.anchor, tc.candidate) + if err != nil { + t.Fatalf("HostAnchored(%q, %q): %v", tc.anchor, tc.candidate, err) + } + if anchored != tc.want { + t.Errorf("HostAnchored(%q, %q) = %v, want %v", + tc.anchor, tc.candidate, anchored, tc.want) + } + }) + } +} + +func TestRedactURL(t *testing.T) { + tests := map[string]struct{ raw, want string }{ + "strips the signed query": { + "https://cdn.example/doc?sig=abc&kid=ex.v1&exp=1700000600&agent_id=tp", + "https://cdn.example/doc", + }, + "strips userinfo and fragment": { + "https://agent:secret@cdn.example/doc?sig=abc#frag", + "https://cdn.example/doc", + }, + "no query is unchanged": {"https://cdn.example/doc", "https://cdn.example/doc"}, + "unparseable yields none": {"https://cdn.example/\x7f\x00", ""}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + if got := helpers.RedactURL(tc.raw); got != tc.want { + t.Errorf("RedactURL() = %q, want %q", got, tc.want) + } + }) + } +} + +// The credential lives in the query, so the one thing this must never do is leak +// a signature into whatever reads the redacted value. +func TestRedactURL_NeverCarriesTheCredential(t *testing.T) { + const signed = "https://cdn.example/doc?agent_id=tp&exp=1700000600&kid=ex.v1&sig=SUPERSECRETSIGNATURE" + got := helpers.RedactURL(signed) + for _, leak := range []string{"SUPERSECRETSIGNATURE", "sig=", "agent_id=", "kid=", "exp="} { + if got != "" && strings.Contains(got, leak) { + t.Errorf("redacted URL %q still carries %q", got, leak) + } + } +} + +func TestRetrievalAuthFailureReasonFromToken(t *testing.T) { + // Every token the map claims, resolved back to a non-zero typed reason. + for _, token := range []string{ + "expired", "missing_exp", "signature_mismatch", + "missing_agent_key", "keyid_mismatch", "thumbprint_mismatch", + "pop_missing_created", "pop_missing_exp", "pop_expired", "pop_sig_invalid", + } { + reason, ok := helpers.RetrievalAuthFailureReasonFromToken(token) + if !ok { + t.Errorf("token %q is not mapped", token) + continue + } + if reason == 0 { + t.Errorf("token %q mapped to the unspecified reason", token) + } + } +} + +// Tokens the edge can emit that are deliberately NOT promoted to a typed reason: +// "missing_sig" because both the URL check and the proof check emit it and the +// body does not say which ran, and the parse-level tokens because the enum has no +// value for them. An unmapped token still reaches the caller as a raw string. +func TestRetrievalAuthFailureReasonFromToken_RefusesTheAmbiguousAndUnknown(t *testing.T) { + for _, token := range []string{ + "missing_sig", + "bad_agent_key", "malformed_sig_input", "unsupported_alg", + "bad_covered_components", "pop_future_created", + "", "not_a_token", + } { + if reason, ok := helpers.RetrievalAuthFailureReasonFromToken(token); ok { + t.Errorf("token %q must not be promoted, got %v", token, reason) + } + } +} diff --git a/sdk/go/helpers/pop.go b/sdk/go/helpers/pop.go new file mode 100644 index 00000000..bb789ba5 --- /dev/null +++ b/sdk/go/helpers/pop.go @@ -0,0 +1,246 @@ +package helpers + +import ( + "context" + "crypto/ed25519" + "encoding/base64" + "errors" + "fmt" + "net/http" + "strings" +) + +// Agent-binding proof of possession for signed delivery URLs (ADR-013). +// +// When a signed URL carries an agent_id — the RFC 7638 thumbprint of the key +// that signed the offer acceptance — a code-capable edge requires the fetcher to +// prove possession of that key, fully offline. The proof is two headers on the +// GET: the raw public key in X-RAMP-Agent-Key, and an RFC 9421 signature over +// @method + @target-uri. The edge then enforces a three-way identity: +// +// agent_id (URL) == keyid (Signature-Input) == thumbprint(presented key) +// +// The last equality is the one that cannot be dropped. Verifying the signature +// against the presented key alone proves nothing — any actor could present their +// own key plus a valid self-signature. +// +// This is the SIGN face. The verify face ships in sdk/ts and sdk/python (the +// edge runs there); Go is the byte oracle both are pinned to, through +// testdata/pop-vectors.json. + +// AgentKeyHeader carries the raw Ed25519 public key the fetcher presents, as +// base64url with no padding. ADR-013 chose a dedicated header over an inline JWK +// in keyid: the edge hashes this value and requires the digest to equal the +// URL's agent_id, so a fetcher cannot present one key while naming another. +const AgentKeyHeader = "X-RAMP-Agent-Key" + +// popLabel is the only signature label this profile emits. Unlike the RAMP +// request profile there is no forwarding chain here — a delivery fetch is a +// single hop to the edge — so sigN>1 never arises and the label is fixed rather +// than computed. +const popLabel = "sig1" + +// ErrMissingTargetURI signals a proof requested without the URL it is meant to +// bind. @target-uri is half the covered set; signing without it would produce a +// proof valid for any URL the presented key is offered against, which is the +// replay this profile exists to stop. +var ErrMissingTargetURI = errors.New("helpers: missing target URI (required by the agent-binding profile)") + +// ErrKeyIDMismatch signals that the keyid does not match the RFC 7638 +// thumbprint of the key presented alongside it. The edge checks the same +// equality and answers thumbprint_mismatch, but by then the cause — a custody +// layer that paired a keyid with the wrong key — is several hops from the +// symptom. Refusing here names it at the source. +var ErrKeyIDMismatch = errors.New("helpers: keyid is not the thumbprint of the presented key") + +// ErrInvalidPoPInput signals a proof input that cannot be written into a +// signature base without changing its shape — a control byte in the method or the +// target URI, which the line-delimited base would read as a component boundary. +var ErrInvalidPoPInput = errors.New("helpers: proof input is not usable in a signature base") + +// isControlByte reports whether r is a C0 control or DEL. Applied to the two +// values written verbatim into the signature base. +func isControlByte(r rune) bool { return r < 0x20 || r == 0x7f } + +// PoPOptions carries what a delivery-URL proof of possession needs beyond the +// key material. Only URL, Created and Expires are required. +type PoPOptions struct { + // URL is the signed delivery URL, used VERBATIM as @target-uri — the exact + // bytes the Exchange minted, query parameters and all. + // + // This is a string and not a request value on purpose. The edge rebuilds the + // base from the raw request line it received, so the signing side must not + // route the URL through a parsed value first: doing so yields the DECODED + // path, expanding every percent-escape before it reaches the signed bytes. + // %2F is the sharpest case, since it decodes to a real separator and the + // signature would then cover a different path structure than the wire carried. + // The result is a proof that cannot verify, surfacing as a blanket 403 with no + // indication that the URL was the problem. + URL string + // KeyID is the RFC 9421 keyid: the RFC 7638 thumbprint of the presented key. + // It is the anchor of the three-way identity the edge enforces. Empty means + // "take the Signer's own key id". Either way it is cross-checked against the + // presented key before anything is signed. + KeyID string + // Created is the unix-seconds instant the proof was made. The edge rejects a + // created more than 300s in its own future, so a signer whose clock runs fast + // fails closed. + Created int64 + // Expires is the unix-seconds cutoff after which the proof is stale. Keep it + // short: the covered set is only method and URL, so within this window the + // proof is replayable by anyone who observes the request. + Expires int64 + // Method is the HTTP method being signed. Empty means GET. A signed URL is + // read-only in practice, but @method is covered precisely so a proof made for + // a GET cannot be lifted onto a write. + Method string +} + +// AgentBinding is the proof a fetcher attaches to a bound delivery request: the +// three header values, ready to apply. It is returned as values rather than +// written onto a request so a caller can sign before it has built one, so this +// tier stays free of any dialing surface, and so the emitted bytes can be +// asserted directly against the shared cross-language vectors. +type AgentBinding struct { + // AgentKey is the X-RAMP-Agent-Key value: base64url, no padding. + AgentKey string + // SignatureInput is the full Signature-Input value, label included. + SignatureInput string + // Signature is the full Signature value, label included. The byte string + // inside the colons is STANDARD base64 while AgentKey above is base64url — an + // asymmetry that comes from RFC 8941's byte-sequence encoding meeting a header + // this profile defines itself, and one a verifier will not forgive. + Signature string +} + +// Apply writes the binding's three headers onto h. +func (b AgentBinding) Apply(h http.Header) { + h.Set(AgentKeyHeader, b.AgentKey) + h.Set("Signature-Input", b.SignatureInput) + h.Set("Signature", b.Signature) +} + +// popSignatureBase builds the RFC 9421 signature base for the agent-binding +// profile: the two covered components followed by the parameters line, joined +// with newlines and with no trailing newline. +// +// It takes raw strings rather than a request value for the reason PoPOptions.URL +// documents — the verbatim URL is the contract — and because the RAMP base +// builder in sigbase.go reconstructs @target-uri from a parsed URL's decoded +// path, which is exactly the transformation this profile must not perform. The +// two builders are pinned against each other in the internal base test: they +// agree on every ordinary delivery URL and diverge only where the path carries a +// percent-escape. +// +// rawParams is taken as given rather than rebuilt: on the verifying side it +// arrives verbatim in the Signature-Input header, so treating it as an opaque +// string here is what keeps the two faces symmetric. +func popSignatureBase(method, rawURL, rawParams string) string { + return strings.Join([]string{ + `"@method": ` + strings.ToUpper(method), + `"@target-uri": ` + rawURL, + `"@signature-params": ` + rawParams, + }, "\n") +} + +// popSignatureParams renders the @signature-params value for this profile. +// +// The parameter ORDER is part of the wire contract, not a formatting choice: the +// base is signed over this exact string and the verifier reconstructs it from the +// header as received. keyid, alg, created, expires — matching the TypeScript and +// Python faces. A generic RFC 9421 emitter tends to produce +// created;expires;alg;keyid instead, and that mismatch is the whole reason this +// profile builds its own base instead of routing through SignRequest. +func popSignatureParams(keyID string, created, expires int64) string { + return fmt.Sprintf( + `("@method" "@target-uri");keyid=%q;alg=%q;created=%d;expires=%d`, + keyID, AlgEd25519, created, expires, + ) +} + +// SignAgentBinding produces the proof of possession a fetcher presents when it +// retrieves a delivery URL bound to an agent key. The covered set is exactly +// @method and @target-uri: a GET carries no body to digest, and the signed URL is +// itself the credential, already covered by @target-uri, so there is no +// Authorization header worth binding. That is why SignRequest cannot serve this +// profile — it enforces the five-component RAMP set. +// +// The key arrives as a Signer plus the public half rather than as a raw private +// key: custody stays with the application (a KMS or HSM signer never exposes its +// key), and the public half must be supplied separately because the presented-key +// header carries it and a Signer cannot yield it. +func SignAgentBinding(ctx context.Context, signer Signer, pub ed25519.PublicKey, opts PoPOptions) (AgentBinding, error) { + keyID, err := validateAgentBinding(signer, pub, opts) + if err != nil { + return AgentBinding{}, err + } + method := opts.Method + if method == "" { + method = http.MethodGet + } + params := popSignatureParams(keyID, opts.Created, opts.Expires) + raw, err := signer.Sign(ctx, []byte(popSignatureBase(method, opts.URL, params))) + if err != nil { + return AgentBinding{}, fmt.Errorf("helpers: sign agent binding: %w", err) + } + return AgentBinding{ + AgentKey: base64.RawURLEncoding.EncodeToString(pub), + SignatureInput: popLabel + "=" + params, + Signature: popLabel + "=:" + base64.StdEncoding.EncodeToString(raw) + ":", + }, nil +} + +// validateAgentBinding checks every precondition of a bound fetch and returns the +// keyid the proof will declare. It runs before any signing so a caller that has +// mispaired a key and a keyid learns it here rather than from a 403 that names +// nothing. +func validateAgentBinding(signer Signer, pub ed25519.PublicKey, opts PoPOptions) (string, error) { + if signer == nil { + return "", errors.New("helpers: agent-binding signer is nil") + } + if len(pub) != ed25519.PublicKeySize { + return "", fmt.Errorf("helpers: ed25519 public key must be %d bytes, got %d", ed25519.PublicKeySize, len(pub)) + } + if opts.URL == "" { + return "", ErrMissingTargetURI + } + // The base is line-delimited and both values are written into it verbatim, so + // a control byte in either would add or split a line and the bytes signed here + // would stop describing the request the verifier reconstructs. Refused rather + // than escaped: no legitimate method or target URI contains one, and a + // signature base is the wrong place to be lenient. + // The offset is reported in BYTES, and named as such: strings.IndexFunc counts + // them, the Python and TS mirrors are written to count the same, and an + // unlabelled number under identical wording would otherwise mean three things. + if i := strings.IndexFunc(opts.URL, isControlByte); i >= 0 { + return "", fmt.Errorf("%w: target URI carries a control byte at byte %d", ErrInvalidPoPInput, i) + } + if i := strings.IndexFunc(opts.Method, isControlByte); i >= 0 { + return "", fmt.Errorf("%w: method carries a control byte at byte %d", ErrInvalidPoPInput, i) + } + // A proof carrying no created sails through as a signature claiming 1970: the + // edge bounds how far created may lead its clock, not how far it may lag, so + // freshness would be silently absent rather than loudly wrong. + if opts.Created <= 0 { + return "", ErrMissingCreated + } + if opts.Expires <= 0 { + return "", ErrMissingExpires + } + if signer.Algorithm() != AlgEd25519 { + return "", fmt.Errorf("%w: agent binding requires %q, signer offers %q", + ErrUnsupportedAlgorithm, AlgEd25519, signer.Algorithm()) + } + keyID := opts.KeyID + if keyID == "" { + keyID = signer.KeyID() + } + thumb, err := Thumbprint(pub) + if err != nil { + return "", fmt.Errorf("helpers: derive keyid thumbprint: %w", err) + } + if keyID != thumb { + return "", fmt.Errorf("%w: keyid %q, presented-key thumbprint %q", ErrKeyIDMismatch, keyID, thumb) + } + return keyID, nil +} diff --git a/sdk/go/helpers/pop_base_internal_test.go b/sdk/go/helpers/pop_base_internal_test.go new file mode 100644 index 00000000..64a043ec --- /dev/null +++ b/sdk/go/helpers/pop_base_internal_test.go @@ -0,0 +1,109 @@ +package helpers + +import ( + "net/http" + "strings" + "testing" +) + +// Why the agent-binding profile builds its own signature base instead of routing +// through the RAMP one. +// +// buildSignatureBase reconstructs @target-uri from a PARSED URL, which re-encodes +// the path and normalizes the host. The edge rebuilds its base from the raw +// request line it received, so for any URL that does not survive that round trip +// the two disagree, the signature cannot verify, and the edge answers an +// undifferentiated 403 that names nothing. +// +// These tests pin both halves: agreement where the URL is round-trip stable, and +// DISAGREEMENT where it is not. The disagreement cases are the reason +// popSignatureBase takes the verbatim string, so anyone tempted to "simplify" it +// back onto the request type fails here with the reason attached. + +// rampBaseFor renders the RAMP base builder's view of the same two covered +// components, so the two builders can be compared directly. +func rampBaseFor(t *testing.T, method, rawURL, keyID string, created, expires int64) string { + t.Helper() + req, err := http.NewRequest(method, rawURL, nil) + if err != nil { + t.Fatalf("build request for %q: %v", rawURL, err) + } + base, err := buildSignatureBase(req, sigParams{ + Label: popLabel, + Covered: plainComponents("@method", "@target-uri"), + KeyID: keyID, + Alg: AlgEd25519, + Created: created, + Expires: expires, + }) + if err != nil { + t.Fatalf("build ramp signature base for %q: %v", rawURL, err) + } + return base +} + +const ( + baseTestKeyID = "wSp1Ud8Phi33WBixrTcV5U38Q5JZ0VGpTAetgQUQw2k" + baseTestCreated = int64(1_700_000_000) + baseTestExpires = int64(1_700_000_600) +) + +func popBaseFor(rawURL string) string { + return popSignatureBase(http.MethodGet, rawURL, + popSignatureParams(baseTestKeyID, baseTestCreated, baseTestExpires)) +} + +// Where the two builders agree, and it is a wider set than one might assume: a +// URL value preserves host case, an explicit default port, and a raw space in the +// path, so none of those is a reason for this profile to own a builder. +func TestPopSignatureBase_AgreesWithRAMPBuilderOnStableURLs(t *testing.T) { + stable := map[string]string{ + "ordinary delivery url": "https://cdn.example/doc?agent_id=" + baseTestKeyID, + "full signed query": "https://cdn.example/a/b/c.html?exp=1700000600&kid=ex.v1&sig=abc", + "no query": "https://cdn.example/doc", + "mixed-case host": "https://CDN.Example/doc?agent_id=" + baseTestKeyID, + "explicit default port": "https://cdn.example:443/doc?agent_id=" + baseTestKeyID, + "raw space in path": "https://cdn.example/a b/doc?agent_id=" + baseTestKeyID, + } + for name, rawURL := range stable { + t.Run(name, func(t *testing.T) { + want := rampBaseFor(t, http.MethodGet, rawURL, baseTestKeyID, baseTestCreated, baseTestExpires) + if got := popBaseFor(rawURL); got != want { + t.Errorf("signature bases disagree on a stable URL\n got %q\nwant %q", got, want) + } + }) + } +} + +// The one shape that genuinely diverges, and the reason this profile signs the +// verbatim string: the RAMP builder reads the DECODED path off the URL value, so +// every percent-escape in the path is expanded before it reaches the signed bytes. +// +// %2F is the sharpest case — it decodes to a real separator, so the signature +// would cover a different path structure than the wire carried. %20 and a +// non-ASCII escape lose their encoding the same way. An edge rebuilding its base +// from the raw request line reconstructs the ESCAPED form, so a signer that +// signed the decoded one produces a proof that cannot verify, and the failure +// surfaces as a blanket 403 naming nothing. +func TestPopSignatureBase_DivergesOnAPercentEncodedPath(t *testing.T) { + tricky := map[string]string{ + "encoded separator in path": "https://cdn.example/a%2Fb/doc?agent_id=" + baseTestKeyID, + "encoded space in path": "https://cdn.example/a%20b/doc?agent_id=" + baseTestKeyID, + "encoded non-ascii in path": "https://cdn.example/caf%C3%A9?agent_id=" + baseTestKeyID, + } + for name, rawURL := range tricky { + t.Run(name, func(t *testing.T) { + ramp := rampBaseFor(t, http.MethodGet, rawURL, baseTestKeyID, baseTestCreated, baseTestExpires) + pop := popBaseFor(rawURL) + if pop == ramp { + t.Errorf("expected the two builders to disagree on %q, but both produced %q — "+ + "if the RAMP builder stopped decoding the path, this profile's verbatim contract needs rechecking", + rawURL, pop) + } + // The profile's own base must carry the URL exactly as handed over. + if want := `"@target-uri": ` + rawURL; !strings.Contains(pop, want) { + t.Errorf("agent-binding base did not carry the verbatim URL\n got %q\nwant it to contain %q", pop, want) + } + }) + } +} diff --git a/sdk/go/helpers/pop_test.go b/sdk/go/helpers/pop_test.go new file mode 100644 index 00000000..adf51e61 --- /dev/null +++ b/sdk/go/helpers/pop_test.go @@ -0,0 +1,296 @@ +package helpers_test + +import ( + "context" + "crypto/ed25519" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" +) + +// The agent-binding sign face. The positive assertion is a replay of the shared +// cross-language corpus: pop-vectors.json stores the seed that produced each +// stored signature, so re-signing the same inputs must reproduce the stored +// header bytes exactly (Ed25519 is deterministic). Python replays the same file +// against its own signer, which is what makes the three faces one contract. + +const popVectorsFile = "pop-vectors.json" + +type popVector struct { + Name string `json:"name"` + Method string `json:"method"` + URL string `json:"url"` + AgentID string `json:"agent_id"` + PresentedKeyB64URL string `json:"presented_key_b64url"` + SignerSeedHex string `json:"signer_seed_hex"` + SignatureInput string `json:"signature_input"` + Signature string `json:"signature"` + ExpectedValid bool `json:"expected_valid"` +} + +func loadPopVectors(t *testing.T) []popVector { + t.Helper() + raw, err := os.ReadFile(filepath.Join("testdata", popVectorsFile)) + if err != nil { + t.Fatalf("read pop vectors: %v", err) + } + var vectors []popVector + if err := json.Unmarshal(raw, &vectors); err != nil { + t.Fatalf("unmarshal pop vectors: %v", err) + } + if len(vectors) == 0 { + t.Fatal("pop vectors file has no vectors") + } + return vectors +} + +// validPopVector returns the one vector whose stored proof is expected to verify. +func validPopVector(t *testing.T) popVector { + t.Helper() + for _, v := range loadPopVectors(t) { + if v.ExpectedValid { + return v + } + } + t.Fatal("no valid pop vector found") + return popVector{} +} + +// signerFor rebuilds the keypair a vector was produced with. +func signerFor(t *testing.T, seedHex, keyID string) (helpers.Signer, ed25519.PublicKey) { + t.Helper() + seed, err := hex.DecodeString(seedHex) + if err != nil { + t.Fatalf("decode signer seed: %v", err) + } + priv := ed25519.NewKeyFromSeed(seed) + signer, err := helpers.NewEd25519Signer(keyID, priv) + if err != nil { + t.Fatalf("build signer: %v", err) + } + pub, ok := priv.Public().(ed25519.PublicKey) + if !ok { + t.Fatal("signing key has no ed25519 public half") + } + return signer, pub +} + +// parseWindow pulls created and expires back out of a stored Signature-Input, so +// the replay signs over the same window the vector was minted with. +func parseWindow(t *testing.T, signatureInput string) (created, expires int64) { + t.Helper() + var c, e int64 + for _, part := range strings.Split(signatureInput, ";") { + switch { + case strings.HasPrefix(part, "created="): + c = mustAtoi(t, strings.TrimPrefix(part, "created=")) + case strings.HasPrefix(part, "expires="): + e = mustAtoi(t, strings.TrimPrefix(part, "expires=")) + } + } + if c == 0 || e == 0 { + t.Fatalf("signature input carries no created/expires: %q", signatureInput) + } + return c, e +} + +func mustAtoi(t *testing.T, s string) int64 { + t.Helper() + var n int64 + for _, r := range s { + if r < '0' || r > '9' { + t.Fatalf("non-numeric signature parameter %q", s) + } + n = n*10 + int64(r-'0') + } + return n +} + +func TestSignAgentBinding_ReproducesSharedVector(t *testing.T) { + v := validPopVector(t) + signer, pub := signerFor(t, v.SignerSeedHex, v.AgentID) + created, expires := parseWindow(t, v.SignatureInput) + + got, err := helpers.SignAgentBinding(context.Background(), signer, pub, helpers.PoPOptions{ + URL: v.URL, KeyID: v.AgentID, Created: created, Expires: expires, Method: v.Method, + }) + if err != nil { + t.Fatalf("sign agent binding: %v", err) + } + if got.SignatureInput != v.SignatureInput { + t.Errorf("Signature-Input\n got %q\nwant %q", got.SignatureInput, v.SignatureInput) + } + if got.Signature != v.Signature { + t.Errorf("Signature\n got %q\nwant %q", got.Signature, v.Signature) + } + if got.AgentKey != v.PresentedKeyB64URL { + t.Errorf("X-RAMP-Agent-Key got %q, want %q", got.AgentKey, v.PresentedKeyB64URL) + } +} + +// The two encodings differ on purpose and a verifier will not forgive a swap: +// the presented key is base64url-no-pad, the RFC 8941 byte sequence is standard +// base64 inside colons. +func TestSignAgentBinding_EncodingAsymmetry(t *testing.T) { + v := validPopVector(t) + signer, pub := signerFor(t, v.SignerSeedHex, v.AgentID) + created, expires := parseWindow(t, v.SignatureInput) + + got, err := helpers.SignAgentBinding(context.Background(), signer, pub, helpers.PoPOptions{ + URL: v.URL, KeyID: v.AgentID, Created: created, Expires: expires, + }) + if err != nil { + t.Fatalf("sign agent binding: %v", err) + } + if strings.ContainsAny(got.AgentKey, "+/=") { + t.Errorf("agent key is not base64url-no-pad: %q", got.AgentKey) + } + if _, err = base64.RawURLEncoding.DecodeString(got.AgentKey); err != nil { + t.Errorf("agent key does not decode as base64url-no-pad: %v", err) + } + inner := strings.TrimSuffix(strings.TrimPrefix(got.Signature, "sig1=:"), ":") + if _, err = base64.StdEncoding.DecodeString(inner); err != nil { + t.Errorf("signature byte string does not decode as standard base64: %v", err) + } +} + +// The covered set is exactly the two components. content-digest and authorization +// are absent by design: a GET has no body, and the signed URL is itself the +// credential and is already covered by @target-uri. +func TestSignAgentBinding_CoversExactlyMethodAndTargetURI(t *testing.T) { + v := validPopVector(t) + signer, pub := signerFor(t, v.SignerSeedHex, v.AgentID) + created, expires := parseWindow(t, v.SignatureInput) + + got, err := helpers.SignAgentBinding(context.Background(), signer, pub, helpers.PoPOptions{ + URL: v.URL, KeyID: v.AgentID, Created: created, Expires: expires, + }) + if err != nil { + t.Fatalf("sign agent binding: %v", err) + } + if !strings.HasPrefix(got.SignatureInput, `sig1=("@method" "@target-uri");`) { + t.Errorf("covered set is not the agent-binding pair: %q", got.SignatureInput) + } + for _, banned := range []string{"content-digest", "authorization", "signature-agent"} { + if strings.Contains(got.SignatureInput, banned) { + t.Errorf("covered set carries %q, which this profile must not bind: %q", banned, got.SignatureInput) + } + } + // keyid, alg, created, expires — the order the verifiers reconstruct from. + wantOrder := []string{";keyid=", ";alg=", ";created=", ";expires="} + at := 0 + for _, token := range wantOrder { + idx := strings.Index(got.SignatureInput[at:], token) + if idx < 0 { + t.Fatalf("signature parameters out of order or missing %q: %q", token, got.SignatureInput) + } + at += idx + len(token) + } +} + +func TestAgentBinding_ApplyWritesTheThreeHeaders(t *testing.T) { + binding := helpers.AgentBinding{AgentKey: "key", SignatureInput: "sig1=()", Signature: "sig1=::"} + h := http.Header{} + binding.Apply(h) + + if got := h.Get(helpers.AgentKeyHeader); got != "key" { + t.Errorf("%s = %q, want %q", helpers.AgentKeyHeader, got, "key") + } + if got := h.Get("Signature-Input"); got != "sig1=()" { + t.Errorf("Signature-Input = %q", got) + } + if got := h.Get("Signature"); got != "sig1=::" { + t.Errorf("Signature = %q", got) + } + if len(h) != 3 { + t.Errorf("Apply wrote %d headers, want exactly 3: %v", len(h), h) + } +} + +// Every precondition is refused BEFORE anything is signed, so a mispaired key or +// an absent window is named here rather than surfacing as an undifferentiated 403. +func TestSignAgentBinding_RefusesBadPreconditions(t *testing.T) { + v := validPopVector(t) + signer, pub := signerFor(t, v.SignerSeedHex, v.AgentID) + created, expires := parseWindow(t, v.SignatureInput) + ok := helpers.PoPOptions{URL: v.URL, KeyID: v.AgentID, Created: created, Expires: expires} + + // A second keypair whose thumbprint is NOT the vector's agent_id. + otherSigner, otherPub := signerFor(t, strings.Repeat("44", ed25519.SeedSize), v.AgentID) + + tests := []struct { + name string + signer helpers.Signer + pub ed25519.PublicKey + opts helpers.PoPOptions + want error + }{ + {"nil signer", nil, pub, ok, nil}, + {"short public key", signer, pub[:16], ok, nil}, + {"empty target uri", signer, pub, withURL(ok, ""), helpers.ErrMissingTargetURI}, + {"missing created", signer, pub, withWindow(ok, 0, expires), helpers.ErrMissingCreated}, + {"missing expires", signer, pub, withWindow(ok, created, 0), helpers.ErrMissingExpires}, + {"keyid is not the presented key's thumbprint", otherSigner, otherPub, ok, helpers.ErrKeyIDMismatch}, + // Both values are written verbatim into a line-delimited signature base, + // so a control byte would add or split a component line and the signed + // bytes would stop describing the request a verifier reconstructs. + {"newline in the target uri", signer, pub, + withURL(ok, v.URL+"\n\"@authority\": evil.test"), helpers.ErrInvalidPoPInput}, + {"carriage return in the target uri", signer, pub, + withURL(ok, v.URL+"\r"), helpers.ErrInvalidPoPInput}, + {"newline in the method", signer, pub, + withMethod(ok, "GET\n\"@authority\": evil.test"), helpers.ErrInvalidPoPInput}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got, err := helpers.SignAgentBinding(context.Background(), tc.signer, tc.pub, tc.opts) + if err == nil { + t.Fatalf("expected a refusal, got binding %+v", got) + } + if tc.want != nil && !errors.Is(err, tc.want) { + t.Errorf("error %v does not match sentinel %v", err, tc.want) + } + }) + } +} + +// An absent keyid falls back to the Signer's own key id, which is the common +// case: the agent's signing key id already IS its thumbprint. +func TestSignAgentBinding_KeyIDDefaultsToTheSigner(t *testing.T) { + v := validPopVector(t) + signer, pub := signerFor(t, v.SignerSeedHex, v.AgentID) + created, expires := parseWindow(t, v.SignatureInput) + + got, err := helpers.SignAgentBinding(context.Background(), signer, pub, helpers.PoPOptions{ + URL: v.URL, Created: created, Expires: expires, + }) + if err != nil { + t.Fatalf("sign agent binding: %v", err) + } + if got.SignatureInput != v.SignatureInput { + t.Errorf("defaulted keyid changed the bytes\n got %q\nwant %q", got.SignatureInput, v.SignatureInput) + } +} + +func withURL(o helpers.PoPOptions, url string) helpers.PoPOptions { + o.URL = url + return o +} + +func withWindow(o helpers.PoPOptions, created, expires int64) helpers.PoPOptions { + o.Created, o.Expires = created, expires + return o +} + +func withMethod(o helpers.PoPOptions, method string) helpers.PoPOptions { + o.Method = method + return o +} diff --git a/sdk/go/helpers/retrievaltoken.go b/sdk/go/helpers/retrievaltoken.go new file mode 100644 index 00000000..bcc78fbe --- /dev/null +++ b/sdk/go/helpers/retrievaltoken.go @@ -0,0 +1,58 @@ +package helpers + +import ( + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" +) + +// The delivery edge answers a refused fetch with a small JSON body carrying its +// own refusal token — a string vocabulary, because the edge is a code-capable +// worker with no protobuf runtime. RetrievalAuthFailureReason is the typed +// counterpart, and ramp.proto records the token each value stands for. +// +// Mapping the token back onto the enum is what lets a fetch refusal reach a +// caller through the SAME typed vocabulary an RPC refusal does, instead of a +// string the caller has to substring-match. + +// retrievalAuthFailureTokens maps the edge's refusal token to its typed reason. +// +// Two tokens the edge can emit are deliberately ABSENT rather than guessed at: +// +// - "missing_sig" is emitted by BOTH checkers — the signed-URL check (no sig +// query parameter) and the proof check (no Signature header) — and the enum +// has a distinct value for each. The body does not say which ran, so mapping +// it would attribute the failure to a check that may not have fired. +// - the edge's parse-level tokens (a malformed signature input, an unsupported +// alg, a wrong covered set, a created too far in the future) have no enum +// value at all. +// +// An unmapped token still reaches the caller as the raw refusal string; only the +// typed reason is withheld, which is the honest outcome when the wire cannot say +// which failure occurred. +var retrievalAuthFailureTokens = map[string]rampv1.RetrievalAuthFailureReason{ + // Signed-URL checks. + "expired": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_URL_EXPIRED, + "missing_exp": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_URL_EXPIRY_MISSING, + "signature_mismatch": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_URL_SIGNATURE_MISMATCH, + // Proof-of-possession checks. + "missing_agent_key": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_AGENT_KEY_MISSING, + "keyid_mismatch": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_KEYID_MISMATCH, + "thumbprint_mismatch": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_THUMBPRINT_MISMATCH, + "pop_missing_created": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_PROOF_CREATED_MISSING, + "pop_missing_exp": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_PROOF_EXPIRY_MISSING, + "pop_expired": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_PROOF_EXPIRED, + "pop_sig_invalid": rampv1.RetrievalAuthFailureReason_RETRIEVAL_AUTH_FAILURE_REASON_PROOF_SIGNATURE_INVALID, +} + +// RetrievalAuthFailureReasonFromToken resolves a delivery edge's refusal token to +// its typed RetrievalAuthFailureReason. The second result is false for a token +// this SDK does not recognise or cannot attribute unambiguously — the caller then +// falls back to its own failure class, which it owns, rather than promoting a +// value the edge did not actually state. +// +// Fail-closed by construction: the token arrives in a body written by the host +// the fetch just went to, so an unrecognised value is never surfaced as a typed +// protocol reason. +func RetrievalAuthFailureReasonFromToken(token string) (rampv1.RetrievalAuthFailureReason, bool) { + reason, ok := retrievalAuthFailureTokens[token] + return reason, ok +} diff --git a/sdk/go/helpers/signedurl.go b/sdk/go/helpers/signedurl.go index 11d0391f..1ffaf27f 100644 --- a/sdk/go/helpers/signedurl.go +++ b/sdk/go/helpers/signedurl.go @@ -267,6 +267,27 @@ func (v VerifiedURL) CheckProofOfPossession(presentedPub ed25519.PublicKey) erro return nil } +// RedactURL reduces a signed URL to scheme://host/path, for a value headed +// somewhere more durable than the caller who already holds it — a log line, or an +// error an operator will read. +// +// url.URL.Redacted() is NOT the tool for this. It masks userinfo passwords, and a +// delivery URL carries its credential in the QUERY: sig, kid, exp and agent_id. +// Redacted() would pass the signature through untouched while reading like a +// redaction, which is worse than not redacting at all. +// +// An unparseable input yields "" rather than the original: a value that could not +// be sanitized is not one to emit. +func RedactURL(raw string) string { + parsed, err := url.Parse(raw) + if err != nil { + return "" + } + stripped := *parsed + stripped.RawQuery, stripped.Fragment, stripped.User = "", "", nil + return stripped.String() +} + // HashURL returns the SHA-256 digest of a signed URL (the // transaction_log.signed_url_hash value, 32 bytes). func HashURL(signed string) []byte { diff --git a/sdk/go/helpers/verify_maxage_internal_test.go b/sdk/go/helpers/verify_maxage_internal_test.go index 705452a7..35424888 100644 --- a/sdk/go/helpers/verify_maxage_internal_test.go +++ b/sdk/go/helpers/verify_maxage_internal_test.go @@ -43,7 +43,7 @@ func TestEnforceCreatedExpires_MaxAgeClamp(t *testing.T) { maxAge: 5 * time.Minute, wantErr: ErrSignatureLifetimeTooLong, }, { - name: "clamp: signer-chosen 10y window rejected under a minutes clamp", + name: "clamp: signer-chosen 10y window rejected under a minutes clamp", created: nowUnix, expires: nowUnix + int64((10 * 365 * 24 * time.Hour).Seconds()), maxAge: 5 * time.Minute, wantErr: ErrSignatureLifetimeTooLong, }, diff --git a/sdk/go/internal/endpointrule/endpointrule.go b/sdk/go/internal/endpointrule/endpointrule.go new file mode 100644 index 00000000..d1781fba --- /dev/null +++ b/sdk/go/internal/endpointrule/endpointrule.go @@ -0,0 +1,59 @@ +// Package endpointrule holds the one predicate that decides whether an endpoint a +// well-known manifest advertises may be used at all. +// +// The rule is normative — WellKnownManifest.endpoint states it as a MUST — and it +// is checked in two places for two different reasons: the resolver refuses to hand +// such an endpoint back, and the client refuses to send a signed call to one even +// when a caller injected its own resolver. Two call sites, two error vocabularies, +// but there must only ever be ONE predicate. Written twice it drifts, and this +// repo has watched a duplicated host predicate drift inside a single commit. +// +// Internal because Python and TypeScript will grow their own implementations +// rather than binding to a Go export, and the shared public surface belongs with +// that work. +package endpointrule + +import ( + "fmt" + "net/url" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" +) + +// Vet reports whether endpoint may be used for host — the host that served the +// manifest advertising it — returning nil when it may and a describing error when +// it may not. +// +// Two conditions refuse. The endpoint must be on host or a subdomain of it: the +// manifest is only as trustworthy as the host that served it, so an endpoint +// naming an unrelated host would let whoever answers for that document redirect a +// signed call to a party the offer's signature never covered. A dial-time address +// guard has no objection to an unrelated PUBLIC host, so nothing below this +// catches it. +// +// And the endpoint must carry no userinfo. The host comparison reads the +// authority's host and ignores any user:password before it, so credentials would +// otherwise pass the first check and then have net/http stamp an Authorization +// header the SDK never chose — on a leg that already carries the agent's own +// signature. +// +// The caller supplies the vocabulary: the errors here describe what was wrong, and +// each call site wraps them in whatever sentinel its own tier classifies on. +func Vet(host, endpoint string) error { + parsed, err := url.Parse(endpoint) + if err != nil { + return fmt.Errorf("host=%q endpoint=%q is not a URL: %w", host, endpoint, err) + } + if parsed.User != nil { + // Deliberately does not echo the endpoint: it carries the credential. + return fmt.Errorf("host=%q advertises an endpoint carrying userinfo", host) + } + anchored, err := helpers.HostAnchored(host, endpoint) + if err != nil { + return fmt.Errorf("host=%q endpoint=%q: %w", host, endpoint, err) + } + if !anchored { + return fmt.Errorf("host=%q advertises endpoint %q on a different host", host, endpoint) + } + return nil +} diff --git a/sdk/go/internal/failure/failure.go b/sdk/go/internal/failure/failure.go new file mode 100644 index 00000000..d47973d3 --- /dev/null +++ b/sdk/go/internal/failure/failure.go @@ -0,0 +1,80 @@ +// Package failure holds the rendering the SDK's two failure taxonomies share. +// +// The client's CallError and the content tier's FetchError are deliberately +// separate types over separate vocabularies — the content tier knows nothing +// about RPCs, and only one of them can decline to send. What is NOT deliberate is +// that they rendered themselves with byte-identical code, down to the reason a +// status with no name prints bare. One copy is what stops the two drifting into +// answering differently for the same failure. +// +// Internal on purpose: error prose is not a protocol concept and has no +// cross-language counterpart to mirror. +package failure + +import ( + "fmt" + "net/http" +) + +// Render builds the message both failure types print. +// +// prefix names the tier that failed (the packages differ, and a reader should be +// able to tell from the first word which one produced this); op names what was +// being attempted; kind is the failure class's own token. Status, reason and +// cause are each appended only when present, so a bare classification renders as +// a bare classification rather than as a string of empty parentheses. +func Render(prefix, op, kind string, status int, reason string, cause error) string { + msg := prefix + ": " + op + ": " + kind + if status != 0 { + // StatusText is empty for a code net/http does not know, and a bare + // "(HTTP 599 )" reads like a truncation. The number alone is the honest + // render. + if text := http.StatusText(status); text != "" { + msg += fmt.Sprintf(" (HTTP %d %s)", status, text) + } else { + msg += fmt.Sprintf(" (HTTP %d)", status) + } + } + if reason != "" { + msg += ": " + reason + } + if cause != nil { + msg += ": " + cause.Error() + } + return msg +} + +// ReasonOr returns the peer's own token when it sent one, otherwise the failure +// class the SDK assigned. The most specific machine-readable answer available, +// which is what a caller branching on a reason wants. +func ReasonOr(reason, kind string) string { + if reason != "" { + return reason + } + return kind +} + +// Name renders a classification constant, falling back to "unknown" for a value +// outside the set. Both taxonomies are integer enums with a name map, and both +// must answer for a value they do not know rather than print a number. +func Name[K comparable](names map[K]string, k K) string { + if s, ok := names[k]; ok { + return s + } + return "unknown" +} + +// RefuseRedirect builds the http.Client CheckRedirect both credentialed legs +// install: neither an RPC nor a bound fetch is ever legitimately redirected, and +// following one would re-sign for a target the peer chose. +// +// prefix names the tier and why names what may not be redirected, because the two +// legs decline for different reasons and a caller should read the right one. What +// is shared is the part that must NOT drift: the target is redacted rather than +// passed through url.URL.Redacted(), which masks userinfo passwords only and +// would render an attacker-chosen query into a log. +func RefuseRedirect(prefix, why string, redact func(string) string) func(*http.Request, []*http.Request) error { + return func(req *http.Request, _ []*http.Request) error { + return fmt.Errorf("%s: refusing redirect to %s: %s", prefix, redact(req.URL.String()), why) + } +} diff --git a/sdk/go/internal/lrucache/lrucache.go b/sdk/go/internal/lrucache/lrucache.go new file mode 100644 index 00000000..aff75679 --- /dev/null +++ b/sdk/go/internal/lrucache/lrucache.go @@ -0,0 +1,133 @@ +// Package lrucache is the one bounded, least-recently-used map the SDK's caching +// tiers share. +// +// Both callers key on a host an incoming offer named, so the key space is +// open-ended and caller-influenced — an unbounded map over one is somewhere an +// authenticated caller can make the process grow without limit. Both therefore +// need the same structure, and having written it twice the second copy carried +// the first's rationale paragraph verbatim. One implementation is what stops an +// eviction fix landing in one and not the other. +// +// Least-recently-used and not drop-the-whole-map: dropping empties the cache +// exactly when it is under most pressure, and it makes which entries survive a +// function of the order a caller names hosts — a property the caller controls. +// +// It is internal on purpose. A bounded map is not a protocol concept, so it has +// no cross-language counterpart to mirror and no business on the SDK's public +// surface. +package lrucache + +import ( + "container/list" + "sync" +) + +// Cache holds at most cap entries, evicting the least recently used to make room. +// The zero value is not usable; build one with New. +// +// Safe for concurrent use. Every method takes the same lock, including the loader +// GetOrCreate runs — so a value is constructed exactly once per key even when two +// callers race for it. +type Cache[K comparable, V any] struct { + cap int + mu sync.Mutex + order *list.List // front is most-recently-used; values are *entry[K, V] + entries map[K]*list.Element +} + +// entry is what the recency list holds: the key beside its value, so eviction can +// find the map key from the list element it is dropping. +type entry[K comparable, V any] struct { + key K + val V +} + +// New returns a cache bounded at cap entries. A cap below one is treated as one: +// a zero-capacity cache would evict what it just stored, which no caller wants and +// which would make every lookup a miss. +func New[K comparable, V any](cap int) *Cache[K, V] { + if cap < 1 { + cap = 1 + } + return &Cache[K, V]{ + cap: cap, + order: list.New(), + entries: make(map[K]*list.Element, cap), + } +} + +// Get returns the value stored for key and promotes it to most-recently-used. +func (c *Cache[K, V]) Get(key K) (V, bool) { + c.mu.Lock() + defer c.mu.Unlock() + el, ok := c.entries[key] + if !ok { + var zero V + return zero, false + } + c.order.MoveToFront(el) + return el.Value.(*entry[K, V]).val, true +} + +// Put stores val for key, promoting it and evicting the least-recently-used entry +// once the cache is full. An existing key is updated in place rather than +// consuming a second slot. +func (c *Cache[K, V]) Put(key K, val V) { + c.mu.Lock() + defer c.mu.Unlock() + c.put(key, val) +} + +// GetOrCreate returns the value stored for key, building it with make on a miss. +// +// make runs under the cache's lock, so a key is built once however many callers +// race. That is deliberate for the callers here — both build cheap in-process +// plumbing, never anything that dials — and it is why this is not a general-purpose +// memoizer. +func (c *Cache[K, V]) GetOrCreate(key K, make func(K) V) V { + c.mu.Lock() + defer c.mu.Unlock() + if el, ok := c.entries[key]; ok { + c.order.MoveToFront(el) + return el.Value.(*entry[K, V]).val + } + val := make(key) + c.put(key, val) + return val +} + +// put is the shared insert. The caller holds the lock. +func (c *Cache[K, V]) put(key K, val V) { + if el, ok := c.entries[key]; ok { + el.Value.(*entry[K, V]).val = val + c.order.MoveToFront(el) + return + } + if len(c.entries) >= c.cap { + if oldest := c.order.Back(); oldest != nil { + c.order.Remove(oldest) + delete(c.entries, oldest.Value.(*entry[K, V]).key) + } + } + c.entries[key] = c.order.PushFront(&entry[K, V]{key: key, val: val}) +} + +// Len reports how many entries the cache holds, and Has whether one is present +// without disturbing recency. +// +// Both exist for the eviction tests. The bound has no observable behaviour of its +// own other than a dial or fetch count, and counting those would mean mocking the +// very transport the suites deliberately never mock. +func (c *Cache[K, V]) Len() int { + c.mu.Lock() + defer c.mu.Unlock() + return len(c.entries) +} + +// Has reports whether key is currently held, without promoting it. +func (c *Cache[K, V]) Has(key K) bool { + c.mu.Lock() + defer c.mu.Unlock() + _, ok := c.entries[key] + return ok +} diff --git a/sdk/go/internal/lrucache/lrucache_test.go b/sdk/go/internal/lrucache/lrucache_test.go new file mode 100644 index 00000000..b75f6611 --- /dev/null +++ b/sdk/go/internal/lrucache/lrucache_test.go @@ -0,0 +1,91 @@ +package lrucache_test + +import ( + "fmt" + "testing" + + "github.com/RAMP-Protocol/protocol/sdk/go/internal/lrucache" +) + +// The bound is the security property worth pinning: both callers key on a host an +// incoming offer named, so the key space is open-ended and caller-influenced. +// +// One test now covers what two near-identical copies covered before, which is the +// point of having one implementation. + +func TestCache_EvictsLeastRecentlyUsedAtTheCap(t *testing.T) { + const cap = 8 + c := lrucache.New[string, int](cap) + + for i := range cap { + c.Put(fmt.Sprintf("k%d", i), i) + } + if got := c.Len(); got != cap { + t.Fatalf("Len() = %d, want %d", got, cap) + } + + // Read the oldest so it becomes the most recently used. If eviction were + // "drop the whole map", this would not survive the next insert — and which + // entries survived would be a function of the order a caller named keys. + if _, ok := c.Get("k0"); !ok { + t.Fatal("k0 should still be held") + } + + c.Put("overflow", 99) + + if got := c.Len(); got != cap { + t.Errorf("Len() = %d, want the cap to hold at %d", got, cap) + } + if !c.Has("k0") { + t.Error("the most-recently-used key was evicted; eviction must be least-recently-used") + } + if !c.Has("overflow") { + t.Error("the newly stored key is missing") + } + if c.Has("k1") { + t.Error("the least-recently-used key survived; it should have been evicted") + } +} + +// Re-putting a known key refreshes it in place rather than consuming a second +// slot — otherwise every TTL refresh would evict an unrelated entry. +func TestCache_PutOnAKnownKeyKeepsOneSlot(t *testing.T) { + c := lrucache.New[string, string](4) + c.Put("a", "one") + c.Put("a", "two") + if got := c.Len(); got != 1 { + t.Errorf("Len() = %d, want 1", got) + } + if v, ok := c.Get("a"); !ok || v != "two" { + t.Errorf("Get() = %q/%v, want the refreshed value", v, ok) + } +} + +// GetOrCreate builds once per key and reuses thereafter — the pool's contract, +// where the value is transport plumbing that must not be rebuilt per call. +func TestCache_GetOrCreateBuildsOncePerKey(t *testing.T) { + c := lrucache.New[string, *int](4) + builds := 0 + make := func(string) *int { builds++; n := builds; return &n } + + first := c.GetOrCreate("a", make) + second := c.GetOrCreate("a", make) + if first != second { + t.Error("a known key must reuse its built value") + } + if builds != 1 { + t.Errorf("builds = %d, want the loader to run once", builds) + } + if c.GetOrCreate("b", make); builds != 2 { + t.Errorf("builds = %d, want a fresh key to build", builds) + } +} + +// A cap below one would evict what it just stored, making every lookup a miss. +func TestCache_CapBelowOneStillHoldsAnEntry(t *testing.T) { + c := lrucache.New[string, int](0) + c.Put("a", 1) + if _, ok := c.Get("a"); !ok { + t.Error("a zero cap must be treated as one, not as a cache that stores nothing") + } +} diff --git a/sdk/go/resolvers/cachedofferkeyresolver.go b/sdk/go/resolvers/cachedofferkeyresolver.go index ab1fe7cf..fe854f0c 100644 --- a/sdk/go/resolvers/cachedofferkeyresolver.go +++ b/sdk/go/resolvers/cachedofferkeyresolver.go @@ -5,16 +5,25 @@ import ( "crypto/ed25519" "net" "net/http" - "sync" "time" rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/lrucache" ) // defaultOfferKeyTTL bounds how long a fetched offer-signing key is served from the // per-domain cache when no TTL is configured. const defaultOfferKeyTTL = 5 * time.Minute +// maxCachedOfferKeys bounds the per-domain key cache. The key is a domain off +// Offer.exchange, so which entries appear is driven by incoming offers — an +// open-ended, caller-influenced key space, and an unbounded map over one is +// somewhere a caller can make the process grow without limit. An expiry is a +// FRESHNESS check rather than a removal, so a stale entry holds its slot until +// something reclaims it. A real deployment sees a handful of exchanges. Mirrors the +// bound the endpoint cache and the per-origin client pool already carry. +const maxCachedOfferKeys = 256 + // OfferDirectoryFetcher fetches a domain's WBA identity directory. It is the // injected IO seam of CachedOfferKeyResolver: the default (NewWBADirectoryFetcher) // GETs scheme://domain[:port]/.well-known/http-message-signatures-directory through @@ -55,8 +64,10 @@ type CachedOfferKeyResolver struct { now func() time.Time revoked func(string) bool - mu sync.Mutex - cache map[string]offerKeyEntry + // cache evicts least-recently-used at a fixed cap and carries its own lock, so + // this type holds no mutex. Concurrent misses for one domain still both fetch, + // exactly as they did when the read and the write were separately locked. + cache *lrucache.Cache[string, offerKeyEntry] } type offerKeyEntry struct { @@ -84,7 +95,7 @@ func NewCachedOfferKeyResolver(cfg CachedOfferKeyResolverConfig) *CachedOfferKey ttl: ttl, now: now, revoked: cfg.Revoked, - cache: map[string]offerKeyEntry{}, + cache: lrucache.New[string, offerKeyEntry](maxCachedOfferKeys), } } @@ -96,12 +107,9 @@ func NewCachedOfferKeyResolver(cfg CachedOfferKeyResolverConfig) *CachedOfferKey // rejects the offer fail-closed. func (r *CachedOfferKeyResolver) Resolve(ctx context.Context, domain string) (ed25519.PublicKey, error) { now := r.now() - r.mu.Lock() - if e, ok := r.cache[domain]; ok && now.Before(e.exp) { - r.mu.Unlock() + if e, ok := r.cache.Get(domain); ok && now.Before(e.exp) { return e.key, nil } - r.mu.Unlock() dir, err := r.fetch(ctx, domain) if err != nil { @@ -115,9 +123,7 @@ func (r *CachedOfferKeyResolver) Resolve(ctx context.Context, domain string) (ed // the window is only re-checked on a cache miss, so a key could keep verifying up // to a full TTL beyond its not_after. exp := clampOfferKeyExpiry(now, r.ttl, notAfter) - r.mu.Lock() - r.cache[domain] = offerKeyEntry{key: key, exp: exp} - r.mu.Unlock() + r.cache.Put(domain, offerKeyEntry{key: key, exp: exp}) return key, nil } diff --git a/sdk/go/resolvers/cachedofferkeyresolver_internal_test.go b/sdk/go/resolvers/cachedofferkeyresolver_internal_test.go new file mode 100644 index 00000000..ea5d4786 --- /dev/null +++ b/sdk/go/resolvers/cachedofferkeyresolver_internal_test.go @@ -0,0 +1,80 @@ +package resolvers + +// The per-domain offer-key cache, tested from inside the package. +// +// Eviction itself is pinned once on the shared bounded map this cache is built +// from. What is left here is the resolver's own layer: that it is bounded at all, +// and that a refresh of a known domain does not consume a second slot. + +import ( + "context" + "crypto/ed25519" + "encoding/base64" + "fmt" + "testing" + "time" + + rampv1 "github.com/RAMP-Protocol/protocol/gen/go/ramp/v1" +) + +// oneKeyDirectory returns a fetcher that answers every domain with a directory +// carrying one window-active key, so a Resolve reaches the cache write. +func oneKeyDirectory(t *testing.T, now time.Time) OfferDirectoryFetcher { + t.Helper() + pub, _, err := ed25519.GenerateKey(nil) + if err != nil { + t.Fatalf("generate key: %v", err) + } + dir := &rampv1.WBAFile{Keys: []*rampv1.JsonWebKey{{ + Kty: "OKP", + Crv: "Ed25519", + Use: "sig", + Alg: "EdDSA", + X: base64.RawURLEncoding.EncodeToString(pub), + NotBefore: now.Add(-time.Hour).UTC().Format(time.RFC3339), + NotAfter: now.Add(time.Hour).UTC().Format(time.RFC3339), + }}} + return func(context.Context, string) (*rampv1.WBAFile, error) { return dir, nil } +} + +func TestOfferKeyCache_IsBoundedAtTheCap(t *testing.T) { + now := time.Unix(1_700_000_000, 0) + r := NewCachedOfferKeyResolver(CachedOfferKeyResolverConfig{ + Fetch: oneKeyDirectory(t, now), + TTL: time.Hour, + Now: func() time.Time { return now }, + }) + for i := range maxCachedOfferKeys + 10 { + if _, err := r.Resolve(context.Background(), fmt.Sprintf("ex%d.test", i)); err != nil { + t.Fatalf("resolve %d: %v", i, err) + } + } + // EXACTLY the cap, not merely at-or-under it: this is what pins that the + // resolver passed its own constant to the shared cache rather than some other + // bound. Ordering is pinned once, on the shared type. + if got := r.cache.Len(); got != maxCachedOfferKeys { + t.Errorf("cache size = %d, want exactly %d", got, maxCachedOfferKeys) + } +} + +// Re-resolving a known domain after its TTL lapses refreshes it in place rather +// than consuming a second slot — otherwise every refresh would evict an unrelated +// exchange. +func TestOfferKeyCache_RefreshingAKnownDomainKeepsOneSlot(t *testing.T) { + now := time.Unix(1_700_000_000, 0) + r := NewCachedOfferKeyResolver(CachedOfferKeyResolverConfig{ + Fetch: oneKeyDirectory(t, now), + TTL: time.Minute, + Now: func() time.Time { return now }, + }) + if _, err := r.Resolve(context.Background(), "ex.test"); err != nil { + t.Fatalf("first resolve: %v", err) + } + now = now.Add(2 * time.Minute) // expire the entry, forcing the write path again + if _, err := r.Resolve(context.Background(), "ex.test"); err != nil { + t.Fatalf("second resolve: %v", err) + } + if got := r.cache.Len(); got != 1 { + t.Errorf("cache size = %d, want 1", got) + } +} diff --git a/sdk/go/resolvers/contentfetch.go b/sdk/go/resolvers/contentfetch.go new file mode 100644 index 00000000..44e62c9e --- /dev/null +++ b/sdk/go/resolvers/contentfetch.go @@ -0,0 +1,378 @@ +package resolvers + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "mime" + "net/http" + "net/url" + "regexp" + "time" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/failure" +) + +// The content leg: fetching the bytes a signed delivery URL names, presenting the +// agent key that URL is bound to. +// +// It lives in this tier because it DIALS — a retrieval endpoint is chosen by a +// party on the network, not by configuration, which is the exact threat shape +// this package exists to contain. The transport-neutral tiers above stay free of +// any dialing surface. + +// DefaultContentTimeout bounds one content fetch. An agent is blocked on the call +// that triggered it, so a fetch that has not answered by now is more useful as a +// reported failure than as a hang. +const DefaultContentTimeout = 30 * time.Second + +// DefaultMaxContentBytes caps one fetched body at 8 MiB. This is a memory bound +// on the fetching process, not a judgement about how large licensed content may +// be: the body is buffered whole and held for the life of the call, and a batch +// fetches one per item. +const DefaultMaxContentBytes int64 = 8 << 20 + +// maxErrorBodyBytes caps how much of a refusal body is read before the edge's +// reason is parsed out of it. The payload is a small JSON object; anything past +// this is not a refusal that can be interpreted. +const maxErrorBodyBytes int64 = 4 << 10 + +// defaultContentMIMEType is what a body with no usable Content-Type is labelled. +// Guessing from the bytes would be worse: the caller is told what the publisher +// said, and "unknown" is a true answer where a sniffed guess might not be. +const defaultContentMIMEType = "application/octet-stream" + +// ProofSigner mints the proof of possession for one bound fetch. It is an +// injected seam so this tier never holds key material: the caller composes it +// over whatever custody it uses, and decides the proof window. +type ProofSigner interface { + // SignFetch returns the agent binding for a GET of targetURL. The URL is + // passed verbatim because the proof covers it as an exact string. + SignFetch(ctx context.Context, targetURL string) (helpers.AgentBinding, error) +} + +// ContentFetchOptions configures a ContentFetcher. Every field has a safe default. +type ContentFetchOptions struct { + // BaseTransport carries the caller's own transport settings — a tuned + // connection pool, client certificates via TLSClientConfig — UNDERNEATH the + // SSRF guard. It is never a replacement for the guard: a delivery URL names a + // host chosen by another party, so the address pin and the https-only scheme + // check are applied in every case. Nil means a fresh transport under the guard. + // + // A custom TLS dialer on this base is dropped rather than honoured: net/http + // would prefer it over the pinned dialer on https and the address check would + // never run. Configure TLS through TLSClientConfig, which is kept. + // + // The redirect policy is likewise a property of this profile rather than a + // detail a caller supplies. A caller that could inject a whole client would be + // asserting against its own policy instead of the one production runs. + // + // The only way to reach a private or plaintext endpoint is the deliberate, + // deployment-level SKIP_SSRF / ALLOW_INSECURE opt-out, which is one decision + // recorded in one place instead of a per-caller copy of it. + BaseTransport *http.Transport + // Timeout bounds one fetch, proof minting included. Defaults to + // DefaultContentTimeout. + Timeout time.Duration + // MaxBytes caps one fetched body. Defaults to DefaultMaxContentBytes. + MaxBytes int64 + // RequestID mints the value of the X-Request-ID correlation header stamped on + // each delivery GET. Nil sends no header. + // + // It matters on THIS leg in particular. The RPC legs correlate through an + // interceptor, which a plain GET never traverses, so without a hook here the + // delivery fetch is the one leg carrying no id — and a delivery edge that mints + // its own when the header is absent then logs a refusal under an id nothing + // else knows. That is the leg where delivery failures are diagnosed. + // + // A func rather than a string because one fetcher serves many requests, and the + // point of the header is that each carries its own id. It is `func() string` + // rather than the transport tier's named RequestIDFunc so this package needs no + // dependency on that tier for one alias; a named type is assignable here. + RequestID func() string +} + +// Content is one fetched resource. +type Content struct { + // URL is the signed delivery URL that was fetched, echoed back so a caller + // correlating a batch does not have to keep its own map. + URL string + // MIMEType is the media type the edge served, parameters stripped. + MIMEType string + // Body is the fetched bytes. + Body []byte +} + +// FetchFailure classifies why a content fetch failed, so a caller can branch on +// the class without reading the message. +type FetchFailure int + +const ( + // FetchUnknown is the zero value; it carries no classification. + FetchUnknown FetchFailure = iota + // FetchRefused is an edge that answered and said no. Reason carries the + // edge's own token when it sent one. + FetchRefused + // FetchUnreachable is an edge that did not answer: dial failure, timeout, or + // a refused redirect. + FetchUnreachable + // FetchTooLarge is a body past the configured cap. Deliberately distinct from + // FetchRefused: the edge did nothing wrong and the URL is still good, so the + // caller can retry with a larger budget. + FetchTooLarge + // FetchNotSignable is the proof failing to be produced. No request leaves on + // this path — a custody backend that hangs lands here too, as a deadline, + // because the timeout covers proof minting. + FetchNotSignable + // FetchMalformed is a delivery URL this client cannot sign faithfully. + FetchMalformed +) + +var fetchFailureNames = map[FetchFailure]string{ + FetchRefused: "refused", + FetchUnreachable: "unreachable", + FetchTooLarge: "too_large", + FetchNotSignable: "not_signable", + FetchMalformed: "malformed", +} + +// String renders the failure class for logging and for the reason a caller sees +// when the edge supplied none. +func (f FetchFailure) String() string { return failure.Name(fetchFailureNames, f) } + +// FetchError is this tier's canonical content-fetch error. +// +// Reason exists so the edge's own refusal token survives as a value rather than +// being flattened into a sentence here. Those tokens are the difference between +// "the publisher refused us" and "our own key wiring is broken", and the layer +// that decides how a refusal reads can only tell them apart if the token arrives +// intact. +type FetchError struct { + Failure FetchFailure + Op string + Status int // HTTP status when the edge answered; 0 otherwise + Reason string // the edge's refusal token when it sent one + Err error +} + +func (e *FetchError) Error() string { + return failure.Render("resolvers", e.Op, e.Failure.String(), e.Status, e.Reason, e.Err) +} + +// Unwrap keeps the cause matchable, so a caller can still reach a custody +// sentinel through errors.Is after the failure has been classified here. +func (e *FetchError) Unwrap() error { return e.Err } + +// ReasonOf returns the most specific machine-readable reason available: the +// edge's own token when it sent one, otherwise the failure class. +func (e *FetchError) ReasonOf() string { return failure.ReasonOr(e.Reason, e.Failure.String()) } + +// ContentFetcher fetches licensed content from a signed delivery URL. Build it +// with NewContentFetcher; it is safe for concurrent use. +type ContentFetcher struct { + http *http.Client + timeout time.Duration + maxBytes int64 + requestID func() string +} + +// NewContentFetcher returns a fetcher whose zero-value options are safe defaults: +// the SSRF-guarded transport, a 30-second bound, an 8 MiB body cap, and redirects +// refused. +func NewContentFetcher(opts ContentFetchOptions) *ContentFetcher { + // The guard is composed here and cannot be handed in already-built: a caller + // supplies what sits UNDER it, never what replaces it. + // + // The redirect policy is this profile's own, which is why the guarded CLIENT + // is not reused: it follows up to five hops, which is right for a public + // well-known document and wrong for anything carrying a credential. + transport := NewGuardedTransport(opts.BaseTransport) + timeout := opts.Timeout + if timeout <= 0 { + timeout = DefaultContentTimeout + } + maxBytes := opts.MaxBytes + if maxBytes <= 0 { + maxBytes = DefaultMaxContentBytes + } + return &ContentFetcher{ + http: &http.Client{Transport: transport, CheckRedirect: refuseContentRedirect}, + timeout: timeout, + maxBytes: maxBytes, + requestID: opts.RequestID, + } +} + +// refuseContentRedirect stops the client following any 3xx. +// +// Following one either replays a proof bound to the old URL — which the edge's +// own check rejects — or, if the proof were re-minted per hop, hands a fresh +// proof of possession of the agent's key to whatever host the first hop named. +var refuseContentRedirect = failure.RefuseRedirect( + "resolvers", "a bound fetch is never redirected", helpers.RedactURL) + +// Fetch retrieves the content at signedURL, presenting the proof of possession +// signer mints for it. +func (f *ContentFetcher) Fetch(ctx context.Context, signedURL string, signer ProofSigner) (Content, error) { + const op = "fetch content" + if signer == nil { + return Content{}, &FetchError{Failure: FetchNotSignable, Op: op, + Err: errors.New("no proof signer supplied")} + } + // The deadline is derived BEFORE the request is built, because building it + // mints a proof — which may call out to a custody backend bounded only by that + // backend's own client otherwise. A timeout covering the round trip alone + // would leave "bounds one content fetch" untrue against a degraded custody + // service, and a batch pays that cost once per item. + ctx, cancel := context.WithTimeout(ctx, f.timeout) + defer cancel() + + req, err := f.request(ctx, op, signedURL, signer) + if err != nil { + return Content{}, err + } + resp, err := f.http.Do(req) + if err != nil { + return Content{}, &FetchError{Failure: FetchUnreachable, Op: op, Err: redactTransportError(err)} + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { + return Content{}, &FetchError{ + Failure: FetchRefused, Op: op, Status: resp.StatusCode, Reason: edgeReason(resp.Body), + } + } + body, err := f.read(resp.Body) + if err != nil { + return Content{}, err + } + return Content{URL: signedURL, MIMEType: mimeTypeOf(resp.Header.Get("Content-Type")), Body: body}, nil +} + +// request builds the signed GET. It is separate from Fetch so the preconditions +// are in one place and no partially-built request can reach the wire. +// +// The round-trip check runs BEFORE the proof is minted, so a URL that cannot be +// sent faithfully never costs a signing operation. +func (f *ContentFetcher) request(ctx context.Context, op, signedURL string, signer ProofSigner) (*http.Request, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, signedURL, nil) + if err != nil { + // The cause is NOT wrapped: a parse failure prints the offending URL, and a + // delivery URL carries a live credential in its query. Nothing safe can be + // named either — redaction itself needs a parseable value — so the class is + // the whole message. + return nil, &FetchError{ + Failure: FetchMalformed, Op: op, + Err: errors.New("delivery url is not parseable (value withheld: it carries a live credential)"), + } + } + // The proof covers @target-uri as the VERBATIM string, while the request line + // carries whatever the URL value re-serializes to. The signed-URL contract + // treats scheme/host/path as opaque bytes, so an Exchange can legitimately mint + // a URL those two disagree on — a raw space in the path is the reachable case, + // since the request line escapes it. The signature then cannot verify and the + // edge reports only an undifferentiated 403, so refusing here names the cause + // instead. (A percent-escape does not trip this: the URL value preserves it.) + if req.URL.String() != signedURL { + // The given URL is deliberately NOT echoed: this error reaches a log, and + // the value carries a live credential in its query. The re-serialized form + // is what an operator compares against what the Exchange minted. + return nil, &FetchError{ + Failure: FetchMalformed, Op: op, + Err: fmt.Errorf("url is not round-trip stable: it re-serializes to %s (query redacted)", + helpers.RedactURL(req.URL.String())), + } + } + // Stamped BEFORE the binding, so the covered headers are written last and + // nothing here can be mistaken for part of the proof. The correlation id is not + // covered by the signature and is not meant to be: it identifies the request in + // two sets of logs, it authorises nothing. + if f.requestID != nil { + req.Header.Set(helpers.RequestIDHeader, f.requestID()) + } + binding, err := signer.SignFetch(ctx, signedURL) + if err != nil { + // Wrapped, not replaced: a caller must still be able to reach a custody + // sentinel underneath through errors.Is. + return nil, &FetchError{Failure: FetchNotSignable, Op: op, Err: err} + } + binding.Apply(req.Header) + return req, nil +} + +// redactTransportError strips the credential out of a transport failure. +// +// The HTTP client wraps every failure in a *url.Error carrying the full URL it +// was dialing — query included. For a delivery fetch that query IS the +// credential, and on a refused redirect it is the credential of a URL the FIRST +// HOP chose, so the wrapper leaks even when this package's own message is already +// redacted. Rebuilding it is the only way to keep the value out of whatever reads +// the error; the underlying cause is preserved so errors.Is still reaches it. +func redactTransportError(err error) error { + var urlErr *url.Error + if !errors.As(err, &urlErr) { + return err + } + return fmt.Errorf("%s %s: %w", urlErr.Op, helpers.RedactURL(urlErr.URL), urlErr.Err) +} + +// read consumes the body under the configured cap. +// +// It reads one byte past the cap so an oversized body is DETECTED rather than +// silently truncated. Truncated content that looks whole is worse than a refusal: +// the caller has paid for it and has no way to tell it is incomplete. +func (f *ContentFetcher) read(r io.Reader) ([]byte, error) { + body, err := io.ReadAll(io.LimitReader(r, f.maxBytes+1)) + if err != nil { + return nil, &FetchError{Failure: FetchUnreachable, Op: "read content", Err: err} + } + if int64(len(body)) > f.maxBytes { + return nil, &FetchError{ + Failure: FetchTooLarge, Op: "read content", + Err: fmt.Errorf("body exceeds the %d byte cap", f.maxBytes), + } + } + return body, nil +} + +// edgeReason pulls the edge's own refusal token out of a rejection body. The edge +// answers {"error": "...", "reason": "..."} on a binding failure; anything else +// yields "", and the caller falls back to the failure class. +func edgeReason(r io.Reader) string { + var payload struct { + Reason string `json:"reason"` + } + if err := json.NewDecoder(io.LimitReader(r, maxErrorBodyBytes)).Decode(&payload); err != nil { + return "" + } + if !edgeReasonToken.MatchString(payload.Reason) { + return "" + } + return payload.Reason +} + +// edgeReasonToken is the shape a refusal token may have. +// +// The body this is read from is written by the host just fetched from, and the +// value is promoted over this SDK's own classification. Unchecked, a publisher +// could answer any 4 KiB of text and have it render as though the SDK had said +// it. Anything that is not token-shaped falls back to the failure class, which +// the SDK does own. +var edgeReasonToken = regexp.MustCompile(`^[a-z][a-z0-9_]{0,63}$`) + +// mimeTypeOf reduces a Content-Type to its media type, dropping parameters such +// as charset. The charset belongs to whoever decodes the bytes; the content +// carries the media type alone. +func mimeTypeOf(header string) string { + if header == "" { + return defaultContentMIMEType + } + mediaType, _, err := mime.ParseMediaType(header) + if err != nil || mediaType == "" { + return defaultContentMIMEType + } + return mediaType +} diff --git a/sdk/go/resolvers/contentfetch_test.go b/sdk/go/resolvers/contentfetch_test.go new file mode 100644 index 00000000..7439d440 --- /dev/null +++ b/sdk/go/resolvers/contentfetch_test.go @@ -0,0 +1,495 @@ +package resolvers_test + +import ( + "context" + "crypto/ed25519" + "encoding/base64" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" +) + +// The content leg, driven through its public surface against a real HTTP server. +// The server verifies the proof the way the edge does — rebuilding the signature +// base from the raw request line and the Signature-Input as received — so these +// tests assert the wire contract rather than the SDK agreeing with itself. + +// popSigner is the application-supplied ProofSigner seam. The SDK never holds a +// key; this is the caller composing one. +type popSigner struct { + signer helpers.Signer + pub ed25519.PublicKey + created int64 + expires int64 + err error +} + +func (p popSigner) SignFetch(ctx context.Context, target string) (helpers.AgentBinding, error) { + if p.err != nil { + return helpers.AgentBinding{}, p.err + } + return helpers.SignAgentBinding(ctx, p.signer, p.pub, helpers.PoPOptions{ + URL: target, Created: p.created, Expires: p.expires, + }) +} + +func newPopSigner(t *testing.T) popSigner { + t.Helper() + pub, priv, err := ed25519.GenerateKey(nil) + if err != nil { + t.Fatalf("generate agent key: %v", err) + } + thumb, err := helpers.Thumbprint(pub) + if err != nil { + t.Fatalf("thumbprint: %v", err) + } + signer, err := helpers.NewEd25519Signer(thumb, priv) + if err != nil { + t.Fatalf("build signer: %v", err) + } + now := time.Now().Unix() + return popSigner{signer: signer, pub: pub, created: now, expires: now + 30} +} + +// verifyProofLikeTheEdge reproduces the offline three-way identity check a +// code-capable edge runs: the presented key's thumbprint must equal the keyid on +// the signature, and the signature must verify over a base rebuilt from the raw +// request line. +func verifyProofLikeTheEdge(t *testing.T, r *http.Request) { + t.Helper() + presented := r.Header.Get(helpers.AgentKeyHeader) + if presented == "" { + t.Error("request carries no agent key header") + return + } + pubBytes, err := base64.RawURLEncoding.DecodeString(presented) + if err != nil { + t.Errorf("agent key is not base64url-no-pad: %v", err) + return + } + sigInput := r.Header.Get("Signature-Input") + rawParams, ok := strings.CutPrefix(sigInput, "sig1=") + if !ok { + t.Errorf("unexpected Signature-Input label: %q", sigInput) + return + } + keyID := betweenQuotes(rawParams, "keyid=") + thumb, err := helpers.Thumbprint(pubBytes) + if err != nil { + t.Errorf("thumbprint presented key: %v", err) + return + } + if keyID != thumb { + t.Errorf("keyid %q is not the presented key's thumbprint %q", keyID, thumb) + } + sigValue := r.Header.Get("Signature") + encoded := strings.TrimSuffix(strings.TrimPrefix(sigValue, "sig1=:"), ":") + sig, err := base64.StdEncoding.DecodeString(encoded) + if err != nil { + t.Errorf("signature is not standard base64: %v", err) + return + } + // The edge rebuilds @target-uri from the raw request line it received. + target := "http://" + r.Host + r.URL.RequestURI() + base := `"@method": ` + r.Method + "\n" + + `"@target-uri": ` + target + "\n" + + `"@signature-params": ` + rawParams + if !ed25519.Verify(pubBytes, []byte(base), sig) { + t.Errorf("proof does not verify over the base the edge reconstructs:\n%s", base) + } +} + +func betweenQuotes(s, key string) string { + rest, ok := strings.CutPrefix(s[strings.Index(s, key):], key+`"`) + if !ok { + return "" + } + end := strings.Index(rest, `"`) + if end < 0 { + return "" + } + return rest[:end] +} + +// plainFetcher builds a fetcher that may reach a loopback httptest server. The +// guard is not removable by option — a caller supplies what sits UNDER it — so a +// test opts out the way a deployment does, through the two documented env flags. +func plainFetcher(t *testing.T, opts resolvers.ContentFetchOptions) *resolvers.ContentFetcher { + t.Helper() + t.Setenv("SKIP_SSRF", "1") + t.Setenv("ALLOW_INSECURE", "1") + return resolvers.NewContentFetcher(opts) +} + +func TestContentFetcher_PresentsAVerifiableProof(t *testing.T) { + signer := newPopSigner(t) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + verifyProofLikeTheEdge(t, r) + w.Header().Set("Content-Type", "text/html; charset=utf-8") + _, _ = w.Write([]byte("licensed")) + })) + defer srv.Close() + + got, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/doc?agent_id=tp", signer) + if err != nil { + t.Fatalf("fetch: %v", err) + } + if string(got.Body) != "licensed" { + t.Errorf("body = %q", got.Body) + } + if got.MIMEType != "text/html" { + t.Errorf("MIMEType = %q, want the media type with parameters stripped", got.MIMEType) + } + if got.URL != srv.URL+"/doc?agent_id=tp" { + t.Errorf("URL = %q, want the fetched URL echoed back", got.URL) + } +} + +func TestContentFetcher_MIMEFallback(t *testing.T) { + tests := map[string]string{ + "": "application/octet-stream", + "not a media type": "application/octet-stream", + "application/pdf": "application/pdf", + } + for header, want := range tests { + t.Run("content-type "+header, func(t *testing.T) { + signer := newPopSigner(t) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if header != "" { + w.Header().Set("Content-Type", header) + } else { + // net/http sniffs a type on write unless the header is + // explicitly suppressed, and an absent Content-Type is the case + // under test. + w.Header()["Content-Type"] = nil + } + _, _ = w.Write([]byte("bytes")) + })) + defer srv.Close() + + got, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/doc", signer) + if err != nil { + t.Fatalf("fetch: %v", err) + } + if got.MIMEType != want { + t.Errorf("MIMEType = %q, want %q", got.MIMEType, want) + } + }) + } +} + +// A redirect is refused, and the crucial half of the assertion is that the target +// is never contacted: following one would hand a fresh proof of possession of the +// agent's key to whatever host the first hop named. +func TestContentFetcher_RefusesRedirectsAndNeverContactsTheTarget(t *testing.T) { + var targetHits atomic.Int64 + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + targetHits.Add(1) + _, _ = w.Write([]byte("attacker content")) + })) + defer target.Close() + + origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/stolen", http.StatusFound) + })) + defer origin.Close() + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), origin.URL+"/doc", newPopSigner(t)) + if err == nil { + t.Fatal("expected the redirect to be refused") + } + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchUnreachable { + t.Errorf("error = %v, want a FetchUnreachable FetchError", err) + } + if n := targetHits.Load(); n != 0 { + t.Errorf("redirect target was contacted %d times; it must never be reached", n) + } +} + +// A URL that does not survive re-serialization is refused BEFORE anything is +// signed or sent, because the proof covers the verbatim string and a mismatch +// would surface at the edge as an undifferentiated 403. +func TestContentFetcher_RefusesANonRoundTripStableURLWithoutSending(t *testing.T) { + var hits atomic.Int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits.Add(1) + _, _ = w.Write([]byte("never served")) + })) + defer srv.Close() + + // A RAW space in the path. The signed-URL contract covers scheme/host/path as + // opaque bytes, so an Exchange can mint exactly this — but the request line + // escapes it to %20, so the bytes sent would not be the bytes the proof covers. + // (A percent-escape does NOT trip this: the URL value preserves the raw path.) + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/a b/doc", newPopSigner(t)) + if err == nil { + t.Fatal("expected the unstable URL to be refused") + } + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchMalformed { + t.Errorf("error = %v, want a FetchMalformed FetchError", err) + } + if n := hits.Load(); n != 0 { + t.Errorf("server was contacted %d times; the refusal must be local", n) + } +} + +// The proof is minted after the URL check, so a signer failure means nothing left +// the process. +func TestContentFetcher_UnsignableProofSendsNothing(t *testing.T) { + var hits atomic.Int64 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits.Add(1) + })) + defer srv.Close() + + custodyDown := errors.New("custody backend unavailable") + signer := newPopSigner(t) + signer.err = custodyDown + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/doc", signer) + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchNotSignable { + t.Fatalf("error = %v, want a FetchNotSignable FetchError", err) + } + if !errors.Is(err, custodyDown) { + t.Error("the custody cause must stay reachable through errors.Is") + } + if n := hits.Load(); n != 0 { + t.Errorf("server was contacted %d times; an unsignable fetch must send nothing", n) + } +} + +// An oversized body is DETECTED, not truncated: content that looks whole but is +// not is worse than a refusal, because it has been paid for. +func TestContentFetcher_OversizedBodyIsDetectedNotTruncated(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write(make([]byte, 64)) + })) + defer srv.Close() + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{MaxBytes: 16}). + Fetch(context.Background(), srv.URL+"/doc", newPopSigner(t)) + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchTooLarge { + t.Fatalf("error = %v, want a FetchTooLarge FetchError", err) + } +} + +// A body exactly at the cap is served, so the boundary is inclusive. +func TestContentFetcher_BodyAtTheCapIsServed(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write(make([]byte, 16)) + })) + defer srv.Close() + + got, err := plainFetcher(t, resolvers.ContentFetchOptions{MaxBytes: 16}). + Fetch(context.Background(), srv.URL+"/doc", newPopSigner(t)) + if err != nil { + t.Fatalf("a body exactly at the cap must be served: %v", err) + } + if len(got.Body) != 16 { + t.Errorf("body length = %d, want 16", len(got.Body)) + } +} + +func TestContentFetcher_EdgeRefusal(t *testing.T) { + tests := map[string]struct { + body string + wantReason string + }{ + "token surfaces": {`{"error":"forbidden","reason":"pop_expired"}`, "pop_expired"}, + "non token shaped is dropped": {`{"reason":"You are not allowed, friend."}`, ""}, + "absent reason is dropped": {`{"error":"forbidden"}`, ""}, + "unparseable body is dropped": {`not json at all`, ""}, + "oversized body is dropped": {`{"reason":"` + strings.Repeat("a", 8192) + `"}`, ""}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(tc.body)) + })) + defer srv.Close() + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/doc", newPopSigner(t)) + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchRefused { + t.Fatalf("error = %v, want a FetchRefused FetchError", err) + } + if ferr.Status != http.StatusForbidden { + t.Errorf("Status = %d, want 403", ferr.Status) + } + if ferr.Reason != tc.wantReason { + t.Errorf("Reason = %q, want %q", ferr.Reason, tc.wantReason) + } + // The class is owned by this SDK and is always available. + wantFallback := tc.wantReason + if wantFallback == "" { + wantFallback = "refused" + } + if got := ferr.ReasonOf(); got != wantFallback { + t.Errorf("ReasonOf() = %q, want %q", got, wantFallback) + } + }) + } +} + +// No credential may reach a log or an error string: the signed URL carries sig, +// kid, exp and agent_id in its query. +func TestContentFetcher_ErrorsCarryNoCredential(t *testing.T) { + target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})) + defer target.Close() + origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/stolen?sig=REDIRECTSECRET", http.StatusFound) + })) + defer origin.Close() + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), origin.URL+"/doc?sig=ORIGINSECRET&agent_id=tp", newPopSigner(t)) + if err == nil { + t.Fatal("expected a refusal") + } + for _, leak := range []string{"ORIGINSECRET", "REDIRECTSECRET", "sig=", "agent_id="} { + if strings.Contains(err.Error(), leak) { + t.Errorf("error string leaks %q: %s", leak, err) + } + } +} + +// The delivery GET carries the correlation header when a mint is supplied. +// +// This leg is the one that needs it most and is the one that would silently lose +// it: the RPC legs correlate through an interceptor, which a plain GET never +// traverses. A delivery edge that mints its own id when the header is absent then +// records a refusal under a value nothing on this side can join it to — and the +// delivery leg is where those refusals are diagnosed. +func TestContentFetcher_StampsTheCorrelationHeader(t *testing.T) { + signer := newPopSigner(t) + var got string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Get(helpers.RequestIDHeader) + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + f := plainFetcher(t, resolvers.ContentFetchOptions{ + RequestID: func() string { return "req-abc123" }, + }) + if _, err := f.Fetch(context.Background(), srv.URL+"/doc?agent_id=tp", signer); err != nil { + t.Fatalf("fetch: %v", err) + } + if got != "req-abc123" { + t.Errorf("%s = %q, want the injected mint's value", helpers.RequestIDHeader, got) + } +} + +// With no mint the header is ABSENT rather than invented. This tier holds no clock +// and no random source of its own — every such thing is injected — so a fetcher +// built without a mint must not conjure an id the caller cannot correlate against. +func TestContentFetcher_SendsNoCorrelationHeaderWithoutAMint(t *testing.T) { + signer := newPopSigner(t) + present := true + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, present = r.Header[http.CanonicalHeaderKey(helpers.RequestIDHeader)] + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + f := plainFetcher(t, resolvers.ContentFetchOptions{}) + if _, err := f.Fetch(context.Background(), srv.URL+"/doc?agent_id=tp", signer); err != nil { + t.Fatalf("fetch: %v", err) + } + if present { + t.Errorf("%s was sent with no mint configured", helpers.RequestIDHeader) + } +} + +// hangingSigner stands in for a custody backend that has stopped answering: it +// blocks until the context it was given is done, and reports why. +type hangingSigner struct{} + +func (hangingSigner) SignFetch(ctx context.Context, _ string) (helpers.AgentBinding, error) { + <-ctx.Done() + return helpers.AgentBinding{}, ctx.Err() +} + +// The fetch deadline covers PROOF MINTING, not just the round trip. +// +// The proof is minted while the request is being built, and a ProofSigner is +// application code that may reach a custody backend bounded only by that backend's +// own client. So the deadline is derived before the request is built rather than +// around the transport: derived after, "bounds one content fetch" would be untrue +// against a degraded custody service, and a batch would pay that cost once per +// item with nothing to stop it. +// +// The sibling option test bounds a slow EDGE, which the transport's own deadline +// already handles — it stays green if the ordering here is reversed. This one +// blocks in the signer, which is the only place that tells the two apart. +func TestContentFetcher_TheDeadlineCoversProofMinting(t *testing.T) { + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits.Add(1) + _, _ = w.Write([]byte("must never be reached")) + })) + defer srv.Close() + + f := plainFetcher(t, resolvers.ContentFetchOptions{Timeout: 50 * time.Millisecond}) + + // Run off the test goroutine and give up on our own terms. The failure this + // guards against is not a slow fetch, it is one that NEVER RETURNS: the signer + // waits on a context that, without the deadline, nothing ever cancels. Called + // inline, a regression would hang until the whole suite panicked ten minutes + // later; here it is a named failure in five seconds. + type result struct { + err error + } + done := make(chan result, 1) + go func() { + _, err := f.Fetch(context.Background(), srv.URL+"/doc?agent_id=tp", hangingSigner{}) + done <- result{err: err} + }() + + var err error + select { + case r := <-done: + err = r.err + case <-time.After(5 * time.Second): + t.Fatal("Fetch never returned; the configured deadline did not cover proof minting") + } + + if err == nil { + t.Fatal("a signer that never answers must not produce a successful fetch") + } + // The classification a caller branches on: no request left, so this is not a + // refusal by the edge and not a transport failure. + var fe *resolvers.FetchError + if !errors.As(err, &fe) { + t.Fatalf("error = %v, want a *FetchError", err) + } + if fe.Failure != resolvers.FetchNotSignable { + t.Errorf("failure = %v, want FetchNotSignable", fe.Failure) + } + // The load-bearing assertion. Without it this test would pass on any signer + // error at all, rather than on the CONFIGURED deadline being what stopped it. + if !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("error = %v, want the configured deadline to be the cause", err) + } + if n := hits.Load(); n != 0 { + t.Errorf("the edge was contacted %d time(s); no request leaves on this path", n) + } +} diff --git a/sdk/go/resolvers/endpointresolver.go b/sdk/go/resolvers/endpointresolver.go index 1c8f46a1..50fa95dc 100644 --- a/sdk/go/resolvers/endpointresolver.go +++ b/sdk/go/resolvers/endpointresolver.go @@ -7,10 +7,14 @@ import ( "fmt" "io" "net/http" - "sync" "time" jose "github.com/go-jose/go-jose/v4" + "golang.org/x/sync/singleflight" + + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/endpointrule" + "github.com/RAMP-Protocol/protocol/sdk/go/internal/lrucache" ) // maxWellKnownDocBytes bounds the well-known / JWKS response body read. A hostile @@ -27,6 +31,16 @@ const maxWellKnownDocBytes = 1 << 20 // 1 MiB // "Exchange reachable but not self-advertising an endpoint". var ErrNoEndpoint = errors.New("resolvers: well-known manifest has no endpoint") +// ErrEndpointRefused signals that a manifest WAS read and advertises an endpoint +// this resolver will not hand back: one on a host unrelated to the domain that +// served the manifest, or one carrying userinfo. +// +// Distinct from ErrNoEndpoint and from a transport failure because it is a +// VERDICT — the Exchange answered, and the answer is not usable. A caller that +// classifies retryability reads this as final rather than as something to try +// again in a moment. +var ErrEndpointRefused = errors.New("resolvers: well-known manifest advertises an unusable endpoint") + // wellKnownDoc is the JSON projection of the subset of WellKnownManifest the SDK // resolvers read: the RFC 7517 key set (field 5) and the self-advertised // ExchangeService endpoint (field 12). One fetch decodes the whole document so a @@ -67,6 +81,23 @@ func fetchWellKnownDoc(ctx context.Context, client *http.Client, url string) (*w return &doc, nil } +// maxCachedEndpoints bounds the per-host endpoint cache. The key is an +// Offer.exchange host, so which entries appear is driven by incoming offers — an +// open-ended, caller-influenced key space, and an unbounded map over one is +// somewhere an authenticated caller can make the process grow without limit. A +// real deployment reports to a handful of Exchanges. Mirrors the bound the +// per-origin client pool above this resolver already carries. +const maxCachedEndpoints = 256 + +// maxManifestFetch bounds the SHARED manifest fetch — the one every coalesced +// waiter is served from, which therefore cannot take any single caller's deadline. +// +// It exists because the alternative is no bound at all: WellKnownOptions.HTTP +// accepts any *http.Client, the constructor doc invites one, and http.DefaultClient +// has no timeout. A manifest is a small document from a host an offer named; a +// fetch still running after this is not going to succeed. +const maxManifestFetch = 30 * time.Second + // WellKnownEndpointResolver resolves an Exchange domain to its self-advertised // ExchangeService endpoint by fetching https://{host}/.well-known/ramp.json and // reading WellKnownManifest.endpoint. Unlike WellKnownKeyResolver (one fixed @@ -75,15 +106,19 @@ func fetchWellKnownDoc(ctx context.Context, client *http.Client, url string) (*w // coalescing are all per host. The pre-seeded registry is a TRUST overlay (the // Allow hook), never the source of the endpoint — that is the Offer.exchange // routing invariant: the endpoint always comes from the exchange's own manifest. +// +// Because that host space is caller-influenced, neither per-host structure may +// grow without limit. The cache evicts least-recently-used at a fixed cap; +// coalescing is a singleflight.Group, which drops a host's entry as soon as its +// fetch completes and so holds nothing between calls. type WellKnownEndpointResolver struct { http *http.Client ttl time.Duration now func() time.Time scheme string allow func(host string) bool - mu sync.Mutex - cache map[string]endpointEntry - flight map[string]*sync.Mutex + sf singleflight.Group + cache *lrucache.Cache[string, endpointEntry] } type endpointEntry struct { @@ -126,8 +161,7 @@ func NewWellKnownEndpointResolver(opts WellKnownOptions) *WellKnownEndpointResol now: now, scheme: scheme, allow: opts.Allow, - cache: map[string]endpointEntry{}, - flight: map[string]*sync.Mutex{}, + cache: lrucache.New[string, endpointEntry](maxCachedEndpoints), } } @@ -135,54 +169,138 @@ func NewWellKnownEndpointResolver(opts WellKnownOptions) *WellKnownEndpointResol // well-known manifest. A fresh cache entry short-circuits; a miss or TTL expiry // triggers a single coalesced per-host fetch. A host the Allow overlay rejects // never reaches the network. +// +// host must be a BARE hostname — no scheme, path, query or userinfo, though a +// port is fine. That is checked here rather than assumed, for the same reason +// vetAdvertisedEndpoint runs here: it is a property of building this URL, not of +// any one caller's plans for it. func (r *WellKnownEndpointResolver) ResolveEndpoint(ctx context.Context, host string) (string, error) { + // Checked BEFORE the Allow overlay and before the cache. The fetch URL below is + // built by concatenation, so a value carrying a path or a query would choose + // WHAT gets fetched rather than merely where from — and the raw string is the + // cache key, so admitting one would also put it in a shared map. + bare, err := helpers.IsBareHost(host) + if err != nil { + return "", fmt.Errorf("resolvers: resolve endpoint: %w", err) + } + if !bare { + return "", fmt.Errorf("resolvers: %w: %q is not a bare host", helpers.ErrInvalidHost, host) + } if r.allow != nil && !r.allow(host) { return "", fmt.Errorf("%w: host %q not allowed", ErrNoEndpoint, host) } if ep, ok := r.cached(host); ok { return ep, nil } - fl := r.hostFlight(host) - fl.Lock() - defer fl.Unlock() - if ep, ok := r.cached(host); ok { - return ep, nil // another goroutine fetched while we waited + // A concurrent burst for one host issues ONE fetch and shares its result. The + // group holds a key only while its call is in flight, so coalescing state + // cannot accumulate over the caller-influenced host space — including for the + // hosts whose fetches fail, which is where a hand-rolled per-host mutex map + // grows fastest. + // + // Two contexts, because one caller's deadline must not become everyone's and + // nobody's deadline must not become the fetch's: + // + // - The SHARED fetch does not inherit the winning caller's cancellation. + // Every waiter receives whatever the leader returns, so a leader that walks + // away would otherwise fail a burst of callers whose own contexts are still + // live, and each would read that as the Exchange being unreachable. It + // carries maxManifestFetch instead, so it stays bounded whatever client was + // injected — WellKnownOptions.HTTP admits one with no timeout at all. + // - Each WAITER selects on its OWN context. singleflight is not + // context-aware, so without this the call would honour nobody's deadline: + // a caller with 200ms would sit until the shared fetch finished. The fetch + // continues for the others; only this caller gives up. + shared := r.sf.DoChan(host, func() (v any, err error) { + // Derived INSIDE the closure, which singleflight runs for the leader alone. + // Built before DoChan instead, every coalesced follower would allocate a + // timer whose cancel func only the leader's closure ever calls — one live + // timer per waiter, held until its full expiry. go vet's lostcancel cannot + // see that, because cancelFetch is called on the path it can trace. + fetchCtx, cancelFetch := context.WithTimeout( + context.WithoutCancel(ctx), maxManifestFetch) + defer cancelFetch() + // A panic is turned into this call's error, HERE, because nowhere above can + // do it: when a coalesced call has waiting channels singleflight re-raises + // the panic on a fresh goroutine — `go panic(e)` followed by `select{}` — so + // no caller's recover can reach it and the process dies. The two seams that + // can panic are application-supplied (WellKnownOptions.HTTP and .Now), which + // makes "one lookup fails" the right blast radius, not "the process exits". + defer func() { + if p := recover(); p != nil { + v, err = "", fmt.Errorf("resolvers: resolve endpoint for %q: panic: %v", host, p) + } + }() + if ep, ok := r.cached(host); ok { + return ep, nil // another goroutine fetched while we waited + } + url := r.scheme + "://" + host + "/.well-known/ramp.json" + doc, ferr := fetchWellKnownDoc(fetchCtx, r.http, url) + if ferr != nil { + return "", ferr + } + if doc.Endpoint == "" { + return "", fmt.Errorf("%w: host=%q", ErrNoEndpoint, host) + } + if verr := vetAdvertisedEndpoint(host, doc.Endpoint); verr != nil { + return "", verr + } + r.store(host, doc.Endpoint) + return doc.Endpoint, nil + }) + var v any + select { + case <-ctx.Done(): + return "", fmt.Errorf("resolvers: resolve endpoint for %q: %w", host, ctx.Err()) + case res := <-shared: + v, err = res.Val, res.Err } - url := r.scheme + "://" + host + "/.well-known/ramp.json" - doc, err := fetchWellKnownDoc(ctx, r.http, url) if err != nil { return "", err } - if doc.Endpoint == "" { - return "", fmt.Errorf("%w: host=%q", ErrNoEndpoint, host) + return v.(string), nil +} + +// vetAdvertisedEndpoint decides whether an endpoint a manifest advertises may be +// handed back at all. It runs HERE, in the resolver, rather than in each caller. +// +// The manifest that named this endpoint is served by the very host the call is +// bound for, so the endpoint is only as trustworthy as that host. An Exchange may +// advertise itself or a subdomain of itself, and nothing else — an endpoint on an +// unrelated host is one this resolver refuses to return, whatever the caller +// intends to do with it. A dial-time address guard has no objection to an +// unrelated PUBLIC host, so nothing below this catches it. +// +// It sits in the resolver because every consumer needs it and none of them can be +// relied on to remember: the check is a property of reading an endpoint out of a +// manifest, not of any one caller's plans for it. A caller may of course check +// again. +// +// Userinfo is refused for a different reason with the same shape. The host +// comparison reads the authority's host and ignores any user:password before it, +// so an endpoint carrying credentials would pass the host check and then have +// net/http stamp an Authorization header the SDK never chose, on a leg that +// already carries the agent's own signature. +func vetAdvertisedEndpoint(host, endpoint string) error { + if err := endpointrule.Vet(host, endpoint); err != nil { + return fmt.Errorf("%w: %w", ErrEndpointRefused, err) } - r.store(host, doc.Endpoint) - return doc.Endpoint, nil + return nil } +// cached returns host's endpoint when the entry is present and fresh. Freshness +// is this resolver's own concern and sits on top of the shared bound: a stale +// entry is left in place rather than deleted, since it still holds a slot the +// eviction policy can reclaim and the next successful fetch overwrites it. func (r *WellKnownEndpointResolver) cached(host string) (string, bool) { - r.mu.Lock() - defer r.mu.Unlock() - entry, ok := r.cache[host] + entry, ok := r.cache.Get(host) if !ok || r.now().After(entry.exp) { return "", false } return entry.endpoint, true } +// store records host's endpoint with a fresh TTL. func (r *WellKnownEndpointResolver) store(host, endpoint string) { - r.mu.Lock() - defer r.mu.Unlock() - r.cache[host] = endpointEntry{endpoint: endpoint, exp: r.now().Add(r.ttl)} -} - -func (r *WellKnownEndpointResolver) hostFlight(host string) *sync.Mutex { - r.mu.Lock() - defer r.mu.Unlock() - fl, ok := r.flight[host] - if !ok { - fl = &sync.Mutex{} - r.flight[host] = fl - } - return fl + r.cache.Put(host, endpointEntry{endpoint: endpoint, exp: r.now().Add(r.ttl)}) } diff --git a/sdk/go/resolvers/endpointresolver_internal_test.go b/sdk/go/resolvers/endpointresolver_internal_test.go new file mode 100644 index 00000000..dba941ee --- /dev/null +++ b/sdk/go/resolvers/endpointresolver_internal_test.go @@ -0,0 +1,60 @@ +package resolvers + +// The per-host endpoint cache, tested from inside the package. +// +// Eviction itself is pinned once on the shared bounded map this cache is built +// from. What is left here is the resolver's own layer: that it is bounded at all, +// and that TTL freshness sits correctly on top of a structure that knows nothing +// about time. + +import ( + "testing" + "time" +) + +func TestEndpointCache_IsBoundedAtTheCap(t *testing.T) { + r := NewWellKnownEndpointResolver(WellKnownOptions{TTL: time.Hour}) + for i := range maxCachedEndpoints + 10 { + r.store(string(rune('a'+i%26))+string(rune('0'+i/26))+".test", "https://ep.test") + } + // EXACTLY the cap, not merely at-or-under it: this is what pins that the + // resolver passed its own constant to the shared cache rather than some other + // bound. Ordering is pinned once, on the shared type. + if got := r.cache.Len(); got != maxCachedEndpoints { + t.Errorf("cache size = %d, want exactly %d", got, maxCachedEndpoints) + } +} + +// Re-storing a known host refreshes it in place rather than consuming a second +// slot — otherwise every TTL refresh would evict an unrelated host. +func TestEndpointCache_RefreshingAKnownHostKeepsOneSlot(t *testing.T) { + r := NewWellKnownEndpointResolver(WellKnownOptions{TTL: time.Hour}) + r.store("ex.test", "https://one.test") + r.store("ex.test", "https://two.test") + if got := r.cache.Len(); got != 1 { + t.Errorf("cache size = %d, want 1", got) + } + ep, ok := r.cached("ex.test") + if !ok || ep != "https://two.test" { + t.Errorf("cached = %q/%v, want the refreshed endpoint", ep, ok) + } +} + +// A stale entry is not served, and the slot it holds is reclaimable — the cache +// must not pin an expired host forever just because nothing asked for it again. +func TestEndpointCache_StaleEntryIsNotServedAndItsSlotIsReusable(t *testing.T) { + now := time.Unix(1_700_000_000, 0) + r := NewWellKnownEndpointResolver(WellKnownOptions{ + TTL: time.Minute, + Now: func() time.Time { return now }, + }) + r.store("ex.test", "https://one.test") + now = now.Add(2 * time.Minute) + if _, ok := r.cached("ex.test"); ok { + t.Error("an expired entry was served") + } + r.store("ex.test", "https://two.test") + if got := r.cache.Len(); got != 1 { + t.Errorf("cache size = %d, want the expired slot reused", got) + } +} diff --git a/sdk/go/resolvers/endpointresolver_test.go b/sdk/go/resolvers/endpointresolver_test.go index 2a14a32f..dba59acc 100644 --- a/sdk/go/resolvers/endpointresolver_test.go +++ b/sdk/go/resolvers/endpointresolver_test.go @@ -3,12 +3,16 @@ package resolvers_test import ( "context" "encoding/json" + "errors" "net/http" "net/http/httptest" "net/url" + "strings" + "sync/atomic" "testing" "time" + "github.com/RAMP-Protocol/protocol/sdk/go/helpers" "github.com/RAMP-Protocol/protocol/sdk/go/resolvers" ) @@ -47,12 +51,15 @@ func hostOf(t *testing.T, srv *httptest.Server) string { // this — it is the core host-keyed contract the broker's per-request // Offer.exchange resolution depends on. func TestWellKnownEndpointResolver_perHostIsolation(t *testing.T) { - epA := "https://exchange-a.example/ramp.v1.ExchangeService" - epB := "https://exchange-b.example/ramp.v1.ExchangeService" + // Late-bound: an Exchange advertises ITSELF, so the endpoint is not known + // until its server has an address. The handler reads it per request. + var epA, epB string srvA := httptest.NewServer(manifestHandler(&epA, nil)) defer srvA.Close() srvB := httptest.NewServer(manifestHandler(&epB, nil)) defer srvB.Close() + epA = srvA.URL + "/ramp.v1.ExchangeService" + epB = srvB.URL + "/ramp.v1.ExchangeService" r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ TTL: time.Hour, @@ -78,10 +85,11 @@ func TestWellKnownEndpointResolver_perHostIsolation(t *testing.T) { // TestWellKnownEndpointResolver_cacheHit proves the second resolve for the SAME // host short-circuits and does not refetch (request-counting handler). func TestWellKnownEndpointResolver_cacheHit(t *testing.T) { - ep := "https://exchange.example/ramp.v1.ExchangeService" + var ep string hits := 0 srv := httptest.NewServer(manifestHandler(&ep, &hits)) defer srv.Close() + ep = srv.URL + "/ramp.v1.ExchangeService" r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ TTL: time.Hour, @@ -103,10 +111,11 @@ func TestWellKnownEndpointResolver_cacheHit(t *testing.T) { // TestWellKnownEndpointResolver_ttlRefresh injects the clock (Now option) and // proves TTL expiry triggers a refetch, mirroring keyresolver_test.go. func TestWellKnownEndpointResolver_ttlRefresh(t *testing.T) { - ep := "https://exchange.example/ramp.v1.ExchangeService" + var ep string hits := 0 srv := httptest.NewServer(manifestHandler(&ep, &hits)) defer srv.Close() + ep = srv.URL + "/ramp.v1.ExchangeService" now := time.Unix(1700000000, 0) r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ @@ -180,3 +189,217 @@ func TestWellKnownEndpointResolver_missingEndpointField(t *testing.T) { t.Errorf("fetch hits = %d, want 1 (manifest fetched once)", hits) } } + +// The endpoint an Exchange advertises must be on the host that served the +// manifest, or a subdomain of it. The manifest is only as trustworthy as the host +// serving it, so an endpoint pointing anywhere else is one this resolver will not +// hand back — whatever the caller intended to do with it. +// +// Checked HERE rather than in each caller: every consumer needs the rule and none +// can be relied on to remember it, and a dial-time address guard has no objection +// to an unrelated PUBLIC host. +func TestWellKnownEndpointResolver_refusesAnEndpointOnAnotherHost(t *testing.T) { + cases := map[string]string{ + "unrelated host": "https://evil.example/ramp.v1.ExchangeService", + "label-boundary trick": "https://evil-127.0.0.1.example/ramp.v1.ExchangeService", + "userinfo": "https://user:pass@127.0.0.1/ramp.v1.ExchangeService", + } + for name, ep := range cases { + t.Run(name, func(t *testing.T) { + endpoint := ep + srv := httptest.NewServer(manifestHandler(&endpoint, nil)) + defer srv.Close() + + r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ + TTL: time.Hour, Scheme: "http", HTTP: http.DefaultClient, + }) + got, err := r.ResolveEndpoint(context.Background(), hostOf(t, srv)) + if err == nil { + t.Fatalf("resolve returned %q; the endpoint must be refused", got) + } + // A VERDICT, not a transport failure: the Exchange answered and the + // answer is unusable, so a caller classifying retryability must read + // this as final. + if !errors.Is(err, resolvers.ErrEndpointRefused) { + t.Errorf("error = %v, want it to carry ErrEndpointRefused", err) + } + }) + } +} + +// The PORT is part of the anchor. An endpoint on another port of the serving host +// is a different service — one the party that published the manifest need not +// control — so it is refused like any other mismatch. +func TestWellKnownEndpointResolver_refusesAnEndpointOnAnotherPort(t *testing.T) { + // Port 1 is not the manifest server's, and nothing is listening there, so a + // refusal arriving from anywhere but the rule would show up as a dial error. + endpoint := "http://127.0.0.1:1/ramp.v1.ExchangeService" + srv := httptest.NewServer(manifestHandler(&endpoint, nil)) + defer srv.Close() + + r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ + TTL: time.Hour, Scheme: "http", HTTP: http.DefaultClient, + }) + got, err := r.ResolveEndpoint(context.Background(), hostOf(t, srv)) + if err == nil { + t.Fatalf("resolve returned %q; an endpoint on another port must be refused", got) + } + if !errors.Is(err, resolvers.ErrEndpointRefused) { + t.Errorf("error = %v, want it to carry ErrEndpointRefused", err) + } +} + +// A default port written out and the same port left implicit are the SAME port, +// so an operator who spells :443 in full is not refused for spelling. Driven +// through the predicate the resolver uses, since httptest always binds a +// non-default port and a loopback server cannot express the case. +func TestWellKnownEndpointResolver_acceptsAWrittenOutDefaultPort(t *testing.T) { + for _, tc := range [][2]string{ + {"exchange.example", "https://exchange.example:443/v1"}, + {"exchange.example:443", "https://exchange.example/v1"}, + } { + anchored, err := helpers.HostAnchored(tc[0], tc[1]) + if err != nil || !anchored { + t.Errorf("HostAnchored(%q, %q) = %v, %v; want true", tc[0], tc[1], anchored, err) + } + } +} + +// A subdomain of the serving host IS allowed: an Exchange may delegate to its own +// subdomain, and refusing that would be a rule about names rather than about +// trust. Driven through the predicate the resolver uses, since a loopback server +// cannot serve two hostnames. +func TestWellKnownEndpointResolver_allowsASubdomainOfTheServingHost(t *testing.T) { + anchored, err := helpers.HostAnchored("exchange.example", "https://api.exchange.example/v1") + if err != nil || !anchored { + t.Fatalf("subdomain anchored = %v, %v; want true", anchored, err) + } +} + +// A caller's own deadline bounds its own call, even while a shared fetch for the +// same host is still running. +// +// singleflight is not context-aware, so a coalesced call honours nobody's deadline +// unless each waiter selects on its own. Without that a 200ms caller sits until +// the slow origin answers — which, with an injected client carrying no timeout, is +// however long the origin feels like taking. +func TestWellKnownEndpointResolver_honoursTheCallersOwnDeadline(t *testing.T) { + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + })) + // Defers run last-in-first-out, so release the handler BEFORE closing the + // server: Close waits on the in-flight request, and the shared fetch carries + // maxManifestFetch rather than this caller's spent deadline. + defer srv.Close() + defer close(release) + + r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ + TTL: time.Hour, Scheme: "http", HTTP: http.DefaultClient, // no timeout, deliberately + }) + + ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond) + defer cancel() + start := time.Now() + _, err := r.ResolveEndpoint(ctx, hostOf(t, srv)) + elapsed := time.Since(start) + + if err == nil { + t.Fatal("a call past its own deadline must fail") + } + if !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("error = %v, want the caller's own deadline to be the reason", err) + } + if elapsed > 2*time.Second { + t.Errorf("returned after %v; the caller's deadline was not honoured", elapsed) + } +} + +// ...and the leader walking away does not take the burst with it. The other half +// of the same property: the shared fetch outlives whoever triggered it, so a +// waiter with a live context still gets its answer. +func TestWellKnownEndpointResolver_leaderCancellationDoesNotPoisonWaiters(t *testing.T) { + var ep string + gate := make(chan struct{}) + var hits int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + atomic.AddInt32(&hits, 1) + <-gate // hold the fetch open until both callers are queued + _ = json.NewEncoder(w).Encode(map[string]any{"endpoint": ep}) + })) + defer srv.Close() + ep = srv.URL + "/ramp.v1.ExchangeService" + host := hostOf(t, srv) + + r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ + TTL: time.Hour, Scheme: "http", HTTP: http.DefaultClient, + }) + + leaderCtx, cancelLeader := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { defer close(done); _, _ = r.ResolveEndpoint(leaderCtx, host) }() + + waited := make(chan error, 1) + go func() { + time.Sleep(50 * time.Millisecond) // queue behind the leader's in-flight call + _, err := r.ResolveEndpoint(context.Background(), host) + waited <- err + }() + + time.Sleep(100 * time.Millisecond) + cancelLeader() // the leader walks away mid-fetch + close(gate) // let the origin answer + + <-done + if err := <-waited; err != nil { + t.Fatalf("a waiter with a live context must still get the endpoint: %v", err) + } + if n := atomic.LoadInt32(&hits); n != 1 { + t.Errorf("origin hit %d times, want 1 — the burst must coalesce", n) + } +} + +// panicTransport panics on every round trip, standing in for an +// application-supplied client that misbehaves. +type panicTransport struct{} + +func (panicTransport) RoundTrip(*http.Request) (*http.Response, error) { + panic("injected transport exploded") +} + +// A panic from an injected seam fails ONE lookup; it does not take the process +// down with it. +// +// The coalescing runs through singleflight.DoChan, and when a call has waiting +// channels singleflight re-raises a panic on a fresh goroutine — `go panic(e)` +// then `select{}` — deliberately, so waiters cannot block forever. Nothing above +// the closure can recover from that, which is why the closure recovers for +// itself. WellKnownOptions.HTTP and .Now are both application code, so this is a +// reachable input rather than a hypothetical. +func TestWellKnownEndpointResolver_survivesAPanickingTransport(t *testing.T) { + r := resolvers.NewWellKnownEndpointResolver(resolvers.WellKnownOptions{ + TTL: time.Hour, + Scheme: "http", + HTTP: &http.Client{Transport: panicTransport{}}, + }) + + got, err := r.ResolveEndpoint(context.Background(), "exchange.example") + if err == nil { + t.Fatalf("resolve returned %q; a panicking transport must surface as an error", got) + } + // The host is named so the failure is attributable, and the panic value is + // carried so it is diagnosable rather than merely reported. + if !strings.Contains(err.Error(), "exchange.example") || + !strings.Contains(err.Error(), "injected transport exploded") { + t.Errorf("error = %v, want it to name the host and carry the panic value", err) + } + + // The resolver is still usable afterwards: a panic must not have poisoned the + // singleflight key or the cache. + if _, err := r.ResolveEndpoint(context.Background(), "exchange.example"); err == nil { + t.Error("the second call must fail the same way, not succeed from a poisoned cache") + } +} diff --git a/sdk/go/resolvers/guardedclient_fromenv.go b/sdk/go/resolvers/guardedclient_fromenv.go index b32ebb9f..5fb076e5 100644 --- a/sdk/go/resolvers/guardedclient_fromenv.go +++ b/sdk/go/resolvers/guardedclient_fromenv.go @@ -55,19 +55,43 @@ func allowInsecure() bool { return envFlag(envAllowInsecure) } // (non-skip) path dials through a no-proxy transport, so a set HTTP(S)_PROXY // cannot tunnel a private target past the dial-time address pin. func NewGuardedClientFromEnv() *http.Client { - var base http.RoundTripper - if skipSSRF() { - base = http.DefaultTransport // no address guard - } else { - base = SSRFGuard(nil) // dial-time address pin, no proxy - } return &http.Client{ Timeout: defaultWBAHTTPTimeout, - Transport: &schemeGuardRoundTripper{base: base}, + Transport: NewGuardedTransport(nil), CheckRedirect: schemeCheckRedirect, } } +// NewGuardedTransport returns the guarded round-tripper — the scheme guard over +// the dial-time address pin — with base underneath it, honouring the same two env +// flags NewGuardedClientFromEnv reads. A nil base gets a fresh transport. +// +// base exists so a caller can carry its OWN transport settings (a tuned +// connection pool, client certificates) UNDER the guard rather than instead of +// it. That distinction is the whole point: every consumer of this transport dials +// a host some other party named, so the guard is not a default a caller may +// replace. Handing over a base is the supported way to customise the dial; the +// only way to drop the guard is the deliberate, deployment-level SKIP_SSRF / +// ALLOW_INSECURE opt-out. +// +// SSRFGuard clones what it is given, so the caller's value is never mutated, and +// drops the two settings that would route a dial around the address pin: a proxy, +// which would tunnel a private target past it, and a custom TLS dialer, which +// net/http prefers over the pinned dialer on https. TLSClientConfig is kept, so +// client certificates and a pinned root set are carried the way the pin allows. +func NewGuardedTransport(base *http.Transport) http.RoundTripper { + var inner http.RoundTripper + switch { + case !skipSSRF(): + inner = SSRFGuard(base) // dial-time address pin, no proxy + case base != nil: + inner = base.Clone() + default: + inner = http.DefaultTransport // no address guard + } + return &schemeGuardRoundTripper{base: inner} +} + // schemeGuardRoundTripper enforces the scheme policy on the INITIAL request // before any dial: https is always allowed, plaintext http only under // ALLOW_INSECURE, every other scheme denied. A denied scheme is refused up front diff --git a/sdk/go/resolvers/wbakeyresolver.go b/sdk/go/resolvers/wbakeyresolver.go index 9466ba48..aaf915aa 100644 --- a/sdk/go/resolvers/wbakeyresolver.go +++ b/sdk/go/resolvers/wbakeyresolver.go @@ -230,11 +230,13 @@ func anyAddrBlocked(addrs []netip.Addr) bool { // // base==nil yields a fresh, minimal transport; a non-nil base is cloned so the // caller's other transport settings are kept. In BOTH cases the guard forces -// Proxy=nil: a proxied transport dials the PROXY, so the dial-time check would vet -// the proxy's address instead of the true target — a full bypass. The dial-time -// SSRF pin and an egress proxy are therefore mutually exclusive by construction. +// Proxy=nil and clears any custom TLS dialer: each would route the dial around +// the address check, so each is mutually exclusive with the pin by construction. // Pair it with SSRFCheckRedirect on the *http.Client to also vet redirect schemes // and bound redirect depth. +// +// A caller's TLSClientConfig — client certificates, a pinned root set — is kept +// and still applies; only the dialer itself is dropped. func SSRFGuard(base *http.Transport) *http.Transport { if base == nil { base = &http.Transport{} @@ -245,6 +247,13 @@ func SSRFGuard(base *http.Transport) *http.Transport { // would resolve+check the PROXY, not the destination). Force it off so the // guard always vets the real target. base.Proxy = nil + // net/http prefers a transport's OWN TLS dialer over DialContext whenever the + // scheme is https — which is every RAMP leg — so a base carrying one would + // take the dial through the caller's dialer and the pin below would never run. + // The control would be silently absent rather than weaker, so both the current + // and the legacy field are cleared. Dial needs no such treatment: DialContext + // supersedes it whenever it is set, and it is set on the next line. + base.DialTLSContext, base.DialTLS = nil, nil base.DialContext = guardedDialContext return base } @@ -1262,20 +1271,23 @@ func wbaKeyActiveAt(k *rampv1.JsonWebKey, now time.Time) bool { return !now.Before(notBefore) && now.Before(notAfter) } -// wbaHostAnchored reports whether candidate's host is anchored to anchor — -// equal to it or a subdomain of it (case-insensitive, full label boundary, so -// "evil-a.com" is NOT a subdomain of "a.com"). candidate may be a full URL. +// wbaHostAnchored reports whether candidate is anchored to anchor — the same host +// and port, or a subdomain of that host on that port. +// +// The predicate itself is helpers.HostAnchored, which is the ONE place the rule is +// written; this wrapper exists for the two things that are local to WBA. It answers +// bool rather than (bool, error), because a directory that names an unparseable +// revocation_url is simply not anchored and its caller logs a skip. And it requires +// an ABSOLUTE reference: helpers.HostOf reads a schemeless value as https, which is +// right for an exchange domain but wrong here, where the scheme is the other half +// of the check and one branch below returns before reaching it. func wbaHostAnchored(anchor, candidate string) bool { u, err := url.Parse(candidate) if err != nil || u.Host == "" { return false } - a := strings.ToLower(strings.TrimSuffix(anchor, ".")) - c := strings.ToLower(strings.TrimSuffix(u.Host, ".")) - if a == "" { - return false - } - return c == a || strings.HasSuffix(c, "."+a) + anchored, err := helpers.HostAnchored(anchor, candidate) + return err == nil && anchored } // wbaRevocationAnchored reports whether a directory's advertised revocation_url diff --git a/sdk/go/resolvers/wbakeyresolver_directorybase_internal_test.go b/sdk/go/resolvers/wbakeyresolver_directorybase_internal_test.go index e8f7c34f..e29d622e 100644 --- a/sdk/go/resolvers/wbakeyresolver_directorybase_internal_test.go +++ b/sdk/go/resolvers/wbakeyresolver_directorybase_internal_test.go @@ -281,6 +281,24 @@ func TestRevocationAnchored_schemeMayNotDowngrade(t *testing.T) { {"http directory keeps http", "http://a.example", "a.example", "http://a.example/rev", true}, {"cross-host still refused", "https://a.example", "a.example", "https://evil.example/rev", false}, {"look-alike host refused", "https://a.example", "a.example", "https://evil-a.example/rev", false}, + // The port is compared, after folding a scheme's default into its + // omission — so writing :443 out is not a refusal, and naming another + // port is. + {"default port written out", "https://a.example", "a.example", "https://a.example:443/rev", true}, + {"another port refused", "https://a.example", "a.example", "https://a.example:8443/rev", false}, + {"ported directory keeps its port", "https://a.example:8443", "a.example:8443", "https://a.example:8443/rev", true}, + // The anchor here is a bare authority, so which port counts as the default + // is decided by the revocation_url's scheme rather than by an assumed https. + // Without that, a plaintext directory that spells :80 in full stops anchoring + // its own revocation_url — and a poll that is skipped leaves a revoked key + // resolving, which is worse than the spelling it was refusing. + {"plaintext directory spelling :80", "http://a.example:80", "a.example:80", "http://a.example:80/rev", true}, + {"plaintext directory, port spelled once", "http://a.example:80", "a.example:80", "http://a.example/rev", true}, + // A revocation_url must be ABSOLUTE. The shared predicate reads a + // schemeless reference as https, which is right for an exchange domain and + // wrong here — the base-less branch below returns before the scheme is + // ever compared, so nothing else would catch it. + {"schemeless candidate refused", "", "a.example", "a.example/rev", false}, // A directory cached before the base was threaded through falls back to the // host check rather than refusing every poll. {"empty base falls back to host only", "", "a.example", "http://a.example/rev", true}, diff --git a/sdk/go/resolvers/wbakeyresolver_ssrf_internal_test.go b/sdk/go/resolvers/wbakeyresolver_ssrf_internal_test.go index b434dd25..a0eed96d 100644 --- a/sdk/go/resolvers/wbakeyresolver_ssrf_internal_test.go +++ b/sdk/go/resolvers/wbakeyresolver_ssrf_internal_test.go @@ -4,6 +4,9 @@ package resolvers // unexported constructor, so it is exercised from inside the package. import ( + "context" + "crypto/tls" + "net" "net/http" "net/http/httptest" "net/netip" @@ -131,6 +134,47 @@ func TestSSRFGuardNilsProxy(t *testing.T) { } } +// TestSSRFGuardNilsCustomTLSDialers: net/http prefers a transport's own TLS +// dialer over DialContext for https, so a base carrying DialTLSContext (or the +// legacy DialTLS) would take the dial through the caller's dialer and the +// address pin would never run — on https, which is every RAMP leg. SSRFGuard +// must clear both, so a guarded client built over such a base still refuses a +// loopback target at the dial seam. +// +// The base also carries the server's own TLS config, which is what a caller +// customising TLS legitimately supplies: that must keep working, and only the +// dialer is dropped. +func TestSSRFGuardNilsCustomTLSDialers(t *testing.T) { + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + tlsCfg := srv.Client().Transport.(*http.Transport).TLSClientConfig + + base := &http.Transport{ + TLSClientConfig: tlsCfg, + DialTLSContext: func(_ context.Context, network, addr string) (net.Conn, error) { + return tls.Dial(network, addr, tlsCfg) + }, + DialTLS: func(network, addr string) (net.Conn, error) { + return tls.Dial(network, addr, tlsCfg) + }, + } + guarded := SSRFGuard(base) + if guarded.DialTLSContext != nil || guarded.DialTLS != nil { + t.Fatal("SSRFGuard left a custom TLS dialer installed — net/http prefers it over DialContext on https, so the address pin never runs") + } + if guarded.TLSClientConfig == nil { + t.Error("SSRFGuard dropped the caller's TLS config; only the dialer is mutually exclusive with the pin") + } + client := &http.Client{Transport: guarded} + if _, err := client.Get(srv.URL); err == nil { // srv.URL is loopback + t.Fatal("guarded client over a base with a custom TLS dialer reached a loopback target — the dial guard was bypassed") + } else if !strings.Contains(err.Error(), "SSRF guard") { + t.Fatalf("dial refused for the wrong reason (want SSRF guard): %v", err) + } +} + // TestGuardedClientRedirectDepthCap: the guarded client follows at most // maxWBARedirects redirect hops and refuses the next — the real-dial confirmation // that the client honors the shared redirect corpus (a chain exactly at the cap is diff --git a/sdk/parity/symbol-map.json b/sdk/parity/symbol-map.json index 0dc07008..e5e4ec48 100644 --- a/sdk/parity/symbol-map.json +++ b/sdk/parity/symbol-map.json @@ -1,16 +1,34 @@ { "_comment": "Symbol-level Go->Python->TS API-surface parity map. Go is the oracle (sdk/go/{helpers,resolvers,core,connect,connectserver}). One entry per Go public symbol (pkg-qualified). 'python'/'ts' hold the mapped public name or null when that language exposes no public symbol of that name. allowlist_reason marks a deliberate documented divergence (shrink-only). Enforced by sdk/python/tests/test_api_surface_parity.py. TS object factories use the create* prefix and value generators use generate* (Go NewX stays idiomatic); the guarded-client names (guarded_client/guarded_async_client vs guardedFetchFromEnv) are a documented idiomatic divergence, not a transliteration.", "go_exclusions": { - "connect.ClientOption": "Part of the Go-only typed Connect client; see the Connect-client DECISION in docs/sdk-parity-matrix.md.", - "connect.ExecuteOption": "Part of the Go-only typed Connect client; see the Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.BrokerClient": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.CallError": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.CallErrorKind": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.CallOption": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.ClientOption": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.DefaultCallTimeout": "Go default deadline for a call to an offer-derived Exchange. Python and TS ship no Connect client at all, so neither holds a counterpart today; the default arrives with their unary client work.", + "connect.DefaultMaxRPCReadBytes": "Go default response-size bound for a Connect call. Python and TS ship no unary client to hold one; the bound arrives with their unary client work.", + "connect.EndpointResolver": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.ExecuteOption": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.NewBrokerClient": "Part of the Go-only typed Connect client; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", "connect.NewValidateInterceptor": "Go-only protovalidate interceptor (matrix SERVER-role validation row: TS/Py absent).", - "connect.Validation": "Part of the Go-only typed Connect client validation option; see the Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.Validation": "Part of the Go-only typed Connect client validation option; see the OPEN Connect-client DECISION in docs/sdk-parity-matrix.md.", + "connect.WithAgentKey": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithClientOptions": "Go escape hatch for raw connectrpc.ClientOption values, mirroring connectserver.WithHandlerOptions; py/ts have no Connect option type to pass through.", + "connect.WithContentTimeout": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithEndpointResolver": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithGuardedBaseTransport": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithHTTPClient": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithIdempotencyKey": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithInterceptors": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithKeyResolver": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithMaxContentBytes": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithOfferKey": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithProofWindow": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithRequestIDFunc": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithRequester": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithSignWindow": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "connect.WithSignatureAgent": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithSigner": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithValidation": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connect.WithVerification": "Go functional-option builder; py/ts pass options via kwargs/options objects.", @@ -36,7 +54,9 @@ "connectserver.WithVerifyGate": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "connectserver.WithoutReplayStore": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "core.DefaultRequestID": "Go default request-id minter; py/ts mint request-ids inline.", + "core.DiscoveryResult": "Go per-URI discovery result carrying the fail-closed split plus the typed absence reasons; py/ts gain the same shape with their client verbs.", "core.ErrOfferExpired": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", + "core.OfferGroupResult": "Go per-URI group within a discovery result; py/ts gain the same shape with their client verbs.", "core.RequestIDFunc": "Go request-id function type; py/ts pass a callable inline.", "core.RequestIDMiddleware": "Go-only request-id middleware (matrix SERVER-role request-id row: TS/Py absent).", "core.SigningOption": "Go functional-option type for the signing transport; py/ts pass options objects.", @@ -44,6 +64,7 @@ "core.WithSignPredicate": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "core.WithSignatureAgent": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "core.WithWindow": "Go functional-option builder; py/ts pass options via kwargs/options objects.", + "helpers.AgentBinding": "Go value struct holding the three proof header values; Python returns a tuple of them and TS returns a prepared request, so neither names a public type.", "helpers.AgentIDParam": "Signed-URL query-parameter name; language-idiomatic inline constant, no cross-language public face.", "helpers.AlgEd25519": "RFC 9421 alg tag constant; inlined per language.", "helpers.AllSignaturesFromContext": "Go context.Context accessor; py/ts thread multisig state explicitly.", @@ -57,7 +78,10 @@ "helpers.ErrEmptyMoney": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrExpired": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrFutureCreated": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", + "helpers.ErrInvalidHost": "Go errors.Is sentinel for an unusable host reference; py/ts raise/throw instead of exporting sentinels.", "helpers.ErrInvalidKeyLength": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", + "helpers.ErrInvalidPoPInput": "Go errors.Is sentinel for a proof input that cannot be written into a signature base; py/ts raise/throw instead of exporting sentinels.", + "helpers.ErrKeyIDMismatch": "Go errors.Is sentinel for a keyid that is not the presented key's thumbprint; py/ts raise/throw instead of exporting sentinels.", "helpers.ErrMalformedSignatureInput": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrMissingContentDigest": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrMissingCreated": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", @@ -65,6 +89,7 @@ "helpers.ErrMissingRequiredComponent": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrMissingSignature": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrMissingSignatureInput": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", + "helpers.ErrMissingTargetURI": "Go errors.Is sentinel for a proof requested without the URL it binds; py/ts raise/throw instead of exporting sentinels.", "helpers.ErrOfferExpired": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrOfferSignatureInvalid": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrProofOfPossessionMismatch": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", @@ -79,11 +104,18 @@ "helpers.ErrUnknownFields": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.ErrUnsupportedAlgorithm": "Go errors.Is sentinel; py/ts express verification failures via typed failure unions / exception classes, not per-reason named sentinels.", "helpers.FromContext": "Go context.Context accessor; py/ts thread verified-request state explicitly.", + "helpers.HostAnchored": "Go label-boundary same-host-and-port predicate, folding a scheme's default port into an omitted one. Python and TS carry a PRIVATE near-namesake in their WBA modules, neither of which is a counterpart: TS's compares URL.host, which normalizes the default port but is not exported; Python's compares netloc, which keeps an explicit :443 AND includes userinfo. Exporting an aligned predicate in all three is tracked separately.", + "helpers.HostOf": "Go host-extraction helper behind the two routing predicates. No Python or TS equivalent exists today, private or otherwise; it is a prerequisite of the tracked cross-language endpoint-rule work.", + "helpers.IsBareHost": "Go plain-hostname predicate for the report leg's first check. No Python or TS equivalent exists today; exporting the pair in both is part of the tracked cross-language endpoint-rule work.", "helpers.NewContext": "Go context.Context accessor; py/ts thread verified-request state explicitly.", "helpers.NewEd25519Signer": "Go Ed25519 Signer constructor; py/ts inject a sign function rather than constructing a named signer.", "helpers.NewEd25519SignerFromSeed": "Go Ed25519 Signer-from-seed constructor; py/ts inject a sign function rather than constructing a named signer.", "helpers.NewMultisigContext": "Go context.Context accessor; py/ts thread multisig state explicitly.", + "helpers.PoPOptions": "Go options struct for the delivery-proof signer; Python takes the same values as keyword arguments and TS as an options object.", + "helpers.RedactURL": "Go query-stripping helper for a signed URL headed to a log; py/ts redact inline at the log site.", + "helpers.RetrievalAuthFailureReasonFromToken": "Go lookup from the delivery edge's refusal token to the typed enum; py/ts branch on the token string directly.", "helpers.SharedValidator": "Go protovalidate validator singleton; TS/Python ship no protovalidate face.", + "helpers.SignOfferAcceptanceWith": "Go Signer-custody variant of SignOfferAcceptance so the SDK never holds the key; py/ts pass key material directly to their single acceptance signer.", "helpers.SignOptions": "Go options struct for SignRequest; py/ts pass options via kwargs/options objects.", "helpers.SignatureAgentFromContext": "Go context.Context accessor; py/ts thread signature-agent state explicitly.", "helpers.SignedURL": "Go signed-URL result value type; py/ts return language-native result objects.", @@ -99,12 +131,23 @@ "helpers.VerifyRequestResolved": "Go resolver-injected VerifyRequest overload; py/ts expose a single verify entry point.", "helpers.WithSignatureAgent": "Go functional-option builder; py/ts pass options via kwargs/options objects.", "resolvers.ActiveKeyScanOptions": "Go scan-options struct; py/ts pass scan options inline.", + "resolvers.Content": "Go value struct for one fetched resource; py/ts return their runtime-native body/type pair.", + "resolvers.ContentFetchOptions": "Go options struct for the content-download leg; py/ts pass the same values as kwargs/an options object.", + "resolvers.ContentFetcher": "Part of the Go content-download leg; the TS/Python download verbs are tracked with their unary client work.", + "resolvers.DefaultContentTimeout": "Go default bound for one content fetch. Neither Python nor TS ships a content fetcher, so no counterpart holds this default today; it arrives with the download verbs tracked alongside their unary client work.", + "resolvers.DefaultMaxContentBytes": "Go default body cap for one content fetch. Neither Python nor TS ships a content fetcher, so no counterpart holds this cap today; it arrives with the download verbs tracked alongside their unary client work.", + "resolvers.ErrEndpointRefused": "Go errors.Is sentinel for a manifest-advertised endpoint the resolver will not return (wrong host, or userinfo). Python and TS DO ship endpoint resolvers, and neither enforces this rule yet; closing that gap is tracked separately and will bring a mapped counterpart.", + "resolvers.FetchError": "Go typed error for the content leg; py/ts raise/throw a runtime-native error carrying the same class and reason.", + "resolvers.FetchFailure": "Go failure-class enum for the content leg; py/ts express the same classes as string literals.", + "resolvers.NewContentFetcher": "Go constructor for the content-download leg; py/ts fold construction into their client.", + "resolvers.NewGuardedTransport": "Go constructor composing the SSRF guard over a caller's base transport; py/ts expose their guarded fetch as a single factory with no separable base.", + "resolvers.ProofSigner": "Go interface seam that keeps key custody out of the dialing tier; py/ts inject a signing callable instead of a named interface.", "resolvers.SSRFCheckRedirect": "Go redirect-policy hook; py/ts fold redirect checks into the guard (async_ssrf_guard / the guarded fetch)." }, "symbols": { "connect.Client": { - "allowlist_reason": "Go-only typed Connect client (Discover->Execute orchestration) — deliberate runtime-native divergence, DECISION resolved in docs/sdk-parity-matrix.md", - "decision_anchor": "typed Connect **client**", + "allowlist_reason": "Go-only typed Connect client covering the agent verb set (Discover, Resolve, Execute, ReportUsage, Dispute, Fetch) — OPEN DECISION in docs/sdk-parity-matrix.md: the API-surface design governs and specifies a thin Connect-unary JSON client for TypeScript and Python with the SAME verb names, so this is an implementation difference pending that work, not a settled API divergence", + "decision_anchor": "typed Connect **client** — OPEN", "python": null, "ts": null }, @@ -114,8 +157,8 @@ "ts": "errorDetailFrom" }, "connect.NewClient": { - "allowlist_reason": "Go-only typed Connect client constructor — deliberate runtime-native divergence, DECISION resolved in docs/sdk-parity-matrix.md", - "decision_anchor": "typed Connect **client**", + "allowlist_reason": "Go-only typed Connect client constructor (NewClient plus NewBrokerClient) — OPEN DECISION in docs/sdk-parity-matrix.md, pending the TypeScript and Python unary client that carries the same verb names", + "decision_anchor": "typed Connect **client** — OPEN", "python": null, "ts": null }, @@ -201,6 +244,11 @@ "python": "ACCEPTANCE_SIGNATURE_ALGORITHM", "ts": "ACCEPTANCE_SIGNATURE_ALGORITHM" }, + "helpers.AgentKeyHeader": { + "allowlist_reason": null, + "python": "AGENT_KEY_HEADER", + "ts": "AGENT_KEY_HEADER" + }, "helpers.AppendSignature": { "allowlist_reason": null, "python": "append_signature", @@ -341,6 +389,11 @@ "python": "scopes_subset", "ts": "scopesSubset" }, + "helpers.SignAgentBinding": { + "allowlist_reason": null, + "python": "sign_agent_binding", + "ts": "signInbound" + }, "helpers.SignOffer": { "allowlist_reason": null, "python": "sign_offer_jcs", diff --git a/sdk/python/ramp_sdk/__init__.py b/sdk/python/ramp_sdk/__init__.py index a5d1693c..d30ee4b0 100644 --- a/sdk/python/ramp_sdk/__init__.py +++ b/sdk/python/ramp_sdk/__init__.py @@ -64,7 +64,7 @@ from .idempotency import generate_idempotency_key, validate_idempotency_key from .keyresolver import KeyResolver, StaticKeyResolver from .money import canonicalize_money, format_money, parse_money -from .pop import sign_agent_binding, verify_agent_binding +from .pop import AGENT_KEY_HEADER, sign_agent_binding, verify_agent_binding from .resolvers import ( WBA_DIRECTORY_PATH, DirectoryUnavailableError, @@ -95,6 +95,7 @@ __all__ = [ "ACCEPTANCE_SIGNATURE_ALGORITHM", + "AGENT_KEY_HEADER", "ERROR_DETAIL_TYPE", "OFFER_SIGNATURE_ALGORITHM", "REASON_FIELDS", diff --git a/sdk/python/ramp_sdk/pop.py b/sdk/python/ramp_sdk/pop.py index c417ab9e..455b602f 100644 --- a/sdk/python/ramp_sdk/pop.py +++ b/sdk/python/ramp_sdk/pop.py @@ -174,6 +174,23 @@ def sign_agent_binding( to the sdk/go signer (pinned by pop-vectors.json). ``created``/``expires`` are injected unix seconds: the helper reads no clock (L1-pure). """ + # The signature base is line-delimited and ``url`` is written into it verbatim, + # so a control byte would add or split a component line and the bytes signed + # here would stop describing the request a verifier reconstructs. Refused + # rather than escaped: no legitimate target URI contains one, and a signature + # base is the wrong place to be lenient. Mirrors the Go signer, which refuses + # the same bytes in both the method and the URL; the method arm does not apply + # here because this face always signs GET. + # + # Scanned over the UTF-8 BYTES, not the code points, so the reported offset is + # the same number Go's strings.IndexFunc reports for the same input. Which + # inputs are refused is unaffected — a control byte is always a single UTF-8 + # byte — but an unlabelled index under identical wording meant three units. + raw = url.encode("utf-8") + bad = next((i for i, b in enumerate(raw) if b < 0x20 or b == 0x7F), None) + if bad is not None: + raise ValueError(f"target URI carries a control byte at byte {bad}") + priv = Ed25519PrivateKey.from_private_bytes(signer_seed) pub = priv.public_key().public_bytes_raw() keyid = thumbprint(pub) diff --git a/sdk/python/tests/test_api_surface_parity.py b/sdk/python/tests/test_api_surface_parity.py index 1d41f282..2d7be742 100644 --- a/sdk/python/tests/test_api_surface_parity.py +++ b/sdk/python/tests/test_api_surface_parity.py @@ -299,22 +299,62 @@ def completeness_failures(go_symbols: dict[str, str], parity_map: ParityMap) -> ] +def staleness_failures(go_symbols: dict[str, str], parity_map: ParityMap) -> list[str]: + """Every map key must still name a live Go public symbol. + + The mirror of completeness_failures. That walk starts from the live Go + surface, so it can only see a symbol that was ADDED without a map entry; an + entry left behind when its symbol was REMOVED is invisible to it, and stays + in the published matrix indefinitely claiming a surface that no longer + exists. Both directions are needed to keep the map an accurate description + of the oracle. + + The live enumeration covers exactly _GO_PACKAGES, so a key naming any other + package reads as stale — correctly: the gate claims authority over those + packages and nothing else. + """ + accounted = set(parity_map["symbols"]) | set(parity_map["go_exclusions"]) + return [ + f"{key}: mapped in symbol-map.json but no longer exported by Go — a removed " + f"symbol left in the map keeps appearing in the generated parity matrix" + for key in sorted(accounted - set(go_symbols)) + ] + + # --------------------------------------------------------------------------- # # (ii) COMPLETENESS — needs the Go toolchain # --------------------------------------------------------------------------- # +def _skip_unless_go(half: str) -> None: + """Skip LOUDLY when the Go oracle cannot be enumerated. + + Loud because a silent skip here reads as a passing gate: without `go` on PATH + neither half can run at all, and the whole point of this file is that the Go + surface is the oracle. CI's sdk-l1 job has actions/setup-go and DOES run it. + """ + if _go_available(): + return + pytest.skip( + f"LOUD SKIP: `go` toolchain not on PATH — cannot enumerate the Go oracle " + f"surface, so the {half} half of the API-parity gate cannot run. CI's " + "sdk-l1 job has actions/setup-go and DOES run it; a green local run without " + "Go is NOT a green gate." + ) + + def test_every_go_public_symbol_is_mapped_or_excluded() -> None: - if not _go_available(): - pytest.skip( - "LOUD SKIP: `go` toolchain not on PATH — cannot enumerate the Go oracle " - "surface, so the COMPLETENESS half of the API-parity gate cannot run. CI's " - "sdk-l1 job has actions/setup-go and DOES run it; a green local run without " - "Go is NOT a green gate." - ) + _skip_unless_go("COMPLETENESS") parity_map = _load_map() failures = completeness_failures(enumerate_go(), parity_map) assert not failures, "unmapped Go public symbols:\n " + "\n ".join(failures) +def test_no_map_entry_outlives_its_go_symbol() -> None: + _skip_unless_go("STALENESS") + parity_map = _load_map() + failures = staleness_failures(enumerate_go(), parity_map) + assert not failures, "stale symbol-map entries:\n " + "\n ".join(failures) + + # --------------------------------------------------------------------------- # # (i) PRESENCE — Python + TS surfaces only (no Go toolchain needed) # --------------------------------------------------------------------------- # @@ -496,3 +536,26 @@ def test_completeness_bites_on_an_unmapped_go_symbol() -> None: injected = {"helpers.FakeUnmappedSymbol": "FakeUnmappedSymbol"} failures = completeness_failures(injected, parity_map) assert any("FakeUnmappedSymbol" in f for f in failures), failures + + +def test_staleness_bites_on_a_map_entry_with_no_go_symbol() -> None: + """A map entry naming a symbol Go no longer exports MUST turn the gate RED. + + Checked on BOTH halves: an exclusion is the half that outlived its symbol in + practice, and a `symbols` entry would be just as invisible to the + live-Go-first walk. + """ + live = {"helpers.RealSymbol": "RealSymbol"} + excluded_stale: ParityMap = { + "symbols": {}, + "go_exclusions": {"helpers.RealSymbol": "reason", "connect.GoneOption": "reason"}, + } + failures = staleness_failures(live, excluded_stale) + assert any("connect.GoneOption" in f for f in failures), failures + assert not any("RealSymbol" in f for f in failures), failures + + mapped_stale: ParityMap = { + "symbols": {"helpers.GoneType": {"python": "gone", "ts": "gone", "allowlist_reason": None}}, + "go_exclusions": {}, + } + assert any("helpers.GoneType" in f for f in staleness_failures(live, mapped_stale)) diff --git a/sdk/python/tests/test_parity_matrix_connect_client_decision.py b/sdk/python/tests/test_parity_matrix_connect_client_decision.py index b4a0d277..0fa63874 100644 --- a/sdk/python/tests/test_parity_matrix_connect_client_decision.py +++ b/sdk/python/tests/test_parity_matrix_connect_client_decision.py @@ -1,11 +1,21 @@ """Doc-guard: the Go-only Connect client divergence must be a RECORDED decision. The whole point of the Connect-client parity work is to END the silent absence: -Go ships a typed Connect client (NewClient/Discover/Execute) and Python + TS do not. -The resolution is option (b) — document it as a deliberate runtime-native divergence -in docs/sdk-parity-matrix.md, mirroring the sibling server-handler DECISION. This -guard fails if that DECISION bullet is ever removed, so the gap cannot silently -reappear or be miscounted as "parity complete". +Go ships a typed Connect client and Python + TS do not. docs/sdk-parity-matrix.md +must carry that as a DECISION bullet, mirroring the sibling server-handler one, so +the gap cannot silently reappear or be miscounted as "parity complete". + +The decision is OPEN, not settled. The API-surface design document governs, and it +specifies a thin Connect-unary JSON client for TypeScript and Python carrying the +SAME verb names — so what stays divergent is the transport implementation, not the +API. An earlier wording called this a "deliberate runtime-native divergence, +DECISION resolved"; that was an allowlist reason written in the grammar of a +decision, and it was substantively wrong, since every RAMP RPC is unary and +nothing about it is runtime-native. + +This guard asserts only that the bullet EXISTS and still names the client and its +verbs. It deliberately does not assert the resolved-versus-open wording: that is +the part expected to change when the other two languages gain their client. """ from __future__ import annotations @@ -20,14 +30,18 @@ def test_connect_client_divergence_is_a_recorded_decision() -> None: """docs/sdk-parity-matrix.md records the Go-only Connect client as intentional.""" text = _MATRIX.read_text(encoding="utf-8") assert "DECISION —" in text, "parity matrix carries no DECISION bullets" - # The Connect-client decision must name the client and the runtime-native rationale. + # The decision must name the client. It deliberately does NOT assert the old + # "runtime-native" rationale, which the docstring above records as wrong: every + # RAMP RPC is unary, and the design specifies the same verbs in all three + # languages, so what diverges is the transport, not the API. lowered = text.lower() assert "connect client" in lowered or "connect-client" in lowered, ( "no Connect-client decision recorded in the parity matrix — the Go-only " "typed client gap is still silent" ) - # It must be framed as a deliberate divergence, not an open TODO. + # It must name the verbs it covers, so a reader can tell what "the Go client" + # means without reading the code. assert "discover" in lowered and "execute" in lowered, ( - "the Connect-client decision must reference the Discover/Execute " - "offer-lifecycle orchestration that scopes the reopen trigger" + "the Connect-client decision must name the verbs the Go client covers, " + "which is what scopes the divergence" ) diff --git a/sdk/python/tests/test_pop_sign_helper.py b/sdk/python/tests/test_pop_sign_helper.py index b8b0aa7e..ac6fc398 100644 --- a/sdk/python/tests/test_pop_sign_helper.py +++ b/sdk/python/tests/test_pop_sign_helper.py @@ -151,3 +151,43 @@ def test_sign_agent_binding_in_ramp_sdk_all() -> None: import ramp_sdk assert "sign_agent_binding" in ramp_sdk.__all__ + + +# ---- control bytes in the target URI ------------------------------------- + + +@pytest.mark.parametrize( + "url", + [ + 'https://cdn.test/a\n"@authority": evil.test', + "https://cdn.test/a\r", + "https://cdn.test/a\x00", + "https://cdn.test/a\x7f", + ], + ids=["newline", "carriage_return", "nul", "delete"], +) +def test_sign_agent_binding_refuses_control_bytes_in_the_url(url: str) -> None: + """A control byte in the URL must be refused, not signed. + + The signature base is line-delimited and the URL is written into it verbatim, + so a newline would add or split a component line and the signed bytes would + stop describing the request a verifier reconstructs. Mirrors the Go signer's + refusal (helpers.SignAgentBinding / ErrInvalidPoPInput); Python raises rather + than exporting a sentinel, which is the mapped-correct shape. + + Parametrized rather than looped so a regression names the offending case: in a + loop, a DID-NOT-RAISE failure reports the line and not which URL reached it. + """ + seed = bytes(range(32)) + with pytest.raises(ValueError, match="control byte"): + sign_agent_binding(url=url, signer_seed=seed, created=1, expires=2) + + +def test_sign_agent_binding_still_signs_an_ordinary_url() -> None: + """The refusal is narrow: a normal URL, and a percent-encoded one, still sign.""" + seed = bytes(range(32)) + for url in ("https://cdn.test/a?agent_id=x", "https://cdn.test/a%20b%2Fc"): + key, sig_input, sig = sign_agent_binding( + url=url, signer_seed=seed, created=1, expires=2 + ) + assert key and sig_input.startswith("sig1=") and sig.startswith("sig1=:") diff --git a/sdk/ts/core/sign.ts b/sdk/ts/core/sign.ts index 6fcf0dcc..c0e0c348 100644 --- a/sdk/ts/core/sign.ts +++ b/sdk/ts/core/sign.ts @@ -77,6 +77,23 @@ export async function signInbound( // string form ONCE at the boundary, so the signed @target-uri and the emitted // Request carry the same verbatim bytes. No-op for string callers. const target = opaqueUrl(url); + // The signature base is line-delimited and `target` is written into it + // verbatim, so a control byte would add or split a component line and the bytes + // signed here would stop describing the request a verifier reconstructs. + // Refused rather than escaped, mirroring the Go signer. Checked AFTER the + // coercion above so it inspects the bytes that actually get signed — and before + // `new Request(target, …)` below, which would otherwise throw an opaque + // TypeError instead of naming the reason. + // Scanned over the UTF-8 BYTES, not the code points, so the reported offset is + // the same number Go's strings.IndexFunc reports for the same input. Which + // inputs are refused is unaffected — a control byte is always a single UTF-8 + // byte — but an unlabelled index under identical wording meant three units. + const badAt = new TextEncoder() + .encode(target) + .findIndex((b) => b < 0x20 || b === 0x7f); + if (badAt !== -1) { + throw new TypeError(`target URI carries a control byte at byte ${badAt}`); + } const base = signatureBase("GET", target, rawParams); const sig = await crypto.subtle.sign( "Ed25519", diff --git a/sdk/ts/tests/pop-control-bytes.regression.test.ts b/sdk/ts/tests/pop-control-bytes.regression.test.ts new file mode 100644 index 00000000..9b36c3ec --- /dev/null +++ b/sdk/ts/tests/pop-control-bytes.regression.test.ts @@ -0,0 +1,55 @@ +// The GET-PoP sign face refuses control bytes in the target URI. +// +// The signature base is line-delimited and the target is written into it verbatim, +// so a newline would add or split a component line and the signed bytes would stop +// describing the request a verifier reconstructs. Refused rather than escaped, +// mirroring the Go signer (helpers.SignAgentBinding / ErrInvalidPoPInput). TS +// throws rather than exporting a sentinel, which is the mapped-correct shape per +// the parity map. +// +// Go refuses the same bytes in BOTH the method and the URL; only the URL arm is +// mirrored here because this face always signs GET. + +import { describe, expect, it } from "vitest"; + +import { signInbound } from "../core/sign.ts"; + +const now = () => 1_700_000_000_000; + +async function keypair(): Promise { + return (await crypto.subtle.generateKey({ name: "Ed25519" }, true, [ + "sign", + "verify", + ])) as CryptoKeyPair; +} + +describe("signInbound refuses control bytes in the target URI", () => { + const cases: Record = { + newline: 'https://cdn.test/a\n"@authority": evil.test', + "carriage return": "https://cdn.test/a\r", + nul: "https://cdn.test/a\x00", + delete: "https://cdn.test/a\x7f", + }; + + for (const [name, url] of Object.entries(cases)) { + it(`refuses a ${name}`, async () => { + const kp = await keypair(); + await expect( + signInbound(kp, url, { now, ttlSec: 300 }), + ).rejects.toThrow(/control byte/); + }); + } + + // The refusal is narrow: percent-encoded bytes are not control bytes and must + // still sign, since the target URI is carried verbatim. + it("still signs an ordinary and a percent-encoded URL", async () => { + const kp = await keypair(); + for (const url of [ + "https://cdn.test/a?agent_id=x", + "https://cdn.test/a%20b%2Fc", + ]) { + const req = await signInbound(kp, url, { now, ttlSec: 300 }); + expect(req.headers.get("signature-input")).toMatch(/^sig1=/); + } + }); +}); diff --git a/website/src/components/DeferredSdkNotice.mdx b/website/src/components/DeferredSdkNotice.mdx new file mode 100644 index 00000000..eb047d72 --- /dev/null +++ b/website/src/components/DeferredSdkNotice.mdx @@ -0,0 +1,7 @@ +import { Aside } from '@astrojs/starlight/components'; + + diff --git a/website/src/components/DeferredSdkNoticeDetail.mdx b/website/src/components/DeferredSdkNoticeDetail.mdx new file mode 100644 index 00000000..afb04cee --- /dev/null +++ b/website/src/components/DeferredSdkNoticeDetail.mdx @@ -0,0 +1,14 @@ +import { Aside } from '@astrojs/starlight/components'; + + diff --git a/website/src/content/docs/components/agent-sdk/budget-reporting.mdx b/website/src/content/docs/components/agent-sdk/budget-reporting.mdx index 754c670b..85fbd281 100644 --- a/website/src/content/docs/components/agent-sdk/budget-reporting.mdx +++ b/website/src/content/docs/components/agent-sdk/budget-reporting.mdx @@ -3,6 +3,10 @@ title: "Budget and Usage Reporting" description: "How the RAMP Agent SDK tracks budgets across three enforcement layers and automatically submits usage reports to Exchanges." --- +import DeferredSdkNoticeDetail from '../../../../components/DeferredSdkNoticeDetail.mdx'; + + + ## Budget Management The SDK enforces three budget layers, checked in order before any network call is made. A budget-exceeded condition never reaches the wire. diff --git a/website/src/content/docs/components/agent-sdk/fetch-flow.mdx b/website/src/content/docs/components/agent-sdk/fetch-flow.mdx index 6bbf3172..b87caf3d 100644 --- a/website/src/content/docs/components/agent-sdk/fetch-flow.mdx +++ b/website/src/content/docs/components/agent-sdk/fetch-flow.mdx @@ -3,6 +3,10 @@ title: "Fetch Flow" description: "Step-by-step walkthrough of how the RAMP Agent SDK discovers, transacts, and fetches licensed content -- from URL to delivered content." --- +import DeferredSdkNoticeDetail from '../../../../components/DeferredSdkNoticeDetail.mdx'; + + + ## Single URL Fetch Step-by-step what happens inside `client.Fetch(ctx, url)`: @@ -320,10 +324,13 @@ Per-period persistence: by default, the SDK writes period budget state to a loca ## Testing with Mock Exchange -The SDK ships a `MockExchange` for testing agent code without network calls: +The design pairs the high-tier client with a `MockExchange` for testing agent code +without network calls. Like the rest of this page it is part of the deferred +design: no such package ships today, and the sample below is written against the +planned API rather than the shipped one. ```go -import "github.com/RAMP-Protocol/protocol/sdk-go/testutil" +import "example.com/ramp-agent-sdk/testutil" // planned; not a shipped package func TestAgentLogic(t *testing.T) { mock := testutil.NewMockExchange(testutil.MockConfig{ diff --git a/website/src/content/docs/components/agent-sdk/overview.mdx b/website/src/content/docs/components/agent-sdk/overview.mdx index bc245fa5..5d6b5f47 100644 --- a/website/src/content/docs/components/agent-sdk/overview.mdx +++ b/website/src/content/docs/components/agent-sdk/overview.mdx @@ -3,8 +3,12 @@ title: "Agent SDK Overview" description: "The RAMP Agent SDK is an embeddable library that gives AI agents one-liner content access -- a single Fetch call handling discovery, negotiation, transaction, delivery, budget enforcement, and usage reporting." --- +import DeferredSdkNoticeDetail from '../../../../components/DeferredSdkNoticeDetail.mdx'; + import { Aside } from '@astrojs/starlight/components'; + + ## What the SDK Provides The RAMP Agent SDK is a library that agent developers embed directly in their application process. It provides one-liner content access -- a single `Fetch(url)` call that handles discovery, negotiation, transaction, delivery, budget enforcement, and usage reporting behind the scenes. @@ -430,7 +434,7 @@ The concrete implementations differ in persistence strategy (in-memory vs Redis) | Language | Source | Status | Dependencies | |---|---|---|---| -| Go | [`gen/go/`](https://github.com/RAMP-Protocol/protocol/tree/main/gen/go/) (types) + [`gen/go/ramp/v1/rampv1connect`](https://github.com/RAMP-Protocol/protocol/tree/main/gen/go/ramp/v1/rampv1connect) (Connect client) | Generated, usable via Go modules | `connectrpc.com/connect` + `google.golang.org/protobuf` + stdlib | +| Go | [`gen/go/`](https://github.com/RAMP-Protocol/protocol/tree/main/gen/go/) (types) + [`sdk/go/`](https://github.com/RAMP-Protocol/protocol/tree/main/sdk/go/) (the shipped hand-written SDK: `helpers`, `resolvers`, `core`, `connect`, `connectserver`) | Shipped and CI-gated | `connectrpc.com/connect` + `google.golang.org/protobuf` + stdlib | | TypeScript | — | Planned (npm package not yet published) | `@connectrpc/connect` + `@bufbuild/protobuf` | | Python | — | Planned (PyPI package not yet published) | `httpx` + `fastmcp` | diff --git a/website/src/content/docs/components/broker/deployment.mdx b/website/src/content/docs/components/broker/deployment.mdx index 453366ff..e4f98ece 100644 --- a/website/src/content/docs/components/broker/deployment.mdx +++ b/website/src/content/docs/components/broker/deployment.mdx @@ -3,6 +3,10 @@ title: "Broker Deployment" description: "Deployment models, configuration reference, caching strategy, failure modes, and scaling considerations for the RAMP Broker." --- +import DeferredSdkNotice from '../../../../components/DeferredSdkNotice.mdx'; + + + ## Deployment Models The Broker is designed for three deployment modes. The core logic (selection, budget, reporting) is identical across all three; what changes is the process model, configuration surface, and what the agent is responsible for. diff --git a/website/src/content/docs/components/mcp-server/overview.mdx b/website/src/content/docs/components/mcp-server/overview.mdx index 09225445..abd28e8a 100644 --- a/website/src/content/docs/components/mcp-server/overview.mdx +++ b/website/src/content/docs/components/mcp-server/overview.mdx @@ -3,6 +3,10 @@ title: "MCP Server" description: "Thin proxy that exposes RAMP ExchangeService as MCP tools -- zero-SDK resource access for any MCP-capable AI agent." --- +import DeferredSdkNotice from '../../../../components/DeferredSdkNotice.mdx'; + + + ## What It Is The RAMP MCP Server is a convenience interface layer that exposes ExchangeService RPCs as MCP tools. Any AI agent with MCP support can discover and access resources without custom SDK integration -- just add the MCP server to the agent's tool configuration. diff --git a/website/src/content/docs/getting-started/for-ai-agents.mdx b/website/src/content/docs/getting-started/for-ai-agents.mdx index 464d5037..ce424d57 100644 --- a/website/src/content/docs/getting-started/for-ai-agents.mdx +++ b/website/src/content/docs/getting-started/for-ai-agents.mdx @@ -3,6 +3,10 @@ title: "RAMP for AI Agents" description: "How AI agents access any metered resource — articles, credit reports, drug databases, satellite imagery, court filings — through one protocol" --- +import DeferredSdkNotice from '../../../components/DeferredSdkNotice.mdx'; + + + ## The Problem You Have Your agent is building a due-diligence report. It needs a D&B credit report (one API, one auth scheme, one pricing model). It needs a PACER court filing (completely different API). It needs drug interaction data from DrugBank (yet another). Academic literature from Elsevier (yet another). A satellite image of a property from Planet (yet another). diff --git a/website/src/content/docs/getting-started/poc-walkthrough.mdx b/website/src/content/docs/getting-started/poc-walkthrough.mdx index 678840e5..59717a94 100644 --- a/website/src/content/docs/getting-started/poc-walkthrough.mdx +++ b/website/src/content/docs/getting-started/poc-walkthrough.mdx @@ -3,6 +3,10 @@ title: "PoC Walkthrough" description: "End-to-end demonstration of the RAMP protocol using the live reference implementation. Follow along with curl or integrate the MCP tool into your AI agent." --- +import DeferredSdkNotice from '../../../components/DeferredSdkNotice.mdx'; + + + import { Aside } from '@astrojs/starlight/components'; :::note[Reference Implementation] diff --git a/website/src/content/docs/protocol/exchange-manifest.mdx b/website/src/content/docs/protocol/exchange-manifest.mdx index 95dc2170..d61da1a9 100644 --- a/website/src/content/docs/protocol/exchange-manifest.mdx +++ b/website/src/content/docs/protocol/exchange-manifest.mdx @@ -106,7 +106,7 @@ The Exchange's offer-signing keys are **not** carried in `ramp.json`. They live | `name` | string | No | Human-readable Exchange name | | `operator` | string | No | Organization operating this Exchange | | `operator_domain` | string | No | Operator's corporate domain | -| `endpoint` | string | No | ExchangeService endpoint URL | +| `endpoint` | string | No | ExchangeService endpoint URL. **Must be on the host and port that serve this manifest, or a subdomain of that host on that port, and must not carry userinfo** — see [Endpoint host binding](#endpoint-host-binding) | | (offer-signing keys) | — | — | Not in `ramp.json`. See the [WBA directory](#the-wba-directory) at `/.well-known/http-message-signatures-directory` | | `health_endpoint` | string | No | Health check endpoint URL | | `catalog_endpoint` | string | No | CatalogService endpoint URL | @@ -126,6 +126,47 @@ The Exchange's offer-signing keys are **not** carried in `ramp.json`. They live | `privacy_uri` | string | No | Privacy policy URL | | `ext` | object | No | Extension fields (forward-compatible) | +### Endpoint host binding + +`endpoint` **MUST** be on the same host **and port** that serve this manifest, or +on a subdomain of that host on that port, and **MUST NOT** carry userinfo. An +Exchange at `exchange.example` may advertise `https://exchange.example/v1` or +`https://api.exchange.example/v1`; it may not advertise +`https://cdn.other.example/v1`, nor `https://exchange.example:8443/v1` unless its +`ramp.json` is served from `:8443` too. + +The anchor is the host the consumer **fetched this document from** — not the +`domain` field inside it. That distinction is the whole check: `domain` is +self-asserted, so a hostile manifest that anchored to it could simply set `domain` +to match whatever endpoint it wanted and validate itself. A conformant Exchange +has the two agree, which is why the difference only shows against a manifest worth +refusing. + +The reason is the trust chain. This document is fetched from the domain an offer +named, and it is only as trustworthy as the host that served it. If it could name +an endpoint on an unrelated host, whoever answers for the manifest could redirect +a signed call to a party the offer's signature never covered — and a dial-time +address guard would not object, because the destination is a perfectly ordinary +public host. + +Two details matter in practice: + +- The host match is on a **full dot-delimited label boundary**. `evil-a.com` is + not a subdomain of `a.com`; a bare suffix comparison gets that wrong, and it is + the mistake an attacker registers a domain to exploit. +- The **port is compared too**. Another port is another service, which the party + publishing the manifest need not control, so `https://exchange.example:8443/v1` + is a valid endpoint only where the port is named on both sides. A port equal to + the scheme's default and an omitted port are the **same** port, so + `https://exchange.example`, `https://exchange.example:443` and + `exchange.example` all match one another — writing `:443` out is not a refusal. +- The **scheme is not compared** here. Whether a leg may run in the clear is the + transport's decision, and the default-port rule above is scheme-relative so that + it cannot become a scheme check by accident. + +An Exchange that serves its API from a separate domain should front it under its +own subdomain rather than advertising the other domain directly. + ## The Unified RAMP Well-Known File RAMP defines **one** well-known file. Every participant serves a `WellKnownManifest` at `/.well-known/ramp.json`; the `role` field says which participant it is. @@ -196,7 +237,7 @@ message WellKnownManifest { optional string name = 9; optional string operator = 10; optional string operator_domain = 11; - optional string endpoint = 12; // ExchangeService URL + optional string endpoint = 12; // ExchangeService URL (serving host+port, or a subdomain) optional string health_endpoint = 13; optional string catalog_endpoint = 14; repeated string protocol_versions_supported = 16; diff --git a/website/src/content/docs/reference/changelog.mdx b/website/src/content/docs/reference/changelog.mdx index 4fec60c8..aa7178c7 100644 --- a/website/src/content/docs/reference/changelog.mdx +++ b/website/src/content/docs/reference/changelog.mdx @@ -5,6 +5,97 @@ description: "RAMP protocol changelog" ## Unreleased +**`WellKnownManifest.endpoint` states its host binding (no wire change; conformance-affecting).** +The field said only "Exchange-only. ExchangeService endpoint URL", so nothing told an Exchange +operator that the address it advertises must stay on its own domain. It now does: the endpoint +MUST be on the host AND PORT that SERVE the manifest — not the self-asserted `domain` member +inside it — or on a subdomain of that host on that port, and MUST NOT carry userinfo. The manifest +is only as trustworthy as the host that served it, so an endpoint naming an unrelated host would +let whoever answers for the manifest redirect a signed call to a party the offer's signature never +covered — and a dial-time address guard has no objection to an unrelated PUBLIC host. Another port +is another service, which the party publishing the manifest need not control. The host match is on +a full dot-delimited label boundary, so `evil-a.com` is not a subdomain of `a.com`. A port equal to +the scheme's default and an omitted port are the SAME port, so `https://x`, `https://x:443` and `x` +all match; the scheme itself is not compared, and the default-port folding is scheme-relative so +that it cannot become a scheme check by accident. + +**This is the first entry in this changelog that changes what conforms without changing the +wire.** The classifier is deliberately not `(breaking)`: this change moves no field, message, or +encoding, and `buf breaking` reports nothing — while the bare `(breaking)` entries below all mark +a descriptor delta, and the one qualified use ("breaking for the generated clients") names the +audience it breaks. What this change does instead is narrow what a conformant manifest may say. + +**Two shapes that are conformant today will be refused after this.** The first is an Exchange +serving its API from a separate DOMAIN — a CDN, a hosting provider. The second is an Exchange on +a separate PORT: a single-domain deployment serving `/.well-known/ramp.json` on its default port +and advertising `"endpoint": "https://exchange.example:8443/v1"` is refused, as is the mirror +image (a portless endpoint under a manifest served on `:8443`) and a subdomain reached across +ports. A single domain is therefore no longer sufficient on its own — the authority must match on +both halves. + +Remedies, by shape. For a separate domain, front the API under a subdomain of the domain serving +the `ramp.json`. For a separate port, either move the API onto the port the manifest is served +from, or serve the manifest from the API's own authority — `https://exchange.example:8443/.well-known/ramp.json` +alongside `https://exchange.example:8443/v1`. Writing a scheme's default port out in full is NOT +a mismatch and needs no change. + +Both are refused as `ErrEndpointRefused`, which classifies as a FINAL verdict rather than a +transport failure — so a client will not retry its way out of a misconfiguration, and the symptom +is a usage report that never lands rather than one that is slow. + +Enforcement moved with the rule: it now runs in the SDK's shared endpoint resolver rather than +in one client, so every consumer of that resolver inherits it without changing a line. Two +consequences for anyone re-pinning. Resolution can now fail with a new `ErrEndpointRefused` +sentinel, which is a VERDICT — the Exchange answered and the answer is unusable — and a +classifier that branches only on the older `ErrNoEndpoint` will drop it into its +transport-failure bucket and retry something that will never succeed; add the new sentinel +alongside. And a Broker that resolves endpoints through this package inherits the rule for the +paths that use it. `gen/` and the website mirror are regenerated; proto comments only. + +**Go SDK: the delivery fetch correlates, and the offer-key cache is bounded (additive, no wire +change).** `resolvers.ContentFetchOptions` gained a `RequestID` hook, and `connect.NewClient` +feeds it the same mint the RPC legs read — so `WithRequestIDFunc` now reaches all three legs and +a delivery GET carries `X-Request-ID`. It did not before, and could not: the RPC legs correlate +through a Connect interceptor, which a plain GET never traverses, and there was no seam to add +one. **This changes what arrives at a delivery edge.** An edge that mints its own id when the +header is absent will now see the caller's instead, which is the point — a refused delivery used +to produce two log records under two ids with nothing joining them, on the one leg where +delivery failures are diagnosed. A fetcher built directly with no `RequestID` still sends no +header: this tier mints nothing of its own. + +`resolvers.CachedOfferKeyResolver`'s per-domain cache now evicts least-recently-used at a fixed +cap, like the endpoint cache and the per-origin client pool. Its key is a domain off +`Offer.exchange`, so which entries appear is driven by incoming offers, and an entry's expiry is +a freshness check rather than a removal — a stale entry held its slot indefinitely. Reaching it +needed a resolvable host serving a valid directory per domain, so the case was narrow rather +than open, but two sibling structures over the same key space were already bounded and this one +was not. + +**Go SDK: the Connect client covers the agent verb set, and its signing knobs are reachable +(additive, no wire change).** `connect.Client` gained `ReportUsage`, `Dispute` and `Fetch`, and +`connect.NewBrokerClient` gained `Resolve` — the client previously exposed `Discover` and +`Execute` alone, so a caller needing any of the rest had to assemble its own from +`rampv1connect` plus `core.NewSigningTransport`, which is the duplication the SDK exists to +remove. `Resolve` returns the same fail-closed `{verified, rejected}` split `Discover` does, +through the same `core.Verifier`; `Fetch` performs proof-of-possession on an agent-bound URL and +dials only through the SSRF-guarded client. + +Five client options join them, each because a value the tier below already accepted had no way +in: `WithSignWindow` (the RFC 9421 request freshness window — pair it with +`core.MonotonicWindow` when the peer screens replays on `(key id, signature)`, since one-second +timestamp resolution makes two identical requests inside a second sign to the same bytes), +`WithSignatureAgent` (the WBA directory origin the client signs as), `WithProofWindow`, +`WithContentTimeout` and `WithMaxContentBytes`. + +`WithSignatureAgent` is worth reading twice if you verify signatures. `signature-agent` is one +of the five REQUIRED covered components, so the header is signed whether or not a value was +supplied — a client that does not set it signs an EMPTY one. A peer that resolves the caller's +key by fetching the WBA directory at that origin then has nothing to resolve and refuses the +call at verification, which surfaces as a 401 from an otherwise healthy Exchange rather than as +anything the routing checks would catch. The value is stamped set-if-absent, so a relay +forwarding an originating agent's request does not overwrite the value that agent's own +signature covers. See `docs/sdk-parity-matrix.md` for the per-language surface. + **SDK (all 3 languages): the registration-failure builder can carry the field errors (additive, no wire change).** `helpers.RegistrationFailureDetail` (Go), `registration_failure_detail` (Python) and `registrationFailureDetail` (TS) now accept the