feat(broker): hermes-broker — route the hermes.proto surface across instances - #139
Merged
Conversation
…ss instances A stateless gRPC broker that fronts many hermes-server backends behind one address. Serves the exact hermes.proto SearchService/IndexService (clients switch by re-pointing endpoints) plus a broker-only hermes.broker admin proto and grpc.health.v1. - Discovery: kubernetes pod watch (shard id + role pod labels) or static --backend list; index->backend routing learned via ListIndexes polling with a Suspect/Evicted health machine (grace, two-probe recovery). - Routing: per-index pass-through; writes to the shard master only; CreateIndex placed by glob rules (dated families follow their shard); ambiguous multi-shard names read deterministically and refuse writes until a rule pins them. - Contract: byte-faithful write responses, grpc-timeout propagated with no broker-imposed deadlines, admission mirrors --max-concurrent-searches with the server's exact RESOURCE_EXHAUSTED message, ListIndexes served from cache. - Tests: unit (topology/placement/health/deadline math), integration (real broker binary vs scripted mock backends: pass-through equality, stream re-routing, eviction/recovery, deadline headers), env-gated e2e against real hermes-server subprocesses (CI runs it after building the server). - CI publishes ghcr.io/spacefrontiers/hermes/hermes-broker alongside the server image; both Dockerfiles now copy the new workspace member.
grpcurl-driven verification (the deployment runbook's parity checks) needs reflection: neither the broker nor hermes-server previously answered schema-less tooling. Registers both the hermes.proto and hermes-broker.proto descriptor sets.
ppodolsky
force-pushed
the
hermes-broker
branch
from
August 8, 2026 05:28
fab3da4 to
17e5727
Compare
kube's rustls-tls stack panics at first TLS use unless a process-level CryptoProvider is installed (rustls 0.23); install ring explicitly. A discovery-task failure (error or panic) now drains and exits the server via the internal shutdown channel instead of leaving a zombie broker answering NOT_SERVING forever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A stateless gRPC broker fronting many hermes-server backends behind one address, so consumers scale to multiple hermes nodes by re-pointing one endpoint.
What
hermes.protoSearchService/IndexService(drop-in for all clients) plus a broker-onlyhermes.broker.BrokerServiceadmin proto,grpc.health.v1, and server reflection.hermes.spacefrontiers.org/shard-idpod label, replica role from…/role) or a static--backendlist; learns index→backend routing by pollingListIndexeswith a Suspect/Evicted health machine (grace period, two-probe recovery).docs/broker.md).CreateIndexplaced by glob rules (--placement "documents*=0"), so dated index families stay on their shard; the same rules pin reads/writes when an index name transiently exists on two shards during a host-to-host migration.DocumentErrors untouched), incominggrpc-timeoutpropagated with no broker-imposed deadlines (24h admin merges, untimed index channels), admission mirrors--max-concurrent-searcheswith the server's exactRESOURCE_EXHAUSTEDmessage,ListIndexesserved from cached topology.Tests
IndexDocuments, deadline header presence/absence at the backend, eviction + two-probe recovery, placement.Packaging
publish.ymlbuilds and pushesghcr.io/spacefrontiers/hermes/hermes-brokeralongside the server image; both existing Dockerfiles gain the new workspace member's COPY line.docs/broker.md; metrics documented indocs/metrics.md.