feat(health): probe and serve axon + prometheus endpoint reachability — thousands of endpoints we publish and have never checked

[JSONbored]

Parent: #11066. Producer: the prober container class.

The largest untouched observable surface we have

We already serve ip:port for every neuron through get_subnet_serving and
get_chain_prometheus. None of it has ever been probed. The reason is precise and not a
priority call: src/health-probe-core.ts:113 allowlists exactly http, https, ws, wss, and
axons listen on arbitrary ports.

So the registry publishes an address and can say nothing about whether anything is listening at it —
which, for a registry whose product is "how to call this subnet", is the gap closest to the core
claim.

Scope

• A new operational surface kind for axon/prometheus endpoints, added to the probe-eligible set
• TCP connect result and connect latency per endpoint
• Where the endpoint speaks a known protocol, a protocol-level check rather than a bare connect
• Aggregate per subnet: what fraction of registered UIDs are actually reachable

The number this makes serveable

"This subnet has 256 registered UIDs and 31 reachable axons" is a fact about a subnet that nothing
in the ecosystem publishes, and it is directly comparable across subnets. Pair it with the existing
metagraph data and it prices participation honestly.

Care required

These are third-party hosts and we are opening thousands of connections. Cadence, concurrency and
per-host backoff need stating up front, and the existing SSRF guards (src/health-probe-core.ts:56-160)
must apply to raw TCP targets too — an ip:port from chain data is caller-influenced input and a
private-range address is a probe we must refuse.

Acceptance

• Axon reachability probed and served for at least one subnet end to end
• SSRF guards apply to TCP targets; private, loopback and metadata ranges refused
• Per-subnet reachable-fraction served and comparable across subnets
• A documented per-host rate discipline
• Unreachable and never-probed are distinguishable, per absent is null

[linear-code]

JSO-8210

[JSONbored]

Sampled 2026-08-13 — 889 endpoints, 63.3% answer, and the spread between subnets is the story

One TCP connect per endpoint, 5s timeout, nothing sent, from one location. Addresses come from
/api/v1/subnets/{n}/metagraph -> neurons[].axon.

First, a correction to this issue's framing

I wrote "thousands of ip:port pairs." In the served data it is thinner and wildly uneven —
only about 10% of neurons carry a non-null axon in some subnets and 98% in others:

netuid	neurons	publishing an axon
32	256	251
5	256	247
56	256	185
51	256	140
8	256	29
19	256	15
4	256	10
64	256	6
1	256	4
77	256	2
91 / 103	256 / 4	0

Also: 888 of 889 are direct IPs, not CDN-fronted. I had assumed a lot of Cloudflare; there is
exactly one. These are boxes people run.

Reachability

open                    563   63.3%
TimeoutError            194   21.8%
ConnectionRefusedError  128   14.4%
OSError                   4    0.4%
connect latency: min 37ms  median 101ms  max 1210ms

The spread, which is the actual product

netuid	published	reachable
32	251	250	99.6%
5	247	245	99.2%
51	140	47	34%
8	29	7	24%
4	10	4	40%
56	185	8	4.3%
64	6	1	17%
1	4	1	25%
19	15	0	0%
77	2	0	0%

SN32 and SN5 are at ~99%. SN56 publishes 185 endpoints and 8 answer. SN19 publishes 15 and none
do. Nothing in this ecosystem publishes that comparison, and it is computed entirely from data we
already serve.

The caveat, which has to lead any published version

A refused connection is not proof of a broken miner. Several legitimate configurations produce
exactly the results above:

• an axon firewalled to validator source IPs only — correct operation, refuses me
• geo-filtering; this ran from one location
• rate limiting or connection throttling

The two failure modes are also not equivalent, and the split matters: ConnectionRefused (14.4%)
means the host is up and the port is closed; Timeout (21.8%) means unreachable or silently
filtered. Only the first is strong evidence of anything.

So the honest metric is "accepts a connection from an arbitrary public client", not "works."
That is still a real, comparable, useful fact — a subnet at 99% and a subnet at 4% are genuinely
different — but the field name and the docs have to say what was measured. Publishing this as
"reachable" or, worse, "healthy" would be a misrepresentation this registry should not make.

Scope changes this suggests

• Name the metric for what it is: connection-accepted-from-public-client, with the refused vs
  timeout split preserved as separate fields, never collapsed
• Probe from more than one region before publishing — this pairs with feat(health): serve regional availability — one surface, several vantage points, and a stated rollup rule #11070, and the
  geo-filtering hypothesis is directly testable that way
• Publish the method alongside the number, and give subnet owners a way to say "validator-only
  by design" so the metric does not punish correct configuration
• Track publication rate too (0 to 251 of 256) — that is interesting on its own and costs
  nothing
• Raw results: axon_probe_results.json