Problem
The faucet probe at src/probes/faucet.mjs:51,82 currently:
- Uses an invalid nametag (
infra-probe-do-not-mint-zk7q3xa9p2v) to avoid consuming faucet quota.
- Probes ONLY
coin: 'unicity-usd'.
This combination is structurally blind to per-coin faucet failures.
Concrete failure case (sphere-sdk soak, 2026-05-25 09:46Z)
During a manual-test soak run, the testnet faucet was simultaneously:
- Healthy per
unicity-infra-probe (status: healthy, all 6 services green).
- Disk-full for UCT only: alice's faucet drop succeeded for all 7 tokens (
✓ unicity: 100), then minutes later bob's call returned:
✗ unicity: Failed - /app/data/tokens/token_9248020_bob-23020020_1779702464406.json (No space left on device)
✓ bitcoin: 1
✓ ethereum: 42
✓ solana: 1000
✓ tether: 1000
✓ usd-coin: 1000
✓ unicity-usd: 1000
UCT (unicity) had its own per-coin storage backend (/app/data/tokens/) that filled independently. The probe couldn't have caught this because:
- Its invalid nametag returns HTTP 400 BEFORE reaching the disk-write code path.
- It only sweeps
unicity-usd, never unicity.
The sphere-sdk soak's blast radius: §C.2 (bob pays UCT invoice) failed at Insufficient balance, aborting the iteration. The 5-iteration soak couldn't reach acceptance for an SDK PR (#264 in sphere-sdk).
Suggested fix
Extend the probe's faucet check to iterate all 7 coins served by the faucet:
unicity (UCT — the failed coin)
unicity-usd
bitcoin
ethereum
solana
tether
usd-coin
For each coin, attempt the invalid-nametag probe. If the faucet returns 5xx (or any class of error other than the expected 400-with-Nametag not found body) for ONE specific coin, surface that as a per-coin failure in the probe output.
Caveats
- This still does NOT detect ENOSPC pre-failure — the invalid-nametag short-circuits before the disk-write code path on the faucet side. To catch ENOSPC, the faucet itself must expose disk/inode metrics (see related issue: ops ticket for
/health enhancement on the faucet deployment).
- This DOES narrow the blast radius: if the faucet starts rejecting one coin for reasons other than the nametag (e.g., 5xx on UCT specifically, malformed coin config, etc.), the probe will surface it.
Acceptance
Context
- Related sphere-sdk soak artifacts (preserved):
peer1-bob-faucet.log showing the disk-full + healthy-other-coins pattern.
- Related sphere-sdk PR: #265 (issue #264 — the soak that hit this).
Problem
The faucet probe at
src/probes/faucet.mjs:51,82currently:infra-probe-do-not-mint-zk7q3xa9p2v) to avoid consuming faucet quota.coin: 'unicity-usd'.This combination is structurally blind to per-coin faucet failures.
Concrete failure case (sphere-sdk soak, 2026-05-25 09:46Z)
During a manual-test soak run, the testnet faucet was simultaneously:
unicity-infra-probe(status: healthy, all 6 services green).✓ unicity: 100), then minutes later bob's call returned:UCT (
unicity) had its own per-coin storage backend (/app/data/tokens/) that filled independently. The probe couldn't have caught this because:unicity-usd, neverunicity.The sphere-sdk soak's blast radius: §C.2 (bob pays UCT invoice) failed at
Insufficient balance, aborting the iteration. The 5-iteration soak couldn't reach acceptance for an SDK PR (#264 in sphere-sdk).Suggested fix
Extend the probe's faucet check to iterate all 7 coins served by the faucet:
unicity(UCT — the failed coin)unicity-usdbitcoinethereumsolanatetherusd-coinFor each coin, attempt the invalid-nametag probe. If the faucet returns 5xx (or any class of error other than the expected 400-with-
Nametag not foundbody) for ONE specific coin, surface that as a per-coin failure in the probe output.Caveats
/healthenhancement on the faucet deployment).Acceptance
src/probes/faucet.mjsiterates the full coin list (or a configurable subset) and reports per-coin status./healthdisk metrics).tests/smoke.test.mjs) extended to cover the multi-coin sweep.Context
peer1-bob-faucet.logshowing the disk-full + healthy-other-coins pattern.