Skip to content

Faucet probe: extend liveness check to all 7 coins (currently sweeps only unicity-usd) #4

@vrogojin

Description

@vrogojin

Problem

The faucet probe at src/probes/faucet.mjs:51,82 currently:

  1. Uses an invalid nametag (infra-probe-do-not-mint-zk7q3xa9p2v) to avoid consuming faucet quota.
  2. Probes ONLY coin: 'unicity-usd'.

This combination is structurally blind to per-coin faucet failures.

Concrete failure case (sphere-sdk soak, 2026-05-25 09:46Z)

During a manual-test soak run, the testnet faucet was simultaneously:

  • Healthy per unicity-infra-probe (status: healthy, all 6 services green).
  • Disk-full for UCT only: alice's faucet drop succeeded for all 7 tokens (✓ unicity: 100), then minutes later bob's call returned:
✗ unicity: Failed - /app/data/tokens/token_9248020_bob-23020020_1779702464406.json (No space left on device)
✓ bitcoin: 1
✓ ethereum: 42
✓ solana: 1000
✓ tether: 1000
✓ usd-coin: 1000
✓ unicity-usd: 1000

UCT (unicity) had its own per-coin storage backend (/app/data/tokens/) that filled independently. The probe couldn't have caught this because:

  • Its invalid nametag returns HTTP 400 BEFORE reaching the disk-write code path.
  • It only sweeps unicity-usd, never unicity.

The sphere-sdk soak's blast radius: §C.2 (bob pays UCT invoice) failed at Insufficient balance, aborting the iteration. The 5-iteration soak couldn't reach acceptance for an SDK PR (#264 in sphere-sdk).

Suggested fix

Extend the probe's faucet check to iterate all 7 coins served by the faucet:

  • unicity (UCT — the failed coin)
  • unicity-usd
  • bitcoin
  • ethereum
  • solana
  • tether
  • usd-coin

For each coin, attempt the invalid-nametag probe. If the faucet returns 5xx (or any class of error other than the expected 400-with-Nametag not found body) for ONE specific coin, surface that as a per-coin failure in the probe output.

Caveats

  • This still does NOT detect ENOSPC pre-failure — the invalid-nametag short-circuits before the disk-write code path on the faucet side. To catch ENOSPC, the faucet itself must expose disk/inode metrics (see related issue: ops ticket for /health enhancement on the faucet deployment).
  • This DOES narrow the blast radius: if the faucet starts rejecting one coin for reasons other than the nametag (e.g., 5xx on UCT specifically, malformed coin config, etc.), the probe will surface it.

Acceptance

  • src/probes/faucet.mjs iterates the full coin list (or a configurable subset) and reports per-coin status.
  • Per-coin failures surface in the probe's output (not just an aggregate green/red).
  • README documents the per-coin storage blindness as a known limitation (links to the ops ticket for /health disk metrics).
  • Existing single-coin smoke test (tests/smoke.test.mjs) extended to cover the multi-coin sweep.

Context

  • Related sphere-sdk soak artifacts (preserved): peer1-bob-faucet.log showing the disk-full + healthy-other-coins pattern.
  • Related sphere-sdk PR: #265 (issue #264 — the soak that hit this).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions