engine: node eviction & role downgrade now take and converge fleet-wide#105
Conversation
Two governance bugs made removing or demoting a fleet owner/manager appear to do nothing, and let removed devices linger as controllable on other owners. 1. Withdrawing a role didn't "take" on the device that authored it. `try_ratify`'s local roster mirror open-codes a per-variant subset — RoleGrant, Evict, KindChange, TopologyChange — and silently skipped RoleRevoke, so the cached roster authority tag kept rendering the old role even though `gov.roles` had dropped it. The gossip-adoption path (`mirror_roles_to_roster`) already reprojects the whole map, so only the acting device was wrong. Mirror RoleRevoke too: reset the target's roster tag to `member` (it stays in the roster — a withdraw demotes, it doesn't remove). 2. Evicting a promoted owner/manager resurrected it on gossip-adopting peers. A device admitted as a plain member and later promoted still carries its original member-tier admit in the member log. A governance-tier evict only extended the owner log, so any peer that re-derived membership from the signed logs — a co-owner adopting via gossip after being offline for the kick — folded that stale admit back in and re-added the evicted device as a plain member: still rostered, still authorised to control the fleet, and invisible-as-gone to every such owner. Record the evict in the union-merged member log too so it tombstones the admit; the removal then converges network-wide and survives concurrent authors, exactly like a plain-member evict. Stamp the evict strictly past the target's newest member-log entry so the tombstone wins the last-writer-wins tie instead of the admit. Adds regression tests for both: a single-engine projection test that drives the exact functions the gossip-adoption path is built on (so it fails deterministically without the fix, unlike an online peer that ratifies incrementally), and a withdraw test asserting the authoring device's roster tag drops to member. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
The test seeded two peer ids that must sort lex-above the engine's own (random, ephemeral) identity so it plays the offerer role. `evicted` used "z"*60 — always above any 52-char base32 identity — but `live` used "y"*60, which a ~3% of ephemeral identities (those starting with 'z') sort *above*, tripping the precondition assert. That's an environment-random flake independent of what the test exercises (drop_peer's evicted-reconnect guard). Make `live` all-'z' too (length 61 to stay distinct from `evicted`), so both ids clear every possible identity deterministically. Confirmed: 2/60 failures before, 0/80 after. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
|
The Reproduced it at 2/60 locally, then made Generated by Claude Code |
The evict tombstone (previous commit) makes a removal converge across every peer's *signed* roster. But an owner also keeps a local, un-synced record of the devices it claimed (AllMyStuff's `fleet_members`), and that list is only pruned by the owner who authored the kick. When a *different* owner evicts a device, the claiming owner's local list still carries it — and its background re-assertion loop would re-sign + re-approve it, resurrecting an evicted device fleet-wide. Give clients the signal they need to keep that local bookkeeping honest: `GovernanceState` now returns an `evicted` array alongside `state` — the `member_log_removed` projection, i.e. the devices the signed logs have removed (evicted / member-tier revoked). A client reconciles its local claimed-list against it and drops anyone the fleet has evicted, so the re-assertion loop stops resurrecting them. Additive and back-compatible: an older client just ignores the field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
Problem
Reported from AllMyStuff (which reads its fleet roles and control-authorization straight from this daemon's signed roster):
Root cause
Two bugs in the closed-network governance engine (
engine/governance.rs), both around how a ratified transition is mirrored into the on-disk roster that everything downstream reads.1.
RoleRevokewas never mirrored locally.try_ratifyopen-codes a per-variant roster mirror —RoleGrant,Evict,KindChange,TopologyChange— and skippedRoleRevokeentirely.gov.rolesdropped the role, but the cached roster authority tag kept rendering the old one, so on the very device that authored the withdrawal the demotion never showed. The gossip-adoption path (mirror_roles_to_roster) already reprojects the whole map, so only the acting device was wrong — which is exactly "I withdrew the role and nothing happened."2. Evicting a promoted owner/manager resurrected it on gossip-adopting peers. A device admitted as a plain member and later promoted still carries its original member-tier admit in the union-merged member log. A governance-tier evict only extended the owner log, so any peer that re-derived membership from the signed logs — a co-owner adopting via gossip after being offline during the kick — folded that stale admit back in and re-added the evicted device as a plain member: still rostered, still authorised to control the fleet, and invisible-as-gone to every such owner. (An online peer ratifies the evict incrementally via
apply_transitionand never hits the resurrecting re-projection, which is why this only bit some owners.)Fix
RoleRevoke: reset the target's cached roster tag tomember. The device stays in the roster (a withdraw demotes, it doesn't remove) but its authority drops immediately, matching the gossip-adoption path.member_tier_timestampnow covers evicts at either tier) so the tombstone wins the last-writer-wins tie instead of the same-second admit it means to remove.No public API or wire-format change; AllMyStuff picks this up automatically once the bundled daemon is rebuilt (it reads roles/authorization from
RosterList).Tests
withdrawing_a_role_updates_the_local_roster_tag— asserts the authoring device's roster tag drops tomemberafter a withdraw. Fails without fix Scaffold: pure-Rust mesh workspace ported from MyOwnLLM patterns #1.evicting_a_promoted_member_tombstones_its_member_admit— a single-engine projection test that drives the exact functions (verify_log/verify_member_log/member_log_removed) the gossip-adoption path is built on, so it fails deterministically without fix Engine + transport: full mesh stack, end-to-end tested #2 (an online two-engine test passes even unpatched, because online peers ratify incrementally).Full
closed_network_governance+closed_network_roster_guardsuites and the 271myownmesh-coreunit tests pass;cargo fmt/clippyclean; daemon crate builds.🤖 Generated with Claude Code
https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
Generated by Claude Code