Skip to content

engine: node eviction & role downgrade now take and converge fleet-wide#105

Merged
mrjeeves merged 3 commits into
mainfrom
claude/node-eviction-role-downgrade-k54mhp
Jul 23, 2026
Merged

engine: node eviction & role downgrade now take and converge fleet-wide#105
mrjeeves merged 3 commits into
mainfrom
claude/node-eviction-role-downgrade-k54mhp

Conversation

@mrjeeves

Copy link
Copy Markdown
Owner

Problem

Reported from AllMyStuff (which reads its fleet roles and control-authorization straight from this daemon's signed roster):

  • Evicting a node that is/was an owner appeared to do nothing.
  • Demoting an owner → manager or → member "didn't take" — the role tag never changed.
  • After a node left/was removed, other fleet members still showed it as controllable, and other owners couldn't see it gone — it "remained" in the fleet.

Root cause

Two bugs in the closed-network governance engine (engine/governance.rs), both around how a ratified transition is mirrored into the on-disk roster that everything downstream reads.

1. RoleRevoke was never mirrored locally. try_ratify open-codes a per-variant roster mirror — RoleGrant, Evict, KindChange, TopologyChange — and skipped RoleRevoke entirely. gov.roles dropped the role, but the cached roster authority tag kept rendering the old one, so on the very device that authored the withdrawal the demotion never showed. The gossip-adoption path (mirror_roles_to_roster) already reprojects the whole map, so only the acting device was wrong — which is exactly "I withdrew the role and nothing happened."

2. Evicting a promoted owner/manager resurrected it on gossip-adopting peers. A device admitted as a plain member and later promoted still carries its original member-tier admit in the union-merged member log. A governance-tier evict only extended the owner log, so any peer that re-derived membership from the signed logs — a co-owner adopting via gossip after being offline during the kick — folded that stale admit back in and re-added the evicted device as a plain member: still rostered, still authorised to control the fleet, and invisible-as-gone to every such owner. (An online peer ratifies the evict incrementally via apply_transition and never hits the resurrecting re-projection, which is why this only bit some owners.)

Fix

  • Mirror RoleRevoke: reset the target's cached roster tag to member. The device stays in the roster (a withdraw demotes, it doesn't remove) but its authority drops immediately, matching the gossip-adoption path.
  • Tombstone evicts in the member log: when a governance-tier evict ratifies, record it in the union-merged member log too, so it supersedes the target's stale member-tier admit. The removal then converges network-wide and survives concurrent authors — the same convergence a plain-member evict already gets.
  • Stamp the evict strictly past the target's newest member-log entry (member_tier_timestamp now covers evicts at either tier) so the tombstone wins the last-writer-wins tie instead of the same-second admit it means to remove.

No public API or wire-format change; AllMyStuff picks this up automatically once the bundled daemon is rebuilt (it reads roles/authorization from RosterList).

Tests

Full closed_network_governance + closed_network_roster_guard suites and the 271 myownmesh-core unit tests pass; cargo fmt / clippy clean; daemon crate builds.

🤖 Generated with Claude Code

https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L


Generated by Claude Code

claude added 2 commits July 23, 2026 06:02
Two governance bugs made removing or demoting a fleet owner/manager appear
to do nothing, and let removed devices linger as controllable on other
owners.

1. Withdrawing a role didn't "take" on the device that authored it.
   `try_ratify`'s local roster mirror open-codes a per-variant subset —
   RoleGrant, Evict, KindChange, TopologyChange — and silently skipped
   RoleRevoke, so the cached roster authority tag kept rendering the old
   role even though `gov.roles` had dropped it. The gossip-adoption path
   (`mirror_roles_to_roster`) already reprojects the whole map, so only the
   acting device was wrong. Mirror RoleRevoke too: reset the target's roster
   tag to `member` (it stays in the roster — a withdraw demotes, it doesn't
   remove).

2. Evicting a promoted owner/manager resurrected it on gossip-adopting
   peers. A device admitted as a plain member and later promoted still
   carries its original member-tier admit in the member log. A
   governance-tier evict only extended the owner log, so any peer that
   re-derived membership from the signed logs — a co-owner adopting via
   gossip after being offline for the kick — folded that stale admit back
   in and re-added the evicted device as a plain member: still rostered,
   still authorised to control the fleet, and invisible-as-gone to every
   such owner. Record the evict in the union-merged member log too so it
   tombstones the admit; the removal then converges network-wide and
   survives concurrent authors, exactly like a plain-member evict. Stamp
   the evict strictly past the target's newest member-log entry so the
   tombstone wins the last-writer-wins tie instead of the admit.

Adds regression tests for both: a single-engine projection test that drives
the exact functions the gossip-adoption path is built on (so it fails
deterministically without the fix, unlike an online peer that ratifies
incrementally), and a withdraw test asserting the authoring device's roster
tag drops to member.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
The test seeded two peer ids that must sort lex-above the engine's own
(random, ephemeral) identity so it plays the offerer role. `evicted` used
"z"*60 — always above any 52-char base32 identity — but `live` used "y"*60,
which a ~3% of ephemeral identities (those starting with 'z') sort *above*,
tripping the precondition assert. That's an environment-random flake
independent of what the test exercises (drop_peer's evicted-reconnect
guard). Make `live` all-'z' too (length 61 to stay distinct from `evicted`),
so both ids clear every possible identity deterministically. Confirmed:
2/60 failures before, 0/80 after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L

Copy link
Copy Markdown
Owner Author

The check / linux-x86_64 failure on the first run was a pre-existing flake, not a regression from this change: engine::tests::evicted_peer_drop_never_rearms_reconnect asserts its two hardcoded peer ids sort lex-above the engine's own random ephemeral identity. evicted used "z"*60 (always above any 52-char base32 id) but live used "y"*60, which the ~3% of identities starting with z sort above — tripping the precondition assert before the test reaches the drop-peer logic it exercises. My governance change doesn't touch that path.

Reproduced it at 2/60 locally, then made live all-z too (length 61 to stay distinct) so both ids clear every possible identity deterministically — 0/80 after. Pushed as a separate commit (936830d); CI is re-running.


Generated by Claude Code

The evict tombstone (previous commit) makes a removal converge across every
peer's *signed* roster. But an owner also keeps a local, un-synced record of
the devices it claimed (AllMyStuff's `fleet_members`), and that list is only
pruned by the owner who authored the kick. When a *different* owner evicts a
device, the claiming owner's local list still carries it — and its background
re-assertion loop would re-sign + re-approve it, resurrecting an evicted
device fleet-wide.

Give clients the signal they need to keep that local bookkeeping honest:
`GovernanceState` now returns an `evicted` array alongside `state` — the
`member_log_removed` projection, i.e. the devices the signed logs have
removed (evicted / member-tier revoked). A client reconciles its local
claimed-list against it and drops anyone the fleet has evicted, so the
re-assertion loop stops resurrecting them. Additive and back-compatible: an
older client just ignores the field.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018x1RV6ppUdMDTHxScVo26L
@mrjeeves
mrjeeves merged commit 7aaabbb into main Jul 23, 2026
10 checks passed
@mrjeeves
mrjeeves deleted the claude/node-eviction-role-downgrade-k54mhp branch July 23, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants