Replies: 4 comments 3 replies
|
Follow-up, and it belongs in the post rather than in a footnote: this design has a sharp edge, and I walked straight into it about two hours after writing the above. The post says two things that are individually right and jointly dangerous: that a probe should exercise the behavior rather than assert on a string, and that the verifier runs as the last step of the upgrade command. Put a probe on the upgrade command itself and you get this:
Neither tool had a recursion guard, because each was written believing it was the one doing the verifying. It ran to roughly 700 processes in four minutes, each one fetching release files over the network. It was noticed because the bandwidth was noticed, not because anything reported an error — every individual process was behaving exactly as designed. Two things worth passing on to anyone implementing this pattern: 1. A guard belongs at the spawn site, in the child's environment — not exported into the process. The parent marks the child ( 2. "Cannot answer from here" is not the same verdict as "tried and failed." Nested, the probe now returns a distinct The negative pole for a fork bomb turns out to be pleasingly simple: the command has to terminate. Before the guard, Which is another instance of the point the post was already making — the machinery found the bug in the machinery — except this time the bug was mine and it was one day old. A registry of local changes is worth having, but a probe that invokes the thing that invokes probes needs a guard designed in from the first line, not added after someone notices the fans. The guard, in code, since the description above is only useful if it is copyable. Parent side, at the spawn: // The marker travels in the CHILD's env, as part of the spawn call — never exported into
// the process. An exported value would outlive the command and silently disable the probe
// in legitimate standalone runs, which is the quiet way to lose a verification while the
// report stays green.
const vlp = spawnSync(["bun", join(ROOT, "TOOLS/VerifyLocalPatches.ts")], {
cwd: ROOT,
env: { ...process.env, INSIDE_UPGRADE_CHECK: "1" },
});Child side, in the one probe that would recurse: /** "Cannot answer from here, and trying would be dangerous" — deliberately distinct from
* UNKNOWN, which means "tried and failed". Surfaces as SKIPPED with the reason printed. */
class NestedSkip extends Error {}
"upgrade-check-nonmutating": () => {
if (process.env.INSIDE_UPGRADE_CHECK === "1")
throw new NestedSkip("inside an upgrade --check: probing it would spawn it again — run this entry standalone");
// …fingerprint the files an apply would rewrite, run the command, compare
}and in the runner, so the skip is never silent: } catch (e) {
if (e instanceof NestedSkip) { status = "SKIPPED"; reason = `skipped to avoid recursion — ${e.message}`; }
else { status = "UNKNOWN"; reason = `probe could not run: ${e.message}`; }
}One implementation note that cost an hour to learn: fingerprint the watched files by content hash, not mtime. Entering the command re-runs the settings merge, which rewrites a file byte-for-byte identical — with mtime the probe reported a mutation that never happened, and a red that means nothing teaches you to ignore reds. |
|
Prior art I should have found before writing this, and an answer that already exists. #650 — "feat: Local patch tracking and update-safe SYSTEM file management" (@jlacour-git, closed 2026-06-21) proposed substantially what this post proposes, a month earlier and independently: a registry of local patches to SYSTEM files, staleness detection for generated files, and a reconciliation procedure at update time. That thread also carries two things worth reading before anyone re-litigates the design:
So this post is not a new idea. What I can add is measurement from the other side of a 6.x → 7.x migration, and two failure modes that only show up once probes exist — the fidelity gate that was silently checking 400 of 508 tokens, and the housekeeping tool anchored on the wrong file extension reporting "0 files" while four sat in the directory. The maintainer's answer in #650 also deserves to be quoted here rather than talked past, because it changes what the right shape is:
That is a real answer, and it dissolves part of the problem: if integration is performed per machine rather than shipped as one directory, there is less overwriting to survive in the first place. What I do not think it dissolves is the part this post is actually about. Per-machine integration decides how your system gets assembled; it does not tell you which of your local differences still need to exist. A user who patched around a defect in March still ends up carrying that patch in September unless something asks the question "does this still work — and does it work without me?" every time the upstream code moves. If anything, an installer that wires things in per machine makes each install's delta more individual, not less, and therefore harder to reason about without something written down. So, treating this as the "if it still bites you, file a fresh one" case @danielmiessler invited: it still bites, here is what it cost, and here is what we ran. Fully expecting the answer to be that the agentic installer covers enough of it — that is a fine outcome, and #650's lighter approach would then be the sensible remainder for anyone with a small delta. One thing that would help regardless of which direction wins: a closed issue is not evidence the code changed. Three of the defects we hit this month trace to issues marked closed whose fix never reached the file — and one of them we nearly dropped a live local guard over, precisely because the issue read CLOSED. Whatever replaces directory-cloning, "did this actually land" needs to be answerable from the code, not from the tracker. |
|
The overlay tool that landed this week takes the report-don't-revert half of this design (timestamped backups plus explicit update lists). The behavioral-probes half is the interesting remainder — if you still want it after the next release ships, let's spec it against the real overlay behavior. |
|
Thanks, and thanks for the work generally. Report-don't-revert is the right default. Agreed on waiting for the release. I'll spec the probe half against what the overlay actually does rather than against my guess at it, so point me at the file when it ships. Where it stands here, so you know what you'd be reviewing. The registry is at 48 entries now, up from 29 in July, and the runner is around 1,250 lines. Today's run: The three NO-PROBE are entries I registered and never wrote a probe for. They come out red, which is the point. The contribution would be the generic half: registry schema, verdict model, runner, and the self-test that fails the whole run when an entry names a probe that doesn't exist. One suggestion for whenever we spec this, from having lived with it: it may be worth having the probe layer stop short of picking a winner. Each patch entry in mine carries a second, inverse probe asking "does this still work without my change?", run against an extracted release tree, so the answer comes from the shipped code rather than from an issue state. A yes means upstream has its own solution now, and the entry comes out SUPERSEDED with a non-zero exit rather than being retired. The run writes a comparison dossier instead: what mine does, what breaks without it, what upstream does now, the diff between the two implementations, and an empty answer line. Two solutions that both work don't rank themselves, so the choice seems better left with whoever runs the install. Happy to be talked out of it if the overlay already handles this differently. The seam between the two halves is the part I'd most want to get right. The overlay says which files would change; the probe run can say which behaviors stop working if they do. That's worth more than either list alone, but it only works if the update list is machine-readable output the probe run can take as input. Worth knowing if that format is still moving. |
|
@richardchen874-sys That's what the July post proposed, and the invariant is the part that took longest to get right. An invariant has to be written so it doesn't care who satisfies it. Not "is my marker present in this file", which is an assertion about a string and goes red as soon as upstream fixes the same thing a different way. The example in the post: for a change that makes stray screenshots git-ignored, ask Two things I'd add from running it since:
On the routing question from earlier: different problem. Upgrade survivability doesn't change with which provider serves the call. |
|
@richardchen874-sys Thanks — the questions are doing real work here. Pushing on the layer boundary is what separated routing from upgrade survivability, and asking about the interface before the implementation exists is how the two halves end up composable instead of two reports nobody joins. On the format itself: not mine to answer. The overlay is @danielmiessler's, and I asked the same thing at the end of my last comment. What I can do is state what the consuming side would need, so there's something concrete to measure the format against when it ships. Minimal contract, from the probe-runner side:
The two halves compose only through that list — the overlay says which files would change, the probe run says which behaviours stop working if they do. If the list is human-readable text only, the seam does not exist and each side stays a separate report. Current state here, for reference: 44 HOLDS · 0 SUPERSEDED · 0 REGRESSED · 2 UNKNOWN · 3 NO-PROBE. |
|
We've massively improved the ability to hold on to your own settings when doing upgrades in the last two releases. |
Uh oh!
There was an error while loading. Please reload this page.
The problem: every customized install accumulates an invisible delta
Anyone who runs LifeOS for a few months ends up with local changes to system files — a fix for a bug they hit before it was fixed upstream, a tool they wrote, a setting the shipped default got wrong for their machine. A release upgrade overwrites system files. So each of those changes has exactly two possible fates, and both are bad:
The missing piece is not a merge strategy. It is knowing what your delta is, and whether each piece of it still needs to exist.
What we built and have been running
Three parts, all small:
1. A delta registry — one JSON file listing every local difference. Each entry carries: what it does in plain language, what breaks without it, which files it touches, its upstream status, and the name of a probe.
2. A behavioral probe per entry — a function that answers "does this still work?" by exercising the behavior, deliberately independent of who made it work. Not "is my marker present", which is an assertion about a string; the probe asks the question the user actually cares about. Example, for a fix that makes stray screenshots git-ignored: ask
git check-ignoreabout a synthetic filename. That returns true whether our rule or an upstream rule does it.3. An inverse probe for fixes only — "does it still work WITHOUT our change?" run against an extracted release tree. A yes means upstream now handles it, and our patch has become dead weight.
The verifier runs as the last step of the upgrade command — inside the thing that could have reverted the fix — and emits four verdicts:
HOLDSREGRESSEDSUPERSEDEDNO-PROBEThe two design decisions that make it worth writing down
SUPERSEDEDis a question, not a verdict. The tool never retires anything on its own. It emits a comparison document — what ours does, what theirs does, where they differ — and a human decides. Two reasons, both learned the hard way:A missing probe is red. An entry naming a probe that does not exist reports
NO-PROBEand fails the run. A control that never executes is an assertion wearing a lab coat, and a registry where unverified entries render green is worse than no registry — it produces confidence without coverage. The verifier ships with an A/A self-test: a fixture registry naming a nonexistent probe must make the tool exit non-zero. If that test ever passes green, every other green on the page is worthless.What it caught
The probes were written to guard known changes. Two of them found defects nobody was looking for:
.pngwhile the capture tool had started writing.jpg. It reported "0 files to clean" while four files sat in the directory — a green that only meant I did not look there.Both are the same failure: a probe answering a weaker question than the one you thought you asked. That is the class of bug this machinery exists to surface, and it surfaced it in the machinery itself.
Would this be useful upstream?
We are running it locally over 29 entries (local fixes, our own tools, personalizations, generated files). It is roughly 500 lines: a registry schema, a probe runner, and the comparison output.
If there is interest we would be glad to contribute it — either as a general subsystem, or just the schema + verdict model so each install can write its own probes. Also happy to leave it as a pattern write-up if a registry feels like the wrong shape for the project. Mostly we wanted to raise the underlying question, because every customized install has this problem whether or not it has a name for it.
All reactions