Cross-repo duplication: chtypes.h and the 84 fetch fixtures exist in both repos and must be kept in sync by hand

[EricAndrechek]

For the chtypes-core agent too. Eric's position, and I think it is worth taking seriously rather than defending the status quo: we have content that exists in both repositories and has to be kept in line, and he does not like it. This issue records exactly what is duplicated, why each one is there, what it actually costs, and what the options are — so a decision can be made on evidence rather than on whoever argues last.

What is duplicated

1. The C ABI header

sdk/include/chtypes.h and core/lib/csrc/chtypes.h — 767 lines, byte-identical by assertion.

• The SDK owns it; core pulls it via core/ci/steps/header-sync.sh --pull.
• The assertion runs in core's CI and again in dist/release.sh before every container build.
• Direction is not arbitrary. Core's own comment: "An ABI change is made in the SDK first and PULLED here, never the other way round — which is the order the revision handshake wants anyway: the binding's CHS_ABI_REVISION is what the artifact has to answer."
• Core's stated reason for keeping a copy at all: "This repository compiles the C++ wrapper against lib/csrc/chtypes.h, a copy kept in-tree so that a build never reaches outside the repository."

2. The fetch fixture corpus

sdk/tests/fixtures/fetch/ — 84 files, 97.6 KB. Six miniature fake releases (signed, unsigned, bad-signature, tampered-tarball, sums-index-mismatch, two-builds) plus an expected.json recording what a correct fetcher must do with each.

• Generated by core/tests/sdk/fetch-fixtures/gen.py, committed in the SDK.
• Drift is checked by just fetch-fixtures-check in core, wired into core's pipeline non-blocking, which files and auto-closes a sdk-fetch-fixtures ops issue.
• Core generates them by packing through the real dist/publish.sh, so a fixture is a faithful miniature of production release output.

What it actually costs — with today's evidence

This is not hypothetical. On 2026-09-11 I drifted the header myself, in the course of a restructure and an American-spelling sweep. 34 lines, every one a comment (doc paths and behaviour → behavior), zero declaration changes, CHS_ABI_REVISION 4 on both sides. Harmless in substance — and it still would have failed core's byte-identical assertion on its next fleet rebuild.

That is the honest cost: any edit to either copy, however cosmetic, is a cross-repo event. A tree-wide lint sweep cannot touch one repo without touching the other. The same applies to the fixtures: the SDK cannot rename the directory they live in without core teaching its generator both paths first (which it did, on the same day).

There is also a documented near-miss worth recording: gen.py contains two byte strings spelling spec/fixtures/fetch that are key-derivation labels, not paths — hashed to produce the test key. A well-meaning path sweep that "fixed" them would change the test key and invalidate every signed fixture under it. That trap exists because the same string means two different things across a repo boundary.

Why each one is there — the arguments as they stand

The header. It is Apache 2.0 and public, and it is what the published bindings compile against. Core is private. A single copy living in core would mean a public Apache library whose contract nobody outside can read. A single copy living only in the SDK would mean core's build reaching outside its own repository for a file it compiles against. Both repos have reasons to hold it.

The fixtures. The SDK's CI is hosted-runner-only with no access to core. The fetch tests verify signature checking, hash validation and lock-file pinning, and they must run offline. If the SDK fetched its fixtures it would be using the fetch code to obtain the fixtures that test the fetch code — circular, and it breaks the artifact-free CI job's whole premise. They are generated in core rather than the SDK because they are packed by the same code that packs real releases, which makes the fixture evidence about core's actual output format rather than about the SDK's idea of one.

Options, with their real trades

option	removes the duplication?	what it costs
A. Leave it, improve the guardrails	no	nothing new. Possible improvements: make the header assertion run in the SDK's CI too (today only core notices), and make gen.py's key-derivation labels structurally un-sweepable rather than comment-protected
B. Move the generator into the SDK	yes, for fixtures	the fixture would test the SDK against the SDK's idea of a release, not core's. Loses the contract-test property that is the reason it lives in core
C. Publish the fixtures as a release artifact, fetch them	yes, for fixtures	circular for the fetch tests, and breaks the offline guarantee of the artifact-free CI job
D. Make core consume the header from a submodule / package	yes, for the header	core's build then reaches outside its repo, which is the thing its in-tree copy exists to prevent. Also adds a submodule to a build that deliberately has none
E. Generate the header from a single spec in one repo	yes, for the header	new machinery, and the header is 84% hand-written documentation — a generator would have to own the prose too

My read, offered as input and not as the answer

A duplication audit run today across both repos (250 SDK files against 1,135 core files, hashed) found no other duplication beyond these two and their direct byproducts. Both are deliberate, both have a drift assertion, and the audit's conclusion was that deliberate duplication with a working assertion is usually the correct shape and should not be collapsed.

So my recommendation is A, with the two guardrail improvements — and I want to be plain that Eric's instinct is not wrong, the cost is real, and I was the one who paid it today.

The strongest case for change is the fixtures rather than the header: 84 files is a lot of committed generated content, and option B is genuinely available if the "packed by the real publisher" property turns out to matter less than it sounds. That is worth measuring rather than asserting — has dist/publish.sh's packing format ever actually changed in a way a hand-written packer would have missed? If the answer is no, B gets much more attractive.

Concretely, for whoever picks this up

1. Decide per item — they are independent.
2. If A: add the header assertion to the SDK's CI so drift is caught on the side that authors it, and harden gen.py's key labels structurally.
3. If B: the generator is core/tests/sdk/fetch-fixtures/gen.py. Note the two key-derivation labels before touching anything.
4. Anything here is a cross-repo change and needs both sessions to agree before either moves.

[EricAndrechek]

Correcting my own recommendation in this issue, before anyone acts on it.

Option A's first guardrail — "make the header assertion run in the SDK's CI too (today only core notices)" — is not implementable as written. I wrote it without checking, which is the mistake this repository keeps paying for.

measured, on SDK main:

• The abi job asserts only that include/chtypes.h's CHS_ABI_REVISION matches the four bindings' pinned constants. It is entirely internal to this repository and never looks at core's copy.
• No SDK workflow references core in any form — no checkout, no token, no submodule. The chtypes-core CI (read-only) deploy key was deleted on both sides, and SDK CI runs on hosted runners with no variable and no secret, by design.

So the SDK cannot assert byte-identity against a file it has no way to read. An equality check has to live on the side that can see both copies, which is core. That part of option A stands as it is today, and calling it an improvement was wrong.

What IS available on the SDK side, and is a weaker but real version of the same idea: a check that fires when include/chtypes.h is touched in a pull request, telling the author at authoring time that this file is pulled by another repository and the edit is a cross-repo event. That converts the failure mode from "core's next fleet rebuild fails and someone works backwards" into "the person making the edit is told while making it."

It would have caught the exact thing I did on 2026-09-11 — 34 comment-only lines, harmless in substance, still a cross-repo event. It does not detect drift, because it cannot; it removes the surprise, which is most of what the drift actually costs.

I have not implemented it. It is additive and SDK-only so it needs no cross-repo agreement, but it is a new gate on a file that legitimately changes, and whether that trade is worth it is a judgement — and this issue's own rule is that anything here needs both sessions to agree. Say the word and it is a small PR.

Nothing else in my earlier comment changes: the duplication audit, the two options tables, and the recommendation of A all still stand. It is specifically the "and here is how to improve it" half that was unverified.