Skip to content

chore(vendor): track tokscale upstream alignment and TokenBar adaptations #45

Open
Open
chore(vendor): track tokscale upstream alignment and TokenBar adaptations#45
Labels
enhancementNew feature or request

Description

@Nanako0129
Nanako0129
opened on Jul 13, 2026

Objective

Track TokenBar's selective alignment with junhoyeo/tokscale without erasing TokenBar-specific streaming, cache, FFI, Swift, or Windows downstream semantics. This issue is the public exact ledger for the audited upstream range and the post-merge record for each selected milestone.

The goal is not a byte-for-byte re-vendor. An upstream behavior is aligned only when it survives TokenBar's actual scanner → parser → cache → streaming aggregation → FFI → Swift path, including every sibling-source, dedup, pre-aggregation, and stale-cache seam.

Audit boundary: The audited 111-row range is closed. No row in that historical ledger authorizes future runtime work. New upstream commits are reviewed in separately bounded ranges against the current public shared engine; a fresh audit does not select implementation or integration.

Contents

Current checkpoint

Surface Current value
TokenBar current main tip 9e7741da
Current shared-engine pin 5b5f500d
Historical audit target 366ce643
Historical vendor anchor 0c820a5d406241d85dc6c7fc65ac5a1ee026ccfd
Audited range 0c820a5d..366ce643 -- crates/tokscale-core
Core commits 111
Exact audited ledger on main 79/0/0/18/13/1
Remaining TAKE (empty)
Active cache Identity-aware source-message cache format 2; existing format-1 shards rebuild cold; legacy schema-32 source-message-cache.bin remains unread, unmodified, and undeleted
M23 replacement delivery M23-H PR #82 merged at 1a8ee0c6; M23-D PR #83 merged at f99d9274; M23-V (074619f7) remains DEFER
Fidelity-stop evidence PR #72, PR #74, and PR #86 are closed unmerged; Zcode, Copilot VS Code, and Warp remain deferred
Completed terminal path M26-A PR #90 → M26-B PR #91 → M19-BP PR #92 → M19-BQ PR #93 → PT0 PR #94 → M19-B0 PR #99 → M19-B1 PR #102 / TokenBar-Windows PR #7 → D2 PR #105
PT0 / M19-B0 ledger effect Outside the fixed 111-row audit; counts and cache format unchanged
Fresh upstream audit cutoff e7293751 at 2026-08-11 04:29 UTC; latest release is v4.13.0, and the latest core-touching commit in this range is d8b23d1
Fresh bounded audit Audited over 7e3552a7..e7293751: 175 upstream commits total, 76 touching crates/tokscale-core
Fresh audit scope Existing-client correctness (Kimi, OpenCodeReview, Claude, Antigravity, Grok, and Copilot Desktop), pricing/reporting correctness, and Windows-native path handling require separately bounded review; new-client breadth and submit/CLI/CI/refactor-only work remain outside automatic adoption
Audit interpretation The 175/76 counts measure upstream activity through the fixed cutoff, not a simple unported backlog: the public shared engine already selectively carries or adapts some post-range behavior, and no item is selected for implementation by this inventory update
Next pending graph (none — the historical graph is closed; no runtime successor is implicitly authorized) 
git rev-list --reverse \
  0c820a5d406241d85dc6c7fc65ac5a1ee026ccfd..366ce64395594abf111e0409581d91016561b25a \
  -- crates/tokscale-core

Delivery history

Milestone Delivery Ledger after merge Cache/schema Gates
M15-T — exact checkpoint PR #64, a2f852ac 59/29/0/9/13/1 29 Docs and exact-ledger checkpoint; no runtime change
M20 — OpenCode v2 PR #65, 1bc2fa76 60/28/0/9/13/1 30 CONFIRMED · Codex CLEAN · CI PASS
M15-B — Kiro structured PR #66, f5773ea0 62/26/0/9/13/1 30 CONFIRMED · Codex CLEAN · CI PASS
M16 — parser correctness PR #67, aebbc371 66/21/0/10/13/1 31 CONFIRMED · Codex CLEAN · CI PASS
M19-A — Windows atomic retry PR #68, 11ae1bed 67/20/0/10/13/1 31 CONFIRMED · Codex CLEAN · CI PASS
M17 — Grok unified precedence PR #69, d4ff968b 67/20/0/10/13/1 31 CONFIRMED · Codex CLEAN · CI PASS
M18 — routed pricing PR #70, 0735fd2b 69/18/0/10/13/1 31 CONFIRMED · Codex CLEAN · CI PASS
M21 — new local sources PR #71, 471a7f23 74/13/0/10/13/1 31 CONFIRMED · Codex CLEAN · CI PASS
M22 — Zcode evaluation PR #72, closed unmerged at c41b864b 74/8/0/15/13/1 after the product reclassification 31 Fidelity stop; no runtime merge
M22 — canonical fidelity bookkeeping PR #73, aa935d6e 74/8/0/15/13/1 31 Docs-only classification checkpoint
M25 — reloadable grouping aliases PR #75, 93d1583c 75/7/0/15/13/1 31 CONFIRMED · Codex +1 · CI PASS
Grok billing two-meter (non-ledger) PR #76 unchanged 31 Weekly + monthly quota meters; outside the inventory
Grok turn_completed.usage (non-ledger) PR #77, 7b67bb20 unchanged 32 Schema 31→32; Codex +1 · CI PASS
M23-H — Hermes Windows roots PR #82, 1a8ee0c6 76/5/0/16/13/1 32 CONFIRMED · Codex +1 · CI PASS
M23-D — Copilot Desktop PR #83, f99d9274 78/3/0/16/13/1 32 CONFIRMED · Codex +1 · CI PASS
M23-V — VS Code chatSessions No runtime PR; 074619f7 moved to DEFER Included in the M23-H transition 32 Fidelity stop; no runtime merge
D0 — M23 replacement checkpoint PR #84, fb3ecfeb unchanged 32 Docs-only checkpoint
U1 — Copilot upstream contribution checkpoint PR #85, 15c0241b unchanged 32 Records upstream #938/#939 and #942/#943; outside the fixed range
M24 — Warp fidelity stop Runtime PR #86 closed unmerged at 7dcb6985; docs PR #87 merged at ee2946d9 78/2/0/17/13/1 32 Security/state-machine fidelity stop; no Warp runtime code landed
M26-A — format-1 shard cache PR #90, 95c819c7 79/1/0/17/13/1 format 1 CONFIRMED · Codex +1 · CI PASS
M26-B — format-2 metadata PR #91, cc52c3b9 79/0/0/18/13/1 format 2 CONFIRMED · Codex +1 · CI PASS
M19-BP — portable source roots PR #92, 81d99ecd unchanged format 2 CONFIRMED · current-head review · CI PASS
M19-BQ — Claude version probe PR #93, df2fa096 unchanged format 2 CONFIRMED · Codex +1 · CI PASS
PT0 — provider transport PR #94, 346a58ee unchanged; outside the fixed range format 2 CONFIRMED · Codex +1 · CI PASS · zero unresolved threads
M19-B0 — Native secure storage PR #99, f820b06f unchanged; outside the fixed range format 2 CONFIRMED · Codex +1 · CI PASS · zero unresolved threads · equivalent Windows x64 runtime CI PASS
M20 — OpenCode v2 SQLite

Selected scope

  • Ported upstream 366ce643 / PR #920: v2 session_message assistant rows, nested model/provider resolution with v1 precedence, a shared v1/v2 accumulator, strict legacy JSON roles, workspace attribution, token clamps, and TokenBar cost/provider hardening.
  • Preserved distinct embedded IDs and incompatible same-ID SQLite rows while collapsing only fingerprint-compatible forks.

Authority and dedup

  • One selection authority serves materialized, shipping-streaming, and count paths.
  • Logical payload identity excludes cost. A cost-only overlap promotes provider-reported cost over an estimate.
  • Legacy JSON authority uses message ID plus creation timestamp, replaces one exact deferred SQLite identity, and retains provider-reported cost precedence.
  • Rows without a v1 embedded ID retain the row/file fallback as an alternate key; alias authority expands across connected components before streaming, preventing database order from reopening a legacy fallback.

Cache and evidence

  • Schema 29 → 30: a same-fingerprint hybrid database may already contain a non-empty schema-29 v1-only cache entry.
  • Hermetic coverage rejects and rebuilds that entry to v1+v2, then proves warm materialized, shipping-streaming, count, model, monthly, hourly, and Agents parity.
  • Final review head: d3ffbb1559b21f3eb864d6e327cd79306fb10270.
M15-B — Kiro structured sessions

Selected scope

  • Ported upstream 405ded4a / PR #836, 315549b4 / PR #847, and only the Kiro start-anchor hunk of mixed b64d861e.
  • Discovery is limited to ~/.kiro/sessions/<workspace>/sess_*/session.json; sibling messages.jsonl is a parser dependency.

Parser behavior

  • Structured turns estimate input from contextUsage.usagePercentage × 200,000, estimate output from assistant text plus tool-call arguments, preserve duration/model/workspace, and back-anchor a missing prompt timestamp from turn_end - elapsedTime.
  • Older flat role/content JSONL remains an aggregated fallback.
  • New IDE rows fall back to model auto; existing Kiro CLI and SQLite rows retain unknown.

Cache and evidence

  • One kiro_related_messages_path() seam feeds specialized fingerprints, materialized and shipping-streaming cache lanes, latest-mtime observation, and sibling-aware fail-open pruning.
  • Schema remains 30.
  • Fixtures cover absent-to-created and sibling-only rewrites, warm rebuild, modified_after, M15-A/M15-B coexistence, and every report lane.
  • Final review head: 8001efb452d80fc74fad2f87ca1a2b0bd92a3eac.
M16 — existing-parser correctness

Selected scope

  • Ported 6899ea03 / PR #896, b59979c5 / PR #892, 9155018c / PR #890, 18cd13cc / PR #891, the provider-hardening hunks of mixed 34cfbb50 / PR #887, and the Jcode start-anchor hunk of mixed b64d861e / PR #898.
  • The 9Router bridge, scanner, registry, pricing, scripts, and service integration remain excluded.

Correctness changes

  • Codex keeps legacy UUID-v4 ancestor replays behind the child boundary, accepts numeric task_started.started_at, and start-anchors token snapshots.
  • Bare Claude transcript files suppress character-estimated tool-result usage while preserving explicit tokens and project-scoped estimation.
  • Claude and Copilot duplicate records retain per-field maxima, earliest start, and non-additive maximum duration; Copilot prefers OTEL startTime and back-anchors end-only records.
  • Jcode back-calculates explicit completion timestamps from positive tool_duration_ms.
  • Missing GJC/Pi providers are inferred from model identity; delimited Kimi/MiMo/GLM identities normalize to moonshotai / xiaomi / zai; Antigravity aliases reach priced canonical models without merging Low/Medium tiers.

Cache and evidence

  • Schema 30 → 31.
  • A same-fingerprint schema-30 Jcode fixture proves rejection, rebuilt anchoring, warm stability, all-lane parity, and report parity.
  • Final review head: 47a8491068fdc837fdd62364044ee1114cc66b97.
M19-A — Windows atomic replacement retry
  • Ported only the fs_atomic.rs Windows hunk from upstream a87f0ab6 / PR #906; upstream TUI signal and background-screen changes remain excluded.
  • MoveFileExW(MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) retries only ERROR_ACCESS_DENIED (5) and ERROR_SHARING_VIOLATION (32), stops after five total attempts, and uses 10/20/30/40 ms backoff.
  • Other Windows errors and non-Windows std::fs::rename remain single-attempt.
  • Five deterministic tests cover both retryable codes, success, terminal failure, immediate non-transient failure, and exact attempt/backoff order.
  • Schema remains 31. Final review head: 6b2fc0f01a85db99ff00b21e0b587ddfa70cfd1f.
M17 — Grok unified-log precedence

Source authority

  • Selectively ported non-main ed798642 / upstream issue #849 without replacing TokenBar's hardened legacy parser or report pipeline.
  • Discovery accepts only exact top-level logs/unified.jsonl sources beside primary and inferred Grok homes, canonical-deduplicates physical overlap, and excludes archive/backup copies.
  • Raw unified and legacy sources keep independent cache entries; one session authority selector suppresses only legacy rows covered by unified sessions before materialized, shipping-streaming, count, and report folds.
  • Legacy-only sessions remain available. Exact replays collapse without dropping distinct same-base token rows; model/workspace metadata is borrowed only when unique and non-conflicting.
  • Pricing occurs after selection so carried models cannot retain a pre-selection grok-unknown zero cost.

Cache and lifecycle

  • Schema remains 31.
  • Specialized identities include legacy signals.json, summary.json, and events.jsonl siblings.
  • A topology-sensitive source-change token invalidates graph and live-tail caches when unified sources are created, rewritten, or removed even if the maximum source mtime does not change.
  • Lifecycle fixtures cover source creation/removal, stale cache recovery, model switches, malformed tokens, PID reuse, zero-message tool loops, configured roots, cost parity, and every report lane.

Review closure

  1. Corrected four integration defects from the first Codex pass; rejected within-session partial merging because cumulative legacy deltas and unified inference rows have no stable shared identity.
  2. Corrected coarse replay identity, stale cache-hit dates, and lost parser message counts; fresh verification then exposed and closed an adjacent reasoning-token omission.
  3. Added unique, consistent legacy-model carry-over while leaving conflicts fail-closed.
  4. Moved pricing after model carry-over and expanded configured-root unified discovery.
  5. Added terminal GPT identity, generic configured-root inference, and cache-read clamping.
  6. Preserved explicit zero-message loops, handled missing loop indexes, and cleared PID authority at process restarts while retaining pidless session authority.

Final review head: d4e825bb33f14ed1522f70bc7380bf58f1f99c79. Live smoke exited 0; mutable-source drift diagnostics are not used as parity evidence, which comes from hermetic cold/warm fixtures.

M18 — Sakana/Fugu and routed pricing

Selected scope

  • Ported audited 959cce84 and 6c804711, plus request-level long-context semantics from non-main 548dc124 and routed prefix/suffix composition from non-main 6ea27ca1.
  • Sakana subscription billing-console scraping remains excluded.

Pricing behavior

  • fugu-ultra regular rates: $5 input, $30 output, $0.50 cache-read per million tokens.
  • Verified Sakana requests above 272,000 input-plus-cache-read tokens use $10/$45/$1 whole-request rates; exactly 272,000 remains regular.
  • Verified LiteLLM GPT-5.4/GPT-5.5 identities use the same request-level threshold with their own catalog tiers.
  • Output and reasoning follow the selected tier; cache-write does not select the tier; bare fugu remains intentionally unpriced.

Routed lookup invariants

  • Preserve exact raw/custom and parenthesized first refusal, provider-scoped fail-closed behavior, bounded full-path before terminal fallback, case-insensitive forced-source isolation, provider ranking, cache-rate backfill, and one Claude never-degrade guard across every fallback.
  • Review corrections cover provider-prefixed long-context identities, mixed-case forced sources, unknown-router Cursor precedence, composite Sakana hints, and explicit unknown/blank hints.
  • Pricing remains post-cache retrieval; schema stays 31.
  • Final review head: 41652249131d5569463e9f68ed741aa1e2d1d7b8.
M21 — Kimi Code, Junie, and OpenCodeReview

Selected scope

Client Audited source
Kimi Code 839ce378, 052f43de
Junie 633ea946, 77948d9d, plus the Junie hunk of b64d861e
OpenCodeReview 302d39c3, plus the matching hunk of b64d861e

Integration behavior

  • Kimi Code reuses kimi, dispatches by agents/<agent>/wire.jsonl, honors KIMI_CODE_HOME only with environment roots, isolates legacy Kimi's config.json dependency, and collapses exact replays without dropping distinct turns.
  • Junie preserves finite non-negative provider-reported cost, consumes prompt ownership on the next response even when usage is missing, and start-anchors explicit duration.
  • OpenCodeReview carries workspace/duration metadata, includes recorded end timestamps in replay identity, and saturates oversized unsigned token values.
  • Materialized, shipping-streaming, count, and all report paths share parser dispatch and dedup semantics; raw source messages are cached before pricing.
  • Junie and OpenCodeReview append IDs 31 and 32; Swift uses verified display metadata and the existing initial-letter fallback.

Cache and evidence

  • Schema remains 31.
  • Fixtures cover old-zero/new-nonzero discovery, legacy/Code coexistence, cross-agent replay, prompt ownership, provider-cost authority, duration anchors, cache rewrite/removal lifecycle, all-lane parity, reports, and Swift registry behavior.
  • Final review head: 9677224dee48fde428dbd0d8231043c3f63112fe.
M23 replacements — Hermes Windows and Copilot Desktop

Split decision

M23-H — PR #82

  • Added only the Hermes Windows %LOCALAPPDATA%/hermes and supplied-home AppData/Local/hermes candidates when explicit HERMES_HOME is absent.
  • Preserved explicit root/profile isolation, physical database deduplication, plural materialized/streaming/count consumers, WAL observation, and fail-open pruning.
  • Moved c1aef5e9 to ALREADY_VENDORED; schema remained 32.

M23-D — PR #83

  • Rebuilt the fixed ~/.copilot/data.db token source from upstream f6f7eced + 0b454e60, preserving sessions.agent, input-minus-cache normalization, fractional timestamps, and event model/workspace enrichment while excluding AIU-only rows and provider-reported cost.
  • One raw UnifiedMessage selector gives OTEL whole-session authority before pricing, client/date filters, sessionization, and every report fold across materialized, shipping-streaming, and count consumers.
  • Strict DB, optional WAL, and sorted event dependencies drive raw-cache fingerprints, source change tokens, latest mtime, and fail-open modified-after pruning. Read or metadata failures bypass cache reuse and retain the Desktop source plus the full OTEL suppressor cohort.
  • Hermetic fixtures cover date-window authority, cold/warm parity, event creation and rewrites, WAL-only writes, unreadable DB/WAL/events, DB-only cache collisions, count projection, and graph/model/monthly/hourly/Agents totals.
  • Parser production is 285 lines versus 261 in final upstream 0b454e60 (+9.2%). Schema remains 32.

Deferred scope

  • VS Code chatSessions commit 074619f7 is DEFER: upstream and PR feat(vendor): add Copilot Desktop, VS Code chatSessions, and Hermes Windows roots #74 do not reliably replay ObjectMutationLog kind:2 push/splice mutations, and bare-model, agent, and cost authority remain uncontracted.
  • AIU/credits, custom Copilot roots, extra VS Code families, ±2s matching, and a second authority engine are not part of the shipped replacements.

Counting rules

Each audited commit appears in exactly one category. The category represents the highest remaining action for the whole row; mixed commits stay one row while TokenBar selectively lands their approved hunks.

Classification Meaning
ALREADY_VENDORED The relevant behavior is present in TokenBar main, including verified selective ports or local equivalents
TAKE At least one approved hunk remains to be implemented
ADAPT_FOR_STREAMING The upstream behavior is required but still needs a separately classified streaming/cache adaptation
DEFER Only product-deferred or intentionally postponed behavior remains
SKIP No TokenBar implementation is intended for the audited scope
SUPERSEDED A later design replaces the row while its regression semantics remain covered

The ledger must always satisfy all of the following: the audited range contains exactly 111 commits; every hash appears once; the duplicate set is empty; and both symmetric differences between the range and the classification union are empty.

Current inventory

Classification Count
ALREADY_VENDORED 79
TAKE 0
ADAPT_FOR_STREAMING 0
DEFER 18
SKIP 13
SUPERSEDED 1
Total 111

The prior 50/0/0/35/13/1 99-commit checkpoint is retired. M21 merged at 74/13/0/10/13/1; the Zcode fidelity decision produced 74/8/0/15/13/1; M25 produced 75/7/0/15/13/1; M23-H and M23-D produced 78/3/0/16/13/1. M24 then moved 63a44d7c to DEFER without merging runtime code (78/2/0/17/13/1). M26-A moved ae36db5c to ALREADY_VENDORED (79/1/0/17/13/1), and M26-B moved the mixed cd07bf78 row to DEFER after taking only generic format-2 metadata (79/0/0/18/13/1). M19-BP, M19-BQ, PT0, and M19-B0 do not move audited rows.

Applied milestone Ledger effect
M15-T Established 59/29/0/9/13/1
M20 Moved 366ce643 from TAKE to ALREADY_VENDORED
M15-B Moved 405ded4a and 315549b4 to ALREADY_VENDORED
M16 Moved 6899ea03 b59979c5 9155018c 18cd13cc to ALREADY_VENDORED; moved mixed 34cfbb50 to DEFER after taking only provider hardening
M19-A Moved a87f0ab6 to ALREADY_VENDORED after taking only the Windows atomic-replacement hunk
M17 Used non-main ed798642 / upstream issue #849; counts unchanged
M18 Moved 959cce84 and 6c804711 to ALREADY_VENDORED; non-main semantic sources remain outside the ledger
M21 Moved 839ce378 052f43de 633ea946 77948d9d 302d39c3 to ALREADY_VENDORED
M25 Moved 9a5aeb65 to ALREADY_VENDORED
M22 evaluation PR #72 closed unmerged; moved 640e97b9 f7a124da ed6f8b95 65f8f3e2 b64d861e from TAKE to DEFER as a product decision
M23-H Moved c1aef5e9 to ALREADY_VENDORED; moved 074619f7 to DEFER after the PR #74 fidelity stop
M23-D Moved f6f7eced 0b454e60 to ALREADY_VENDORED
M24 evaluation PR #86 closed unmerged; moved 63a44d7c from TAKE to DEFER; no Warp runtime code landed
M26-A Moved ae36db5c from TAKE to ALREADY_VENDORED
M26-B Took generic format-2 metadata from cd07bf78; the remaining Devin scope moved the row from TAKE to DEFER
M19-BP / M19-BQ Portable source-root and Claude version-probe canonicalization; counts unchanged
PT0 Provider transport, diagnostics, FFI publication ordering, and Swift scalar authority; outside the fixed 111-row audit
M19-B0 Native ownership of the cfg-gated Windows CNG／DACL／secure-open／identity／lock／replace／quarantine storage path; counts unchanged
  • Mixed b64d861e remains one DEFER row because only its Kiro, Jcode, Junie, and OpenCodeReview hunks are vendored while Zcode and Devin remain excluded.
  • The active source-message cache is format 2. Existing format-1 shards rebuild cold; the legacy schema-32 monolith remains unread, unmodified, and undeleted.

Product decision

The selected runtime cycle through M19-B0 is complete. The current product boundary is:

Group Current decision
Existing parser correctness OpenCode v2, Kiro structured sessions, Codex/Claude/Copilot/Jcode/provider/Antigravity fixes, Grok unified-log precedence, Hermes Windows discovery, and Copilot Desktop are merged
New local sources Kimi Code, Junie, and OpenCodeReview are merged; Warp producer/local reporting is deferred after PR #86's fidelity stop
Money correctness Sakana/Fugu pricing, request-level long-context evidence, and the routed-pricing precedence pipeline are merged with TokenBar-specific policy retained locally
Runtime configuration Reloadable model aliases are merged for grouping only; raw model identity remains authoritative for pricing/cache/export
Cache architecture Identity-aware format-2 shards are active; format-1 shards rebuild cold and the legacy schema-32 monolith remains inert
Provider transport PT0 is merged outside the 111-row ledger: account-bound last-good, target-bound refresh write-back, bounded diagnostics, FFI publication generations, and shared Swift scalar/presentation authority
Windows parity M19-BP, M19-BQ, M19-B0, M19-B1 exact Native-to-Windows sync, and D2 are complete. Native owns the cfg-gated Windows secure-storage implementation; there is no implicitly authorized runtime successor

The following feature groups remain DEFER: Zcode legacy/v2; Copilot VS Code chatSessions; Warp producer/local reporting; Command Code; CodeBuddy/WorkBuddy; Devin CLI/Desktop; and 9Router. Zcode and Warp each crossed the fidelity threshold in closed-unmerged runtime PRs. Copilot VS Code remains deferred until ObjectMutationLog replay plus model, agent, and cost authority have a reliable upstream format contract. Sakana subscription billing-console scraping (c634d1a5, #745) remains SKIP; selecting Fugu model pricing does not select the subscription usage provider.

Mixed-commit accounting

Commit Selected work Remaining work Transition
34cfbb50 Provider hardening landed in M16 9Router DEFER; only the excluded 9Router scope remains
b64d861e Kiro landed in M15-B, Jcode in M16, and Junie/OpenCodeReview in M21 Zcode and Devin DEFER; PR #72 closed unmerged and both remaining scopes wait for upstream convergence
c1aef5e9 Hermes Windows discovery landed in M23-H; macOS profile discovery was already present None TAKE → ALREADY_VENDORED in PR #82
ae36db5c Main shard architecture in M26; selected Claude dependency hunks are already present None TAKE → ALREADY_VENDORED after M26
cd07bf78 Generic CACHE_FORMAT_VERSION = 2 and related-file path/exists metadata in M26 Devin parser/discovery TAKE → DEFER after M26

ae36db5c alone ends at cache format 1. The selected M26 format-2 contract therefore depends on the generic cache hunks from cd07bf78; the Devin-specific hunks remain excluded.

Exact 111-commit ledger

ALREADY_VENDORED — 79 
6dfd79f5 d9f2a9b7 44055841 1a305f0f 5c1fe659 7500b303 8493048f 2d90f41d
d4a3bd32 1492b962 b43dc5f8 4101711b 28aec200 aebe4ea8 5017eefb 0ce3d73f
3a68cf52 a75533e6 70fd5249 cbbd0dff 81d721fd 7f48257a 0549e2ed 686f3cf2
783bbb8d 5ff7bf44 c2156fea cb6ebf61 4bc2aa03 afa65ed6 235230ac bc06d4ee
979b7015 7403dafa d5f2c6c4 59421da9 31deb7e6 b8156e64 dcb053e0 23cf62e0
4cbc2f6b 0f84d174 da5e06d2 1752636f b7277d49 85669602 b49cec19 3587f745
d50da475 24e3771c e5cfbae2 b64e4f14 72bf6667 46e01977 31bfd167 09344531
163ec570 a2f7cef5 a0929482 366ce643 405ded4a 315549b4 6899ea03 b59979c5
9155018c 18cd13cc a87f0ab6 959cce84 6c804711 839ce378 052f43de 633ea946
77948d9d 302d39c3 9a5aeb65 c1aef5e9 f6f7eced 0b454e60 ae36db5c
TAKE — 0 
(empty)
ADAPT_FOR_STREAMING — 0 
(empty)
DEFER — 18 
18c7e87f db88138b 1c91cb34 6a1535d1 90d28ec0
20f6d4dd b9b7d09f 0097ba7e ed64e77b 34cfbb50
640e97b9 f7a124da ed6f8b95 65f8f3e2 b64d861e 074619f7 63a44d7c cd07bf78
SKIP — 13 
b2b8c1fc 7ddfa748 b48af31e e644f966 010acd85 46f8fff9 c634d1a5
471ad5a2 9b85b671 64f92fe9 8256280a cd394af2 d01db0a7
SUPERSEDED — 1 
88b32ac8

Execution graph

flowchart TD
    H[M23-H Hermes Windows — PR #82] --> D[M23-D Copilot Desktop — PR #83]
    D --> D0[D0 replacement checkpoint — PR #84]
    D0 --> U1[U1 upstream contribution checkpoint — PR #85]
    U1 --> F[M24 fidelity stop — PR #86 closed / PR #87 docs]
    F --> A[M26-A format-1 shards — PR #90]
    A --> B[M26-B format-2 metadata — PR #91]
    B --> P[M19-BP portable roots — PR #92]
    P --> Q[M19-BQ Claude probe — PR #93]
    Q --> T[PT0 provider transport — PR #94]
    T --> S[M19-B0 secure storage — PR #99]
    S --> W[M19-B1 exact Windows sync — complete]
    W --> Z[D2 final docs checkpoint — complete]

    D -. fidelity defer .-> V[M23-V VS Code chatSessions]
    D0 -. fidelity evidence .-> X[M22 Zcode — PR #72 closed]
    F -. runtime deferred .-> R[M24 Warp]
Loading

M26-A, M26-B, M19-BP, M19-BQ, PT0, M19-B0, M19-B1, and D2 are complete. M24 is a completed fidelity stop, not a cache dependency and not a shipped runtime source. M22 Zcode, M23-V Copilot VS Code, and M24 Warp remain deferred. The exact ledger is terminal at 79/0/0/18/13/1; PT0, M19-B0, M19-B1, and D2 are outside that inventory. The historical execution graph is closed, and future upstream reviews start from separately declared bounded ranges against the current public shared engine rather than extending the 111-row denominator.

Milestone transitions

Milestone Selected outcome Ledger transition after merge Cache/schema
M20 OpenCode v2 SQLite — PR #65 366ce643: TAKE → ALREADY_VENDORED 29 → 30
M15-B Kiro structured sessions — PR #66 405ded4a 315549b4: TAKE → ALREADY_VENDORED; mixed b64d861e remained TAKE at that checkpoint Keep 30
M16 Existing-parser correctness — PR #67 6899ea03 b59979c5 9155018c 18cd13cc: TAKE → ALREADY_VENDORED; 34cfbb50: TAKE → DEFER 30 → 31
M17 Grok unified-log precedence — PR #69 Non-main ed798642; counts unchanged Keep 31
M21 Kimi Code, Junie, OpenCodeReview — PR #71 839ce378 052f43de 633ea946 77948d9d 302d39c3: TAKE → ALREADY_VENDORED Keep 31
M22 Zcode legacy/v2 — PR #72 closed unmerged 640e97b9 f7a124da ed6f8b95 65f8f3e2 b64d861e: TAKE → DEFER; no runtime merge Keep 31
M23-H Hermes Windows roots — PR #82 c1aef5e9: TAKE → ALREADY_VENDORED; 074619f7: TAKE → DEFER Keep 32
M23-D Copilot Desktop — PR #83 f6f7eced 0b454e60: TAKE → ALREADY_VENDORED Keep 32
M23-V VS Code chatSessions — deferred 074619f7 remains DEFER No runtime change
M18 Fugu, long context, routed pricing — PR #70 959cce84 6c804711: TAKE → ALREADY_VENDORED; non-main sources excluded from counts Keep 31
M25 Reloadable grouping aliases — PR #75 9a5aeb65: TAKE → ALREADY_VENDORED Keep 31
M24 Warp runtime PR #86 closed unmerged; docs PR #87 63a44d7c: TAKE → DEFER; no runtime code landed Keep 32
M19-A Windows atomic replacement retry — PR #68 a87f0ab6: TAKE → ALREADY_VENDORED Keep 31
M26-A Identity-aware format-1 shard cache — PR #90 ae36db5c: TAKE → ALREADY_VENDORED Activate format 1; legacy schema-32 monolith remains inert
M26-B Generic format-2 metadata and Claude parent recovery — PR #91 cd07bf78: TAKE → DEFER after taking generic hunks and leaving Devin excluded Activate format 2
M19-BP Portable local source roots — PR #92 Counts unchanged Keep format 2
M19-BQ Claude version probe — PR #93 Counts unchanged Keep format 2
PT0 Provider transport and cross-language publication/scalar authority — PR #94 Outside the fixed 111-row audit; counts unchanged Keep format 2
M19-B0 Native security/storage canonicalization — PR #99 at f820b06f Counts unchanged; outside the fixed 111-row audit Keep format 2; canonical CNG／DACL／secure-open／identity／lock／replace／quarantine source
M19-B1 Exact Native-to-Windows final sync — complete (TokenBar PR #102, TokenBar-Windows PR #7) Counts unchanged Synced format 2, PT0, M19-B0, and legacy provenance
D2 Final docs checkpoint — complete (PR #105) Counts unchanged Historical alignment cycle closed; future findings require a separately bounded milestone

The terminal exact ledger is ALREADY_VENDORED 79, TAKE 0, ADAPT_FOR_STREAMING 0, DEFER 18, SKIP 13, and SUPERSEDED 1, total 111. Every future update must apply transitions to the actual previous six sets and regenerate the count, duplicate set, union, and both symmetric differences rather than trusting a planned intermediate value.

Fidelity audit

The audit separates upstream semantic fidelity from TokenBar's unavoidable streaming/cache/FFI integration. Churn ratio is a size signal, not a correctness score.

Milestone Production size versus selected upstream Assessment
M20 OpenCode v2 2.76x churn High adaptation, but the parser port remains close to upstream; most growth is one TokenBar cross-store/streaming authority seam. Keep merged and freeze scope.
M15-B Kiro structured 0.76–0.86x Faithful selective port that reuses existing TokenBar cache/scanner seams.
M16 parser correctness About 1.06x parser additions overall Faithful except for localized Copilot duplicate-span hardening; the narrow gaps were reported through upstream #938/#939 and #942/#943.
M17 Grok unified 4.26x churn Grandfathered high-drift milestone. Keep issue #849 as evidence and do not expand locally without a new contract.
M18 routed pricing 2.78x churn High local product-policy adaptation. Keep TokenBar's complete pricing precedence and safety rules local.
M21 new sources 1.056x production; 99.8% parser net parity Faithful; added code is the bounded TokenBar streaming/cache/count/report seam.
M19-A Windows atomic retry 1.91x net production Runtime behavior is faithful; extra code is deterministic testability and downstream portability.
M22 Zcode 2.1–2.5x production; 3.1–3.5x total Failed the adoption threshold; PR #72 is closed unmerged and the scope is DEFER.
M25 aliases Small upstream map + reloadable invalidation seam Intentional TokenBar adaptation; raw model identity and pricing remain unchanged.
M23-H Hermes Windows Discovery-only bounded residual Faithful reuse of the existing parser, plural consumers, dedup, WAL, mtime, and pruning seams.
M23-D Copilot Desktop 285 local parser lines versus 261 selected upstream lines (+9.2%) Faithful bounded port plus one TokenBar raw authority seam and strict DB/WAL/event freshness.
M23-V VS Code No shipped implementation Deferred after PR #74 reached about 3.1x Copilot production drift without a reliable ObjectMutationLog/model/agent/cost contract.
M24 Warp No shipped implementation PR #86 exposed repeated security/state-machine and cross-process ownership failures; the fidelity stop moved 63a44d7c to DEFER.
M26-A / M26-B Selective format-1 then format-2 cache adoption Upstream shard serialization and generic dependency metadata are retained; TokenBar-specific streaming lanes, parser dependencies, raw authority, and legacy-monolith isolation remain local adaptations.
M19-BP / M19-BQ Outside the 111-row ledger Portable FFI source-root and process-probe canonicalization required for the downstream Windows consumer.
PT0 Outside the 111-row ledger TokenBar-owned provider transport, credential persistence, diagnostics, FFI publication ordering, and Swift scalar/presentation consistency; not an upstream tokscale bug claim.
M19-B0 Outside the 111-row ledger TokenBar-owned cfg-gated Windows quota-storage security canonicalization; not an upstream tokscale bug claim.
Grok #76 / #77 Outside the 111-row ledger Product quota meters and schema-32 turn_completed.usage; recorded separately from the audited range.

The adoption rule is explicit: preserve necessary TokenBar streaming/FFI seams, but stop when the core parser or authority requires repeated systemic repair, review keeps exposing new failure classes, or downstream invention exceeds the upstream feature. Do not continue because of sunk cost.

Current upstream reportability

The 2026-08-11 fresh upstream audit uses fixed cutoff e7293751 and covers 7e3552a7..e7293751: 175 upstream commits total, 76 touching crates/tokscale-core. Latest release at the cutoff is v4.13.0, and the latest core-touching commit is d8b23d1. Historical audit target 366ce643, range 0c820a5d..366ce643, and the fixed 111-row inventory remain unchanged. TokenBar main is 9e7741da, the reviewed shared-engine pin is 5b5f500d, and the exact ledger remains 79/0/0/18/13/1. The 175/76 counts measure upstream activity through this cutoff, not a simple unported backlog or implementation selection.

Classification Candidate / outcome Evidence and next action
FRESH_AUDIT Existing-source candidates span Kimi, OpenCodeReview, Claude, Antigravity, Grok, Copilot Desktop, pricing/reporting, and Windows-native scan-root handling; new-source breadth includes Augment, Reasonix, Freebuff, Prime Agent, Senpi, and standalone Cline/Kimchi Review each candidate family in a separately bounded range against shared-engine pin 5b5f500d. Submit-only, CLI, CI, docs, and refactor-only changes are not TokenBar runtime backlog; this row does not classify any commit as TAKE or authorize a pin advance
MERGED_UPSTREAM Kimi #922 / #923 at 940c1cb5; Kimi #926 / #927 at 1c74ba4b; OpenCodeReview #928 / #929 at 8d0f887e; Copilot #938 / #939 at 1652852f; Copilot #942 / #943 at ff5e67ad All are merged upstream. They remain outside the fixed 111-row audit unless an audited row already represents the same behavior; contributor merge was not performed locally.
REPORTABLE_NOW Pricing cache-rate backfill (linked upstream #57 / #56); Grok counter epochs; Kimi exact replay identity; Kimi extreme-token overflow; OpenCodeReview u64 clamp; OpenCode logical identity/reported-cost authority; optional long-lived pricing refresh TTL Use issue-first for authority/identity contracts and narrow PRs only for isolated arithmetic or parsing defects. Deduplicate against already merged Kimi/OpenCodeReview work before filing.
ALREADY_REPORTED_OPEN Grok #849; routed prefix+suffix #846; request-level long-context #862 Evidence remains open upstream; do not claim the full TokenBar adaptation as an upstream-ready patch.
REVIVE_EXISTING Cowork and <synthetic> share closed-unmerged upstream PR #708 Ask for direction and revive the existing contribution instead of opening a duplicate.
NEEDS_CONFIRMATION Kiro nested sess_* topology and CLI/SQLite auto versus unknown; Copilot VS Code ObjectMutationLog/model/agent/cost semantics Obtain format and compatibility evidence before calling these confirmed upstream bugs. M23-V remains deferred.
ALREADY_FIXED mux #817; Jcode #819; folds #823; Copilot #880; Grok metadata sibling fingerprint #856; M19-A fs_atomic in current upstream Retain local-patch notes only where selective re-vendoring can still cross an older tree.
TOKENBAR_LOCAL HASH/STORE memo, streaming aggregator, Hourly/Agents pre-fold filters, simple_lane! arms, extra sibling folds, live-tail/FFI/cache/report seams, M18 complete pricing policy, reqwest TLS, M19-BP/BQ, PT0, and M19-B0 Do not present these as upstream tokscale bug claims.
DEFER-EVIDENCE Zcode/GLM from PR #72 and Warp from PR #86 Preserve closed-unmerged evidence and re-audit only after upstream or product contracts converge.

The canonical deduplicated reportability inventory is maintained in vendor/README.md; this issue mirrors its public status. A merged upstream PR is not automatically present in TokenBar's selected vendor tree, and an open upstream issue is not evidence that the corresponding TokenBar adaptation is upstream-ready.

Non-main semantic sources

The following sources are deliberately excluded from the 111-row count:

Source Selected use
ed798642 / issue #849 Grok unified-log precedence in M17
548dc124 / issue #862 Verified request-level long-context pricing in M18
6ea27ca1 / issue #846 Routed prefix/suffix composition as one input to the complete M18 pricing pipeline
d1cd03c2 / PR #636 Pre-anchor Warp producer semantics for M24

TokenBar adaptation rules

TokenBar's shipping path is not upstream's materialized CLI path:

scanner
  -> parser
  -> SourceMessageCache / SourceFingerprint
  -> scan_messages_streaming
  -> per-client dedup / precedence
  -> StreamingAggregator / SessionizeAccumulator
  -> tb_core_ffi JSON mapper
  -> TokenBarCore models
  -> DashboardModel / Swift views
  • Never wholesale replace vendored files that contain TokenBar streaming, cache, FFI, pricing, or Windows adaptations.
  • A parser that reads a sibling, journal, history file, SQLite WAL, or metadata database must wire all four sites: fingerprint, active materialized/streaming lanes, latest-mtime probe, and sibling-aware pruning or fail-open behavior.
  • A serialized parser-output change requires an explicit TokenBar cache-schema decision and a same-fingerprint stale-cache regression.
  • Dedup and precedence must be selected once before report folds; do not stream one source and attempt to subtract it after pre-aggregation.
  • Raw model identity remains authoritative for pricing, cache, submit/export, and persistence; configurable aliases apply only at grouping surfaces.
  • New clients are incomplete until scanner, parser, cache identity, streaming/count/report lanes, FFI/Swift registry, settings, and UI identity agree.
  • Format-2 identity-aware shards own every active materialized, streaming, count, and report lane; no active lane writes the legacy monolith.
  • The final Windows sync starts from the merged Native source, classifies Windows-only residuals before copying, and stops if any residual is mixed or conflicting.

Verification and update protocol

Every selected runtime milestone must include an exact upstream diff review, hermetic old-fail/new-pass evidence, explicit cache/schema handling, materialized/streaming/count/report parity, applicable FFI/Swift checks, mandatory vendor/README.md and canonical-doc updates in the same PR, repository gates, fresh adversarial verification, clean GitHub review threads, and green CI.

After each milestone merges:

  1. Record the actual PR and merge SHA.
  2. Apply the exact ledger transition to the previous six sets.
  3. Re-run the 111-row count, duplicate, union, and both symmetric-difference checks.
  4. Record the active cache schema/format and the fixture/review evidence.
  5. Update this issue with the actual next-ready dependencies; do not pre-label planned work as landed.

A milestone is complete only after both merge and this issue update succeed. The private Project tracks executable milestones only and must not duplicate the 111 commit rows. Tagging and release remain separate decisions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions