TOON-Inspired JSON Compression

[thoeltig]

Hey TOON community! 👋

I discovered TOON yesterday and loved the concept of token reduction through label extraction. While I couldn't apply pure TOON to my JSON data, I explored a hybrid approach that combines:

1. Schema extraction (TOON's core concept) - Extract repeated keys from object arrays
2. Single-char key compression - Compress top-level labels with self-contained _map
3. Self-documenting structure - Schema uses full words, only top-level keys need mapping

Quick Results (Example: business meeting as JSON):

Scenario	Compression	Best Format
Single record	28.6%	V2 (schema only)
5 records (formatted)	39.5%	V4 (schema + single-char)
5 records (minified)	55.7%	V4 (schema + single-char)

Complete breakdown with examples, calculations, and multi-record scenarios:
https://gist.github.com/thoeltig/149f9600af4f8e08177543c85a9ea798

Would love feedback on whether this approach could complement or inform TOON development! Happy to discuss trade-offs, use cases, or implementation details.

[johannschopplich]

Hey there, thanks for the input! The approach in your gist lines up very closely with how TOON thinks about structure, and there are a few concrete takeaways for the format and tooling.

1. What you call "schema extraction" is essentially the same core idea as TOON’s tabular arrays: pull the field list out once and then stream values positionally. Your numbers are a nice independent confirmation that this is where most of the win comes from, especially for repeated records.

2. The multi‑record V4 pattern (shared _map plus per‑array schema) is useful design input for future "schema" extensions. The spec already calls out schemas as possible future work; your write‑up is a good example of how a shared schema registry could look and what kind of gains it provides when the same table repeats many times. I would keep that as an optional higher‑level layer rather than part of the core syntax, but it is a clear candidate for an experimental "TOON + schemas" profile or library.

3. Single‑char key compression is orthogonal to TOON. For LLMs we usually want to keep semantically meaningful field names in the data itself, and let the tokenizer reuse subword fragments, rather than pushing all of that through a sidecar dictionary. That said, your V3/V4 experiments are a good reminder that if someone fully controls their schema they can pre‑rename keys (or apply a _map scheme) before encoding to TOON. That feels like a separate transformation layer we can document as an advanced option, not something to bake into the spec.

4. Your discussion of "field density" and when schema extraction stops paying off matches what we see in the benchmarks for semi‑uniform arrays. We should probably surface this more explicitly in the docs as a rule of thumb, and maybe expose a helper in the tooling that estimates when a given array is a good candidate for tabular encoding.

Summa summarum: I don't intend to change the core TOON syntax to include _map or schema IDs, but your gist is valuable as (a) confirmation that TOON’s tabular idea is the right lever and (b) motivation for optional higher‑level tools and a future "schemas" profile. If you are interested, it would be great to plug your V2/V4 encodings into the existing benchmark harness so we can compare real token counts and retrieval accuracy against JSON and TOON side by side. Feel free to open a PR in the main repo!