TOON benchmark for generation tasks

[vetertann]

Hi @johannschopplich , I am doing TOON benchmark for generation tasks. The goal is to test cases of varying complexity across multiple runs using 21 models

I'm comparing JSON vs JSON+SO (constrained decoding) vs TOON across three metrics:

• One-shot accuracy
• Final accuracy after repair cycles
• Overall token budget

Pipeline:

1. LLM generates output in J, JSO, or T format (21 models via Nebius API)
2. For TOON: CLI decoding to JSON. CLI errors trigger repair cycle
3. Validation of all formats via Pydantic
4. Data canonicalization
5. Comparison with gold standard JSON
6. If data doesn't match, repair cycle starts (up to 3 attempts) with previous output and error text inserted into prompt

Test cases:
users - tabular array
order - nested object + array
invoice- nested objects + array
company- nested arrays within arrays

I run this 10 times for each model, then get mean for each metric per model and per case.

TOON example (I use it across all cases):

"Return data in TOON format (2-space indent; arrays show length and fields).\n"
"Example:\n"
"```toon\n"
"id: 100\n"
"title: Sample\n"
"author:\n"
"  id: 5\n"
"  name: Alex\n"
"items[2]{code,value,price}:\n"
"  A1,10,5.5\n"
"  B2,20,8.0\n"
"summary:\n"
"  count: 2\n"
"  total: 13.5\n"
"notes: Example data.\n"
"```\n\n"
"Output ONLY a TOON code block. Use correct headers and [N] values.\n\n"

Task prompt example (order case):

"Create an order record with fields for id, customer (with id and name), and items array with sku, qty, price fields:\n"
"- Order ID: 101\n"
"- Customer: Ada (ID: 9)\n"
"- Items:\n"
"  * Product A1: quantity 2, price $9.99 each\n"
"  * Product B2: quantity 1, price $14.50 each\n"

I've run preliminary tests and have initial results, but I'm still refining prompt quality and case balance. Could you review this setup and provide suggestions, particularly around:

1. Whether the universal TOON example is good one.
2. Case selection - especially the company case with nested arrays, which seems outside TOON's proved acc for uniform arrays
3. Any prompt improvements that might better demonstrate TOON's strengths
4. Maybe split metrics between TOON strength area of application and weak one.

Thanks!

order.toon.txt
company.toon.txt
invoice.toon.txt
users.toon.txt

[johannschopplich]

This setup lines up very well with what TOON is trying to do, and it would nicely complement the existing retrieval / token-efficiency benchmarks, which only measure "read/comprehend this format" rather than "generate valid data in this format."

A few concrete thoughts on your questions:

Universal TOON example

The example you're using is structurally sound and spec‑conformant: 2‑space indent, nested object, a tabular array with [N]{fields}, and a couple of scalar fields. It's a good pattern to show.

For generation, models usually do better when the example is close to the target shape. If you keep a "universal" example, I'd tweak in two small ways:

• Make the tabular part look like your users/items cases (e.g. users[2]{id,name,role}: …) so the model sees exactly the pattern you care about.
• Keep it as short as possible (2–3 rows, minimal extra fields) and rely on the concrete task description for richness. That matches the "show, don't describe; small example" guidance in the TOON LLM docs.

If you're willing to add a bit of complexity, per‑case TOON examples that match each schema (users, order, invoice, company) would probably reduce structural mistakes further.

Case selection (especially company)

Your four cases map almost perfectly onto "TOON sweet spot" vs "TOON off‑label use":

• users: pure uniform tabular array → best case for TOON.
• order / invoice: nested object + one or more uniform arrays → still in the intended design space.
• company: nested arrays within arrays, non‑uniform structure → deliberately outside TOON's strongest area.

I would keep the company case; it's useful to show where JSON/JSON+SO are preferable. The key is how you present results: group users/order/invoice as "TOON‑aligned" scenarios and company as "non‑aligned," and report metrics per group in addition to per‑case. That mirrors the "When to use / When not to use TOON" sections in the docs.

Prompt tweaks to highlight TOON's strengths

A few small adjustments that should help:

• Make the TOON example schema‑aligned with the task (especially field names and array headers), not just structurally similar.
• In the TOON instruction, emphasize:
  • "Match the field names and nesting from the task exactly."
  • "For every array, set [N] to the actual number of rows you output."
• Keep the natural‑language explanation minimal; the example plus "Output ONLY a TOON code block" is usually enough. Extra prose tends to dilute pattern learning.

Symmetry also matters for fairness: give JSON/JSON+SO similarly small, schema‑aligned examples so you're really testing the format rather than how much hand‑holding each prompt gets.

Splitting metrics by TOON's intended vs non‑intended use

Yes, this split would be very helpful. For each format you're comparing, I'd report:

• Metrics aggregated over users/order/invoice (TOON's design space).
• Metrics aggregated over company (deeply nested / arrays‑of‑arrays).

That will make it much easier to read your results alongside the existing retrieval benchmarks, which already separate "uniform / semi‑uniform / deep" structures.

Next steps

From the project side: this is very aligned with TOON's core idea (especially the strict decode + repair loop; that's exactly what [N] and tabular headers are for). Once you're happy with the setup and have stable results, I'd definitely be interested in:

• A write‑up or summary table we can link to, and
• If you're up for it, a PR that adds a short "generation benchmarks" section under docs/guide/benchmarks (even if the harness itself stays in your Python repo and we just describe the methodology and link out).

Feel free to share a small sample of your results when you have them; that will also help refine whether any prompt tweaks are still needed.

[flazouh]

What is this Repair loop you are talking about ?

I would love to have a "PostProcessing" Logic to this package to fix with code malformed output. I'm currently doing that in my own app because I get a 10% decode error, and its a though problem because you don't want to increase the input tokens amount so much, where as some post processing is instant and easy.

[vetertann]

@flazouh regeneration based on error output (for example if validation failed, you send the details of failure and previous output plus instructions)
Trade-off is token overhead though.