Add option to always quote strings in output

[adam444555]

Hi TOON team👋

I’d like to suggest adding an option to always quote string values in toon’s output.

Motivation

When using toon with smaller language models (or lightweight inference setups), outputs that mix quoted and unquoted strings can cause parsing inconsistencies or misinterpretation.
Having a configuration option like --quoteStrings (or similar) would ensure all string values are consistently quoted, improving compatibility and downstream parsing reliability.

Example

const items = [
  { sku: 'A1', qty: 2, price: 9.99 },
  { sku: 'B2', qty: 1, price: 14.5 }
]

encode({ items }, {quoteStrings: true})

Output:

items[2]{sku,qty,price}:
  "A1",2,9.99
  "B2",1,14.5

[Judahh]

Or at least something to escape the delimiters because something like:

[
  {
    "id": 1,
    "name": "Alice Johnson",
    "age": 30,
    "city": "New York",
    "role": "Developer",
    "Test":{
       "Test2":[2,4,6],
       "Test3":[{
           "Test4":45,
           "Test5":"teste, teste /n \n /t ||| \t/t\t\t\" 'm'",
           "Test6":", wat  , wat"
       }]
    }
  },
  {
    "id": 2,
    "name": "Bob Smith",
    "age": 25,
    "city": "San Francisco",
    "role": "Designer"
  },
  {
    "id": 3,
    "name": "Charlie Brown",
    "age": 35,
    "city": "Chicago",
    "role": "Manager"
  }
]

Would parse to:

items[3]:
  - id: 1
  name: Alice Johnson
  age: 30
  city: New York
  role: Developer
  Test:
    Test2: items[3]:
        - 2
        - 4
        - 6
    Test3: items[1]{Test4,Test5,Test6}:
        45,teste, teste /n 
 /t ||| 	/t		" 'm',, wat  , wat
  - id: 2
  name: Bob Smith
  age: 25
  city: San Francisco
  role: Designer
  - id: 3
  name: Charlie Brown
  age: 35
  city: Chicago
  role: Manager

Tested at https://toonifyit.com/

[DoiiarX]

I need this because the deepseek model I'm using recognized two three-digit fields as one six-digit long field. This is because in some cultures, commas are separators within numbers.

[aizo231]

This should be part of spec, otherwise you can't decode strings containing separator.

[bradvogel]

This should be part of spec, otherwise you can't decode strings containing separator.

How so? It should work fine:

Code example

  it('roundtrips array of objects with deeply nested structures and special characters', () => {
    const input = [
      {
        id: 1,
        name: 'Alice Johnson',
        age: 30,
        city: 'New York',
        role: 'Developer',
        Test: {
          Test2: [2, 4, 6],
          Test3: [{
            Test4: 45,
            Test5: 'teste, teste /n \n /t ||| \t/t\t\t" \'m\'',
            Test6: ', wat  , wat',
          }],
        },
      },
      {
        id: 2,
        name: 'Bob Smith',
        age: 25,
        city: 'San Francisco',
        role: 'Designer',
      },
      {
        id: 3,
        name: 'Charlie Brown',
        age: 35,
        city: 'Chicago',
        role: 'Manager',
      },
    ]

    const encoded = encode(input)
    expect(decode(encoded)).toEqual(input)
  })

[JoeHeoJ]

Mandatory double quotes for strings would bloat large TOON documents with redundant tokens. I propose an optional type annotation system within field lists instead.

By allowing (txt), (num), or (bool) annotations after field names, we achieve:

Parse optimization: Direct type mapping without inference logic

Token efficiency: No per-value quote overhead

Explicit typing: Clear distinction between "123" (string) and 123 (number)

Backward compatibility: Annotations would be optional

Example with 10,000 rows:
Mytable[10000]{Name(txt),Firstname(txt),salary(num),...}:

Deepseek's comment on my suggestion regarding the given example :

Token analysis shows:
for a 10,000×5 table, mandatory double quotes introduce ~60,000 quote tokens. Optional type annotations (txt|num|bool) add only 30 tokens in the header—99.95% less overhead. This means faster parsing (direct type assignment) and dramatically smaller documents for LLM contexts.
The trade-off is clear: 60,010 tokens of quoting overhead vs. 30 tokens of type metadata. With annotations, decoders skip inference logic and map values directly—optimizing both space and parse speed.

[bradvogel]

Another thought -

To keep the module pure, and to handle other edge cases like this, the replacer option could be improved to take special return type rawString. It bypasses toon's own encoder.

import { encode, rawString } from 'toon'; // `rawString` is a proposed new export that doesn't exist today
import { escapeString } from 'toon/shared/string-utils'; // optional util that could also be exposed

  const items = [
    { sku: 'A1', qty: 2, price: 9.99 },
    { sku: 'B2', qty: 1, price: 14.5 }
  ]

  const replacer = (_key, value, _path) => {
      // Quote all encoded values, regardless of type
      return rawString(`"${escapeString(String(value))}"`)
  }

  encode(items, { replacer })

  // Output is all quoted strings:
  // [2]{sku,qty,price}:
  //  "A1","2","9.99"
  //  "B2","1","14.5"

Likewise, it could be used to create multi-line strings (which, in my experience, LLMs tend to better understand) eg:

emailContent: |
   Hi there,
   Nice to connect

(example uses YAML-like block scalar syntax which LLMs should be familiar with)

Of course, using rawString is one-way: it can create invalid toon that can't be decoded. IMO this is a fine contract, as most people using toon are doing one-way encoding for LLM consumption.