Add ensure_ascii option

[Viicos]

Change Summary

Part of pydantic/pydantic#11202.

Related issue number

Checklist

• Unit tests for the changes exist
• Documentation reflects the changes where applicable
• Pydantic tests pass with this pydantic-core (except for expected changes)
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

[codspeed-hq]

CodSpeed Performance Report

Merging #1689 will not alter performance

_{Comparing ensure-ascii (5c32b10) with main (52e9a53)}

Summary

✅ 157 untouched benchmarks

[Viicos]

Looks like a false positive from Clippy? There are already a bunch.

[D-stefaang]

Any chance you'd follow the core python default ensure_ascii=true here? That'd be a breaking change so that's unlikely.
Maybe a mention in the documentation?
https://docs.python.org/3/library/json.html#json.dump

FYI, the httpx library handles unicode nicely, the underlying requests library doesn't... unless explicitly sending UTF-8 bytes.

In [8]: import httpx

In [9]: data = g.model_dump_json()

In [10]: httpx.post('https://httpbin.org/post', data=data)
[info     ] HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" [httpx]
Out[10]: <Response [200 OK]>

In [11]: import requests

In [12]: requests.post('https://httpbin.org/post', data=data)
---------------------------------------------------------------------------
UnicodeEncodeError 
...
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 110-111: Body ('湫莓') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

In [13]: requests.post('https://httpbin.org/post', data=data.encode())
Out[13]: <Response [200 OK]>

[Viicos]

Any chance you'd follow the core python default ensure_ascii=true here?

This can only be done in Pydantic V3, and still I don't know if the motivation is strong enough to do the change.

[davidhewitt]

Suggested change

	If `False` (the default), these characters will be outputted as-is.
	If `False` (the default), these characters will be output as-is.

[Viicos]

Took the wording verbatim from the stdlib json module, but seems like output is by far the most common.

[davidhewitt]

A couple of thoughts here:

• it's probably faster to write whole runs of bytes rather than one char at a time
• I think if the char is below 0xFFFF codepoint it is supposed to be written as that single codepoint. https://www.rfc-editor.org/rfc/rfc8259#section-7

So I came up with this algorithm:

let mut out = Vec::new();
let mut input = "What does 😮💨 this emoji mean?😮";
while let Some((idx, non_ascii_char)) = input
    .chars()
    .enumerate()
    .find(|(_, c)| !c.is_ascii())
{
    if idx > 0 {
        // write all ascii characters before the non-ascii one
        let ascii_run = &input[..idx];
        out.write_all(ascii_run.as_bytes()).unwrap();
    }
    
    let codepoint = non_ascii_char as u32;
    if codepoint < 0xFFFF {
        // write basic codepoint as single escape
        write!(out, "\\u{codepoint:04x}").unwrap();
    } else {
        // encode extended plane character as utf16 pair
        for escape in non_ascii_char.encode_utf16(&mut [0; 2]) {
            write!(out, "\\u{escape:04x}").unwrap();
        }
    }
    
    input = &input[(idx + non_ascii_char.len_utf8())..];
    
}
// write any ascii trailer
out.write_all(input.as_bytes()).unwrap();