Emoji/unicode is incorrectly escaped

[TomWright]

I'm the maintainer of dasel.

I've had an issue raised highlighting that emoji's are converted to escaped unicode whereas it is not for JSON, TOML etc.

I have a PR for a local fix here: TomWright/dasel#549.
I can introduce a hacky fix for this (PR above) and I see a related PR on go-yaml/yaml but that PR is no longer maintained.

Is this something we could get fixed at the source?

Thanks

[ccoVeille]

I feel like the fix could be easy by using the "classic range of emoji" in isPrintable

But we have to keep in mind that Unicode is not fun, and emojis are worst

For example
🏳️‍🌈 = 🏳️ +\x{200D} Zero Width Joiner (ZWJ)+ 🌈

but, 👩🏼‍❤️‍💋‍👨🏾 =

1 👩 \x{1F469} Woman
2 🏼 \x{1F3FC} Type-3 Skin Tone
3 ‍ \x{200D} Zero Width Joiner (ZWJ)
4 ❤️ \x{2764}\x{FE0F} Heavy Black Heart + Variation Selector-16
5 ‍ \x{200D} Zero Width Joiner (ZWJ)
6 💋 \x{1F48B} Kiss Mark
7 ‍ \x{200D} Zero Width Joiner (ZWJ)
8 👨🏾

We can think about a regex or something like that, but it might be incomplete, and new emojis and emojis variations are often added

that's why there are lib out there that are maintained

https://github.com/forPelevin/gomoji
https://github.com/rivo/uniseg (that uses https://github.com/mattn/go-runewidth )

But these would bring dependencies

I feel like we can:

• provide something basic that can detect most of "simple unicode" with a regexp
• we can think about having a way to extend the go-yaml capabilities with a plugin. It will expect a function to validate what is an emoji, for people who want "full emoji support" by using an external lib to catch them all

[ccoVeille]

something like this could work as a "naive implementation"

// Regex handling ZWJ sequences, skin tone modifiers, and extended blocks
var advancedEmojiRegex = regexp.MustCompile(
	`(?x)` + // Verbose mode for readability
	`(?:\x{1F1E6}-\x{1F1FF}){2}|` + // Flags (Regional Indicator Sequences)
	`[\x{1F300}-\x{1FAFF}\x{2600}-\x{27BF}\x{E0000}-\x{E007F}]` + // Core + extended emoji blocks + tags
	`[\x{1F3FB}-\x{1F3FF}]?` + // Optional skin tone modifier
	`(?:\x{200D}[\x{1F300}-\x{1FAFF}\x{2600}-\x{27BF}][\x{1F3FB}-\x{1F3FF}]?)*`, // Repeated ZWJ sequences
)

It can also be coded with the logic of go-yaml using range like isUnicode calling isUnicodeFlag + isUnicodeCore + isUnicodeSkinTone + isUnicodeGlue (ZWJ) ... and using position and next positions

So no regexp

[TomWright]

Could we extend the writer/marshaller with a settable is printable func?

From go-yaml you can call out to it if the standard isPrintable is false, but as a consumer you're able to plug in a third party lib.

I suggest this with minimal research into how go-yaml is working under the hood, or how the linked libraries work so could be way off.

[ccoVeille]

exactly, that's the idea I suggested with the "plugin" thing

[TomWright]

AH sorry, I was catching up on my phone and missed it

[ccoVeille]

No need to be sorry. I simply emphasized we agree 😁

But while the plugin could be an option for Unicode Emojis characters that might be combinations of many things, the fix to support "default emojis" (flags + "classic emojis" without ZWJ) might be easy to add to current code, without having a need to thi k about the possible plugin implementation

The implementation could be about range validation close to what you coded in your PR for dasel

[TomWright]

That approach works for me. Thanks for picking up the issue from my PR and being proactive