Align on properly formed language tags

[carlosjeurissen]

Introduction

Historically, browser extensions have been using language tags with two different syntaxes.

1. Using a hyphen, I.E. en-US. This is the proper language tag format as defined in BCP47
2. Using an underscore, I.E. en_US. This is similar to POSIX, ICU and ISO/IEC 15897.

Both syntaxes have been used and supported with mixed support in different areas of the extensions. Support varies per API and browser. This WECG issue covers those areas in an attempt to come to alignment.

_locales directory

In most documentation, locale directories are supposed to use the underscore variant.

Currently this is a requirement for Chrome while Firefox seems to also support the hyphen -.

Going forward, unless there is a clear reason why underscores should be used my proposal would be to start add support for the proper BCP47 tags and disallow the use of underscores in mv4.

manifest.json default_locale

Following documentation, default_locale is supposed to be using the subdirectory name of _locales.

Currently chrome requires the use of the underscore, while Firefox supports both a hyphen and an underscore.

Going forward I suggest we keep this documentation and follow the switch to BCP47 as mentioned in _locales above.

i18n.getUILanguage

Following documentation, this returns a BCP47 language tag. Historically before version 55, Firefox used to return the underscore variation. I suggest we keep this as is.

i18n.getMessage('@@ui_locale')

This is defined as returning the current locale. In Chrome it returns the underscore variant. While in Firefox, probably since version 56, it returns the BCP47 tag.

Going forward, I suggest this to be equal to the variation used in _locales as it could be used to fetch the messages.json file.

Final words

Basically the goal is to align the behaviour across browsers. My opinion would be to always use BCP47. However this comes with a transitional cost which I believe this is worth it. Alternatively we agree for example to use the underscore only for files. However we still need to agree on default_locale and what is returned by @@ui_locale.

Related: #131

[tophf]

It would break external tools like transifex that use the ISO/IEC 15897 standard for language codes (for example en_US).

[patrickkettner]

From Transifex's help page

When we add support for a language, we follow the BCP47 standard. The multiple language locales are based on region subtags.

[tophf]

From https://developers.transifex.com/docs/using-the-client#language-management

Transifex uses the ISO/IEC 15897 standard for language codes

[patrickkettner]

FWIW, the sentence directly following your quote is "If you use a different format for the local language codes, you can define a mapping in your configuration file .tx/config."

[tophf]

When we add support for a language, we follow the BCP47 standard.

This quote is about the internal UI/management in transifex, but I referred to the transifex tools that are specific to extension development and which process _locales directory.

FWIW, the sentence directly following your quote is "If you use a different format for the local language codes, you can define a mapping in your configuration file .tx/config."

Yeah, but that's a huge pain, multiplied by thousands of extensions and dozens of languages.

Either way, there may be hundreds of other utilities that use the classic _ syntax.

[patrickkettner]

Not arguing for or against it, just wanting to clarify that your specific point was not reflective of my use of their tool. Appreciate you taking the time to bring up issues!

[tophf]

Judging by Transifex example alone, using two different standards may be actually an established practice that's implemented in many such tools, i.e. BCP47 is used internally and ISO/IEC 15897 for the files.

[carlosjeurissen]

@ghostwords @tophf The goal here is to align this cross-browser. Updated the post to mention alternative paths we can take. We could for example always use the underscore just for file paths. However we then still need to agree on default_locale and what returned by @@ui_locale.

If tools like Transifex have some way of exporting data specific to extensions and thus use the underscore, they will likely update this if the extension system updates. However I very much see your point of the cost of switching this.

In general using multiple standards seems very counter intuitive. However if using underscores for the file paths we could simply use BCP47 with all the hyphens replaced with underscores.

The reason to align on this is also motivated by proposals like #641.

[tophf]

Judging by the source code Chromium uses ICU for locales, which uses underscores. Since ICU is an industry standard, the same conventions are likely to be used by many tools for extension development.