RTD documentation translation

[rffontenelle]

Details

• readthedocs-docs project in Transifex: https://www.transifex.com/readthedocs/readthedocs-docs/

Expected Result

I would like to have the documentation translation project up to date in Transifex with latest changes in RTD docs. I also expected to have my translation pulled to the source code and to have it enabled in the published documentation.

Actual Result

Currently, readthedocs-docs project in Transifex (not to be confused with readthedocs which is the application translation project) haven't received a single update since 2018, and still accepts translations.

However it's strings are already very different from the current docs which make it harder to contribute because one may read the docs and not find strings or even a whole page.

Note: I initially thought #4463 was supposed to solve this issue, but it turns it was about readthedocs (application translation)

[rffontenelle]

I would like to highlight the frc-docs-translations workflow brought up in #4463 (comment) and next comments, a job nicely done by @Daltz333 and others, which could be used as inspiration for enabling a workflow for fetching translation from as well updating in Transifex.

[mendax1234]

So @rffontenelle Is there any easy way to pull and push translation sources?

[rffontenelle]

@mendax1234 Do you want an explanation or a PR we can start working on?

[mendax1234]

@mendax1234 Do you want an explanation or a PR we can start working on?

I think a PR is better , but if we want more people to know and use it , the explanation will also be a necessity in PR.:joy: @rffontenelle

[mendax1234]

l just want to make sure that @Daltz333 uses a github workflow to pull and push the sources automatically? @rffontenelle

[Daltz333]

I do not touch the sources. Reviewers approve and then they get pulled in twice a day.

[mendax1234]

I do not touch the sources. Reviewers approve and then they get pulled in twice a day.

Thanks. The first thing l want to know is whether the pull cycle is fixed? The second thing is what does the Reviewers here mean? Are the ones who are in WPILIB on github or in WPILIB on transifex or both are the same ones? How do you push sources to transifex? By one person in his local development environment? Besides, if you want to modify something (like change the file's location) after you push this file to the transifex , how do you deal with the condition that there will be two files on transifex (one is the original file and the other is your newly-modified one)? In addition, if you only modify a specific file and don't change its location , it seems that when you push it to transifex again , the translation of the part you have modifed will disappear, how do you deal with this situation? @Daltz333

[Daltz333]

The first thing l want to know is whether the pull cycle is fixed?

Yes. Every 12 hours.

The second thing is what does the Reviewers here mean?

The language reviewers on Transifex. If I remember correctly, the permissions goes Translator -> Reviewer -> Language Coordinator -> Project Maintainer

How do you push sources to transifex?

We push using the Transifex CLI client alongside. This will likely change in the near future, as the new client stability is pretty awful...

how do you deal with the condition that there will be two files on transifex

The translations are copied (by enabling translation memory in Transifex) and the old file is locked for 7 days. After 7 days, it is deleted from the project. 7 days gives enough time in the situation where a file is moved AND modified, allowing translators to retranslate the file.

the translation of the part you have modifed will disappear

Correct, this is by design. Context is extremely important when translating. Even changing a single word in a sentence can dramatically alter the meaning, or context. So when a sentence is updated in source, the translation gets wiped for that specific line.

[mendax1234]

Yes. Every 12 hours.

Thanks , so you only use a github workflow to pull the translations permitted by the Reviewer (not the Project Maintainer) from transifex? Besides , how to set people to these different titile (like Reviewer, project Maintainer)?

We push using the Transifex CLI client alongside. This will likely change in the near future, as the new client stability is pretty awful...

l notice that this push method is also automatic, so when will the update.sh run and how does it run? Do you also set in other places?

The translations are copied (by enabling translation memory in Transifex) and the old file is locked for 7 days. After 7 days, it is deleted from the project. 7 days gives enough time in the situation where a file is moved AND modified, allowing translators to retranslate the file.

l notice that you have written another .py file for this lock_translation , so does it have conflicts with transifex memory? Since both two have the similiar function l think. Besides , how do you enable transifex memory in your project? Is it payment function?

Correct, this is by design. Context is extremely important when translating. Even changing a single word in a sentence can dramatically alter the meaning, or context. So when a sentence is updated in source, the translation gets wiped for that specific line.

If so , how do you deal with this kind of situation. That's when someone is translating a specific file (we call it file A) . And file A happens to be the file you are to modify. And you modify it and push it to the transifex. So, unfortunately, the translator who translates this part will have a vain work. Or , after you push this file, the previous file will be in the locked files and can still be seen?

[rffontenelle]

In addition, if you only modify a specific file and don't change its location , it seems that when you push it to transifex again , the translation of the part you have modifed will disappear, how do you deal with this situation?

Old unused translations in Transifex are suggested to the translator with the percentage of similarity. So, when new source string are pushed to Transifex (even for a new or renamed file), the translator will see that string untranslated and there will be a suggestion.

[mendax1234]

In addition, if you only modify a specific file and don't change its location , it seems that when you push it to transifex again , the translation of the part you have modifed will disappear, how do you deal with this situation?

Old unused translations in Transifex are suggested to the translator with the percentage of similarity. So, when new source string are pushed to Transifex (even for a new or renamed file), the translator will see that string untranslated and there will be a suggestion.

Is that true? If so , that's understandable. So, l don't need to worry about the modified file's translation will disappear forever?

[rffontenelle]

In addition, if you only modify a specific file and don't change its location , it seems that when you push it to transifex again , the translation of the part you have modifed will disappear, how do you deal with this situation?

Old unused translations in Transifex are suggested to the translator with the percentage of similarity. So, when new source string are pushed to Transifex (even for a new or renamed file), the translator will see that string untranslated and there will be a suggestion.

Is that true? If so , that's understandable. So, l don't need to worry about the modified file's translation will disappear forever?

Once a string is translated, it is added to the Translation Memory (TM). For more info on this TM suggestions and also screenshot, see https://docs.transifex.com/translation/using-translation-memory-and-concordance-search

[mendax1234]

In addition, if you only modify a specific file and don't change its location , it seems that when you push it to transifex again , the translation of the part you have modifed will disappear, how do you deal with this situation?

Old unused translations in Transifex are suggested to the translator with the percentage of similarity. So, when new source string are pushed to Transifex (even for a new or renamed file), the translator will see that string untranslated and there will be a suggestion.

Is that true? If so , that's understandable. So, l don't need to worry about the modified file's translation will disappear forever?

Once a string is translated, it is added to the Translation Memory (TM). For more info on this TM suggestions and also screenshot, see https://docs.transifex.com/translation/using-translation-memory-and-concordance-search

Got it , thanks! It seems that l don't have to configure the translation memory. Then what about hte lockdown files? Can you explain how that is done?

[mendax1234]

The first thing l want to know is whether the pull cycle is fixed?

Yes. Every 12 hours.

The second thing is what does the Reviewers here mean?

The language reviewers on Transifex. If I remember correctly, the permissions goes Translator -> Reviewer -> Language Coordinator -> Project Maintainer

How do you push sources to transifex?

We push using the Transifex CLI client alongside. This will likely change in the near future, as the new client stability is pretty awful...

how do you deal with the condition that there will be two files on transifex

The translations are copied (by enabling translation memory in Transifex) and the old file is locked for 7 days. After 7 days, it is deleted from the project. 7 days gives enough time in the situation where a file is moved AND modified, allowing translators to retranslate the file.

the translation of the part you have modifed will disappear

Correct, this is by design. Context is extremely important when translating. Even changing a single word in a sentence can dramatically alter the meaning, or context. So when a sentence is updated in source, the translation gets wiped for that specific line.

Hi @Daltz333 . l have tried to use your automatic push and pull method on my docs and l have some questions about it. l notice that in your update-translations.yml , it needs secrets.TX_TOKEN. But after l run this workflow successfully on my repo, it doesn't ask me to enter the secrets.TX_TOKEN, is that a normal phenomenon? Besides, can you further explain the expected result of your automatic push and pull method? In my repo, it seems that after l configure the settings and modify a file and push it to origin main branch. But nothing are pushed to transifex. (Is that because the time isn't satisfied?)

Thanks

[mendax1234]

l try to push and pull through github action automatically, but it goes wrong and below is the wrong message .

tx INFO: Updating /home/runner/.transifexrc file...
tx INFO: Using TX_TOKEN environment variable. Credentials won't be saved to .transifexrc.
tx ERROR: Authorization Required
tx ERROR: 
Error: Authentication failed. Please make sure your credentials are valid.
For more information, visit:
https://docs.transifex.com/client/client-configuration#-transifexrc.

l have checked my .transifexrc file and it is correct since l can use tx push -s from the local development environment. So what’s wrong with that? And here is my repo: mendax1234/6940Swerve-docs: FRC Team 6940’s Swerve Documentation Powered by Read the Docs (github.com)

[rffontenelle]

@mendax1234

Got it , thanks! It seems that l don't have to configure the translation memory. Then what about hte lockdown files? Can you explain how that is done?

I'm not very familiar with the lockdown process by Transifex, but the concept I know is: the maintainer wants to work on the translation files and this might take some time, so any translation contributed during this process might get lost.

When I say lost is: the translator already translated a string and then it appears as not translated (with a 100% suggestion by the Translation Memory). So, rework.

So locking the resource is a great way to avoid frustrations and rework for the translators, due to maintenance time.

I try to push and pull through github action automatically, but it goes wrong and below is the wrong message .

tx INFO: Updating /home/runner/.transifexrc file...
tx INFO: Using TX_TOKEN environment variable. Credentials won't be saved to .transifexrc.
tx ERROR: Authorization Required
tx ERROR: 
Error: Authentication failed. Please make sure your credentials are valid.
For more information, visit:
https://docs.transifex.com/client/client-configuration#-transifexrc.

I have checked my .transifexrc file and it is correct since l can use tx push -s from the local development environment. So what’s wrong with that? And here is my repo: mendax1234/6940Swerve-docs: FRC Team 6940’s Swerve Documentation Powered by Read the Docs (github.com)

When using TX_TOKEN, tx ignores the .transifexrc. This is useful so you don't have to set this file in a CI/CD environment.

Have you generated a token in https://www.transifex.com/user/settings/api/ ? Make sure TX_TOKEN has the whole notation 1/something. E.g.: 1/1123lihrfwkjh12k3ug545913401301823

[mendax1234]

Thanks! l have already done all the automatic work successfully! And thanks wpilib for providing me with such good resources.

I'm not very familiar with the lockdown process by Transifex, but the concept I know is: the maintainer wants to work on the translation files and this might take some time, so any translation contributed during this process might get lost.

When I say lost is: the translator already translated a string and then it appears as not translated (with a 100% suggestion by the Translation Memory). So, rework.

So locking the resource is a great way to avoid frustrations and rework for the translators, due to maintenance time.

Yes , you are right , lockdown is a good choice.

When using TX_TOKEN, tx ignores the .transifexrc. This is useful so you don't have to set this file in a CI/CD environment.

Have you generated a token in https://www.transifex.com/user/settings/api/ ? Make sure TX_TOKEN has the whole notation 1/something. E.g.: 1/1123lihrfwkjh12k3ug545913401301823

Yes , l have generated a TX_TOKEN , but l find that you also need to set it in your project's secret section. Just create a new environment variable in your secrets and name it as TX_TOKEN and then paste your TOKEN which you get from transifex.

In addition , l think l will write an instruction on how this automatic method works but l don't know whether @Daltz333 will mind my doing this since most of the code is written by WPILIB and l just change some parts to suit my project. l think this instruction will help you write your PR!

Best
Daniel

[rffontenelle]

What CI/CD method would RTD prefer for automatizing the translation workflow? CircleCI, GitHub Actions, ... ? @mendax1234

[mendax1234]

What CI/CD method would RTD prefer for automatizing the translation workflow? CircleCI, GitHub Actions, ... ? @mendax1234

I use Github action and l think it is enough. But l find that in this RTD repo has both of them. 😂

[mendax1234]

In addition , l think l will write an instruction on how this automatic method works but l don't know whether @Daltz333 will mind my doing this since most of the code is written by WPILIB and l just change some parts to suit my project. l think this instruction will help you write your PR!

@rffontenelle l write a brief instruction on how to use this automatic workflow. Hope that it can be regarded as a reference. Here is the link

[rffontenelle]

@Daltz333 Should the new Transifex client work with the current sphinx-intl somehow? I'm getting Error: Cannot create resource, i18n type is unknown when trying to tx-push, and tx migrate won't work either.

@mendax1234 I'll check it out this weekend, thanks for taking notes

[Daltz333]

No. It hasn't been updated, but my colleague is in the process of a PR that updates it, but we encountered significant roadblocks on the Transifex side. Apparently those have been resolved, but I have yet to test it. For now, just use the Python version of the transifex client.

[rffontenelle]

@Daltz333 If implemented the Python version, should tx migrate do the whole job for the new tx client?

[Daltz333]

I do not believe the new client properly respects language mapping, so there is additional work there.

[rffontenelle]

@rffontenelle l write a brief instruction on how to use this automatic workflow. Hope that it can be regarded as a reference. Here is the link

@mendax1234 I haven't thoroughly tested the instructions, but here some considerations to improve them:

LANG_TO_PULL and LANG_MAP

I recommend that you break "Change the LANG_TO_PULL and LANG_MAP to the language you want to be translated into.". First of all, I felt "to the language you want to be translated into" was direct to me as translator when should be to the project maintainer. So I'd rephrase breaking instructions like this:

Change the LANG_TO_PULL to meet the list of languages of languages you want to pull from Transifex. E.g.:

LANG_TO_PULL=${1:-'fr_CA,es_MX,zh_CN,tr_TR,he_IL,pt'}

NOTE: Make sure to keep the leading ${1:-. This is is a Bash parameter expansion that allows a specific value to be used instead of this whole list, which useful for calling update.sh with a specific language code.

Set LANG_MAP with any language code mapping that is required. This way, The language will be pulled from Transifex not with its variant language code, but the mapped language code. E.g.:

LANG_MAP='es_MX: es, fr_CA: fr, he_IL: he, tr_TR: tr'

update.sh's Step 3 format typo

It seems to have a format issue here: "... ./source/`(where you `conf.py locates ..."

.github/workflow/publish_main.yml

Sorry, I don't know about this file. If it is not required for translations then I feel this should removed and go straight to .github/workflow/update-translations.yml

.github/workflow/update-translations.yml

I missed information on how to generate Transifex API token and explanation of what are secrets (TX_TOKEN!) and how to add them to a project's settings.

[mendax1234]

@rffontenelle l write a brief instruction on how to use this automatic workflow. Hope that it can be regarded as a reference. Here is the link

@mendax1234 I haven't thoroughly tested the instructions, but here some considerations to improve them:

LANG_TO_PULL and LANG_MAP

I recommend that you break "Change the LANG_TO_PULL and LANG_MAP to the language you want to be translated into.". First of all, I felt "to the language you want to be translated into" was direct to me as translator when should be to the project maintainer. So I'd rephrase breaking instructions like this:

Change the LANG_TO_PULL to meet the list of languages of languages you want to pull from Transifex. E.g.:

LANG_TO_PULL=${1:-'fr_CA,es_MX,zh_CN,tr_TR,he_IL,pt'}

NOTE: Make sure to keep the leading ${1:-. This is is a Bash parameter expansion that allows a specific value to be used instead of this whole list, which useful for calling update.sh with a specific language code.

Set LANG_MAP with any language code mapping that is required. This way, The language will be pulled from Transifex not with its variant language code, but the mapped language code. E.g.:

LANG_MAP='es_MX: es, fr_CA: fr, he_IL: he, tr_TR: tr'

update.sh's Step 3 format typo

It seems to have a format issue here: "... ./source/`(where you `conf.py locates ..."

.github/workflow/publish_main.yml

Sorry, I don't know about this file. If it is not required for translations then I feel this should removed and go straight to .github/workflow/update-translations.yml

.github/workflow/update-translations.yml

I missed information on how to generate Transifex API token and explanation of what are secrets (TX_TOKEN!) and how to add them to a project's settings.

Thanks for reminding! I will fix it soon!

[mendax1234]

@Daltz333
Hi, l notice that you have changed the frc-docs-translation project to the new API. However, when l try to implement the same thing on my project. Everything goes well except that the transifex has locked all my resources.

l guess that the tx-push command isn't run. Can you help me with that? Thanks! And here is my project:https://github.com/mendax1234/6940Swerve-docs

[rffontenelle]

I'll leave this patch of .circleci/config.yml, because I haven't tested for months now and I'm afraid of loosing it:

  # requires TX_TOKEN secret set with a valid Transifex APIv3 personal token
  translation-update:
    docker:
      - image: 'cimg/python:3.10'
    steps:
      - run: git clone --depth 1 https://${DOCS_GITHUB_TOKEN}@github.com/readthedocs/readthedocs.org.git
      - run: pip3 install -r requirements/docs.txt -r requirements/pip.txt
      # use the sphinx-intl's first commit (yet unreleased) that supports Transifex APIv3
      - run: pip3 install https://github.com/sphinx-doc/sphinx-intl@6b3913fea5a8c9b819818b9b28b6d0d0b6f022b3
      - run: curl -o- https://raw.githubusercontent.com/transifex/cli/master/install.sh | bash
          working_directory: /usr/local/bin
      - run: cd docs
      - run: make gettext
      - run: rm -f .tx/config
      - run: sphinx-intl create-txconfig
      - run: sphinx-intl update-txconfig-resources --transifex-project-name=readthedocs-docs --locale-dir locale --pot-dir _build/locale
      - run: tx push --source --no-interactive --skip
      - run: tx pull --transltions --use-git-timestamps
      - run: git config credential.helper 'cache --timeout=120'
      - run: git config user.name 'github-actions[bot]'
      - run: git config user.email 'github-actions[bot]@users.noreply.github.com'
      - run: git commit --allow-empty -m "[skip ci] Update docs translations"
      # Push quietly to prevent showing the token in log
      - run: git push -q https://${DOCS_GITHUB_TOKEN}@github.com/readthedocs/readthedocs.org.git master

[agjohnson]

Just a note here: we worked translation pull/push into our release process, which happens weekly. This is far off from daily or twice daily, but automating commits was deemed too noisy. We're still looking for a better workflow here. Transifex is a particular source of pain, and it's quite awful having the translation workflow so disconnected from our development workflow. Crowdin was suggested, but it's not clear if it fixes the issues we have with Transifex automation or not.

[rffontenelle]

@agjohnson Thanks for your input. I suggest checking Weblate, an open-source web-based translation platform that has a pull request option (among others options). This pull request option allows to have a "Update translation via Weblate" pull request with commits auto-updated whenever there is a translation update in the translation project set in Weblate, and the maintainers merge it whenever they want. For free-software projects, using Hosted Weblate is free of charge using "libre hosting" plan.

[agjohnson]

Ah, well I'll have to give it another go with Weblate, it's been a while maybe. I don't think I got to setting that up, but it does sound closer to the workflow we'd prefer. Weblate is another project in the Python/Django ecosystem, so we're familiar 😄 We haven't been frustrated enough with Transifex to change things drastically though.

Thanks for the pointer!

[silopolis]

Le mer. 25 janv. 2023 à 04:00, Anthony ***@***.***> a écrit :

Thanks for the pointer!

Speaking of pointers, these should help you make yourself an idea of Weblate possibilities: * https://docs.weblate.org/en/latest/vcs.html#github-pull-requests * https://docs.weblate.org/en/latest/admin/continuous.html#pushing-changes-from-weblate * https://docs.weblate.org/en/latest/admin/continuous.html#lazy-commits * https://docs.weblate.org/en/latest/admin/addons.html#addon-weblate-git-squash

[rffontenelle]

Was there any evolution on this topic (Fix integration with readthedocs-docs project in Transifex, migrating to Weblate or Crowdin, etc.) ?

Just to mention I've gained some Weblate-administrating skills lately and I administrate Python Docs in Transifex, so let me know if help in any of these platforms is needed.