Integrate Transifex with CircleCI

[agjohnson]

We do a bad job of keeping translation files up to date. Let's automate it!

Docs here:
https://docs.transifex.com/integrations/github/#integrating-the-client-with-travis-ci

The problem is that these docs are bad. This would result in every PR pushing up translations. This is bad, we probably only want this on pushes to master. That's cool, lets just make a contrib script or something that emulates the steps in the Travis config in the example above and just check for master branch before pushing.

[chenrui333]

I wonder if I can help with this, currently I am thinking of adopting the similar workflow in my company as well.

[stsewd]

The problem is that these docs are bad. This would result in every PR pushing up translations.

Actually you can do that in travis https://docs.travis-ci.com/user/conditional-builds-stages-jobs/

Also, I just saw this https://docs.travis-ci.com/user/deployment/transifex/ (not sure if it's updated)

[chenrui333]

good to know that they have deployment script.

[chenrui333]

but we also need to pull the translated contents as well.

[rffontenelle]

I provided a travis-ci YAML file in the above PR, but i was told RTD is now using circle-ci. Having that said, I suggest renaming this issue Travis -> CircleCI (ok, renamed by stsewd).

Also, here is an untested job description to be used in .circleci/config.yml:

  update-translations:
    docker:
      - image: 'cimg/python:3.8'
    steps:
      - pip3 install -r requirements/docs.txt -r requirements/pip.txt transifex-client
      - cd docs
      - sphinx-build -b gettext . _build/locale
      - echo $'[https://www.transifex.com]\napi_hostname = https://api.transifex.com\nhostname = https://www.transifex.com\npassword = '"$TRANSIFEX_API_TOKEN"$'\nusername = api\n' > ~/.transifexrc
      - rm .tx/config
      - sphinx-intl create-txconfig
      - sphinx-intl update-txconfig-resources --transifex-project-name=readthedocs-docs --locale-dir locale --pot-dir _build/locale
      - tx push -s

TRANSIFEX_API_TOKEN is the variable containing the API token to be used by transifex-client. A maintainer of readthedocs-docs organization in Transifex needs to generate a API token and set it in repository settings in CircleCI.

If a test returns the error saying that only (Transifex) project maintainer can push source strings, it is a great progress!

While a similar set of statements works on TravisCI (see PR!), I wasn't able to test it on CircleCI ( because of some obstacles in CircleCI, I mentioned in the PR).

Anyone willing to test it?

EDIT: Bumped Python version to 3.8 in my suggestion, as the same happened in the .circleci/config.yml.

[agjohnson]

This is great, thanks @rffontenelle! Sounds like we have a few things to coordinate on our side to test

[rffontenelle]

Were we able to move forward on this? It would be awesome to have strings updated on Transifex after 3 years. :-)

[astrojuanlu]

In regards to translations, frc-docs has developed a pretty decent workflow at https://github.com/wpilibsuite/frc-docs-translations

These translations are fully automatic and us maintainers require no work.

Originally posted by @Daltz333 in #8605 (comment)

[Daltz333]

@astrojuanlu In regards to the workflow we use at frc-docs. It's a two repository fully automated process. We use two repositories as to avoid several scenarios.

• Avoid cluttering up git history
• Avoid needing to disable branch protection
• Avoid spamming our main frc-docs (high resource) RTD build.

frc-docs-translations has a submodule of frc-docs. We have a scheduled GitHub action that does the following:

• Updates the current submodule
• Push updated strings to transifex (you must push before pulling or you could have outdated strings)
• Pull translated strings from transifex (we only pull reviewed strings)
• Lock deleted articles for 30 days so translators don't translate a deleted article. We opt to lock the resource instead of immediately delete because we use transifexes "language memory" where it would autofill similar strings.
• Delete resource on transifex after a resource has been locked for 30 days

This is how the repository works at a high level overview. The actual commands themselves are a bit more complex due to the requirement of .tx resource slugs (we wipe and regenerate this every build).

On the ReadTheDocs side, we create a new project for every translation, set the language, then in the main project (frc-docs) we set our other projects as translations.

This doesn't affect RTD servers as much as the on every commit solution because its a scheduled service instead of on-demand. You could use workflow triggers to make this on-demand but I really don't see it as necessary. Translators usually aren't translating within 5minutes of a resource getting updated.

Relevant Scripts:

• https://github.com/wpilibsuite/frc-docs-translations/blob/main/scripts/lock-translations.py (handles locking and deleting stale resources)
• https://github.com/wpilibsuite/frc-docs-translations/blob/main/scripts/update.sh (handles the actual push and pull process)

The "sphinx" project itself just "imports" the submodules conf.py. This breaks on locally hosted extensions (not on pypi), you'll have to add extensions in the submodule manually to the translation repo conf.py too.

Another note: This system (and any other system I can reasonably think of) requires trust of your translators or translator maintainers. If an inappropriate translation comes through, we won't see it until late. There is no good solution to this besides well-trained language reviewers (or paying somebody and having accountability in the process).

[Daltz333]

I'd be happy to set up a Proof of Concept of this process using ReadTheDocs (using ReadTheDocs as a submodule).

In terms of better documenting how to automate translations, it may be better to just have a "canned" translation repo template that people can use for now until R&D work can get done on the RTD side.

[Daltz333]

I figured I could also document our update.sh a bit.

Breaking down

!/bin/sh
# update transifex pot and local po files

set -ex

LANG_TO_PULL=${1:-'fr_CA,es_MX,zh_CN,tr_TR,he_IL,pt'}
LANG_MAP='es_MX: es, fr_CA: fr, he_IL: he, tr_TR: tr'
MAINPROJECT=frc-docs

# Set working directory to repo root
cd `dirname $0`/..

# Create POT Files
sphinx-build -T -b gettext $MAINPROJECT/source locale/pot

# Update .tx/config
rm .tx/config
sphinx-intl create-txconfig
echo "lang_map = ${LANG_MAP}" >> .tx/config
sphinx-intl update-txconfig-resources -p locale/pot -d locale --transifex-project-name $MAINPROJECT

# Push and pull from Transifex. It is important to push then pull!
# If you pull then push, the PO files will contain out of date strings.
if [ "$CI" = true ]
then
    tx push --source --no-interactive --skip
fi
tx pull -l $LANG_TO_PULL --mode onlyreviewed --use-git-timestamps

LANG_TO_PULL=${1:-'fr_CA,es_MX,zh_CN,tr_TR,he_IL,pt'}

First, we declare the language codes we want to pull from transifex.

LANG_MAP='es_MX: es, fr_CA: fr, he_IL: he, tr_TR: tr'
MAINPROJECT=frc-docs

then we map these to codes that ReadTheDocs accepts and declare the name of our project as it shown in our submodule (what the name of the submodule directory is)

# Set working directory to repo root
cd `dirname $0`/..

# Create POT Files
sphinx-build -T -b gettext $MAINPROJECT/source locale/pot

Now we go ahead and build out updated strings

# Update .tx/config
rm .tx/config
sphinx-intl create-txconfig
echo "lang_map = ${LANG_MAP}" >> .tx/config
sphinx-intl update-txconfig-resources -p locale/pot -d locale --transifex-project-name $MAINPROJECT

this text does a variety of things. We delete our transifex-client config and recreate it (transifex-client will try to update the config and it usually is terrible, so we just recreate). Then we regenerate the config and override the language codes in .tx. Lastly, we update the config with all of our resources.

# Push and pull from Transifex. It is important to push then pull!
# If you pull then push, the PO files will contain out of date strings.
if [ "$CI" = true ]
then
    tx push --source --no-interactive --skip
fi
tx pull -l $LANG_TO_PULL --mode onlyreviewed --use-git-timestamps

this last part pushes our new strings from our updated submodule and then pulls the updated translation from transifex. The order here is important.

[rffontenelle]

@Daltz333 have you considered adding --parallel flag to tx pull? It makes it a little bit faster.

[Daltz333]

Parallel actually makes us hit transifex rate limit for when we pull 1500+ resources...

Our project is huge and over 200k strings over 200 articles. 200 articles between 8ish languages is a lot. Remember that we are both pushing and pulling. We often hit the 5000 requests per 5 minutes limit with parallel. Plus CI time is cheap

[rffontenelle]

@Daltz333 I also notice the update-translations workflow file didn't create a ~/.transifexrc file (containing the API token), which in my experience is required for tx-pull. How did you make this work?

[Daltz333]

It will also work if you have a TX_TOKEN environment variable.

[agjohnson]

Thanks for the overview @Daltz333 ! I think we'll be affected by similar issues when we go to automate the process, though our source translations are pretty minor in comparison. The superfluous commits will probably be a small hurdle, as we have some additional process around our application releases and branching.

[humitos]

Today I was asked via support why some things were translated and some other strings were not in the flyout. I raised this in Slack and I arrived at this issue.

After reading the issue, it seems the main complicate thing is to make it work automatically on particular merges to avoid running it too frequently. So, I think we can do the push & pull commands part of the release process we already have ops/deploy.sh. That way, this will be executed once a week and we will only have one commit to update all the strings together with the changelog. @readthedocs/backend what do you think?

[rffontenelle]

Just a heads-up:

The Python transifex-client uses Transifex API v2, which going away in November 30, 2022. The new transifex/cli is going to be the new thing, using the API v3. Since the new client is written in Go lang, it won't be installable via Python's requirements.txt. For more info: https://community.transifex.com/t/postponing-api-2-0-2-5-and-transifex-client-sunset-date/2759

@Daltz333 noticed that sphinx-intl uses API v2, so this is another point of issue. See sphinx-doc/sphinx-intl#60

[humitos]

This integration is done already. We integrated the update for translation strings into our release process, so they will be updated with the same frequency as we release a new version of Read the Docs. We did a small initial test and it did work 👍🏼

Note that we are not using CircleCI or any CI to do this, it's just an extra step into our release scripts. They will call inv docker.translations --action pull and inv docker.translations --action push while doing the release. Besides, these extra invoke tasks added allow us to update the translation string without doing a release as well. There is an example of this execution at #9327

I'm closing this issue now. Expect to have more translated string in the Read the Docs application! Thanks to all the people involved into this issue and also to all the translators for their hard work ❤️

Feel free to comment here or open a new issue if you think there is something wrong with the process we developed.

[rffontenelle]

@humitos Just curious: I saw that readthedocs project in Transifex (i.e. translation of the software itself) was updated twice after your last comment above but i had the impression this and readthedocs-docs project (i.e. translation of the docs) would update at the same time. Are they suppose to be auto updated at the same time?

[humitos]

@rffontenelle Hi! I didn't work on updating readthedocs-docs project. I wasn't aware of that project. I'm pinging @agjohnson whom, I think, is the person that knows more about these translations and their status. Anthony, should we add readthedocs-docs to our release process once it's fully tested with the application translations?

[rffontenelle]

That's awkward. I had this impression this issue was about readthedocs-docs (as it hasn't been updated since 2018) but I may have misunderstood it.

[Daltz333]

I also assumed this was about readthedocs-docs

[stsewd]

We also don't have any translations set up in our docs (e.g, https://docs.readthedocs.io/es/stable/), so this is probably another issue (if we should enable translations for our docs).

[rffontenelle]

Started a new issue ☝, since this one is, i mean, was about application translation.