From 6a7f323e73771b8096ba78714f42010f937b0ab7 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 8 Aug 2022 10:53:03 -0400 Subject: [PATCH 01/49] docs: add OEP - Translations Management --- .../oep-00XX-proc-translations-management.rst | 169 ++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 oeps/processes/oep-00XX-proc-translations-management.rst diff --git a/oeps/processes/oep-00XX-proc-translations-management.rst b/oeps/processes/oep-00XX-proc-translations-management.rst new file mode 100644 index 000000000..904671365 --- /dev/null +++ b/oeps/processes/oep-00XX-proc-translations-management.rst @@ -0,0 +1,169 @@ +OEP-XX: Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot +################################################################################################################# + ++---------------+-----------------------------------------------------------------------------------------------------------+ +| OEP | :doc:`OEP-XX ` | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Title | Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Last Modified | 2022-08-08 | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Authors | Carlos Muniz , | +| | Feanil Patel , | +| | Sarina Canelake | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Arbiter | | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Status | | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Type | Process | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Created | 2022-08-08 | ++---------------+-----------------------------------------------------------------------------------------------------------+ +| Resolution | | ++---------------+-----------------------------------------------------------------------------------------------------------+ + +.. contents:: + :local: + :depth: 1 + +Context +******* + +There are problems with the current method of managing translations via the +edx-transifex-bot. The edx-transifex-bot uses a soon to be deprecated (Nov 30, 2022) +version of the Transifex API, and requires admin rights on Transifex in order to +function. In addition, it is difficult to track why some PRs are merged by the bot, and +some are not, and where the bot is creating and merging PRs. Most recently, it was +discovered that the translations were not uploading properly but it has been impossible +to debug exactly why. In the week before the Nutmeg release, this was a significant pain +point. + +Decision +******** + +To alleviate these issues, we will switch from using the edx-transifex-bot to the GitHub +Transifex App, a stable app provided by Transifex. Benefits of this change include being +easier to maintain and solving a lot of the pain points. As part of this proposal, we +suggest moving translations into their own repository, to make using the GitHub +Transifex app more streamlined and straightforward, and in order to make organizing and +using the up to date translations simpler. + +Description +*********** + +* Edx-transifex-bot is a potential security issue + + * The edx-transifex-bot requires admin rights on Transifex in order to function. Admin + rights give access to private/sensitive information as well as the ability to + permanently delete translation and configuration files. At some point, the login to + the edx-transifex-bot user was lost, and without access to the scripts that the bot + uses to function, this edx-transifex-bot is a security issue we cannot control or + debug. + +* Edx-transifex-bot is a black box: Login/scripts for edx-transifex-bot cannot be found + on GitHub or on wiki/secrets pages, and it is difficult to observe what work it is + supposed to do, or whether it is doing it correctly. +* The translations for the Open edX Maple Release were never uploaded to Transifex, + because the automation handled by the edx-transifex-bot never uploaded it. +* The translations for the Open edX Nutmeg Release were nearly not uploaded because the + edx-transifex-bot did not automatically upload it. In addition, the script for + uploading the translations did not work, and the script had to be tinkered for the new + Transifex API in order to upload it in time for release. The files uploaded were still + not in the preferred format because the new version of the Transifex API does not have + all the same features as the previous API version. + + * Running the script to upload translations requires Transifex admin rights, and there + are few people with Admin rights to Transifex and knowledge of the Transifex API. We + should use the solution provided by Transifex to solve this type of problem. + +Rationale +********* + +* This is an upgrade of a system we use regularly, but do not want to have to maintain + regularly. +* Upgrading from a bot (machine user) to an app/workflow is recommended by GitHub and + makes the translation process more open source. +* The GitHub Transifex Integration App is developed and supported by Transifex +* The Github Transifex App is very simple to configure and with many options. We can set + Transifex Projects to automatically upload/download translation files from a repository + once the translations are reviewed and accepted. +* Transifex only allows a one-to-one relationship between repositories and Transifex + Projects. Organizing all of the translation files into one repository and one Transifex + Project has a lower cost: we pay by the project so we end up paying less, and by + putting all translation files in the same place, it is easier to track translation + progress, and debug translation issues. +* A repo that only contains text/binary files, and uses branches to separate translations + related to Open edX releases can make all interactions with translations very quick and + simple due to the ability to clone the branch of a specific release, with translations + organized by repository name. +* By using an app that is maintained by Transifex the organization, we reduce the + maintenance burden and are more future proof of changes they might make since they + maintain both the API and the github App. + +Consequences +************ + +Impact on Translators +===================== + +As we approach the end of the translation upgrade process, we will need to tactically +move from multiple transifex projects to a single project. This will require +coordination with our translators to ensure that moving forward they are providing +translations in the right place. + +Impact on Site Operators +======================== + +Currently the translation files for any given service or library is stored at the same +place as the code, this has generally simplified the deployment story in the past. With +this change, the translations files will move to their own repository. As we deprecate +the old translations files, the relevant deployment tooling will need to be updated to +pull down the translations from the new repo as a part of the deployment process. This +will impact both the old Ansible based tooling as well as any new docker based tooling. + +Impact on Developers +==================== + +While it won’t directly impact the day-to-day workflow of developers, due to the same +reasons that we impact site operators(new translations location), we will have to update +development tools as well. + +Rejected Alternatives +********************* + +Rewriting the Current Tooling for the New API +============================================= + +While rewriting the current tooling may solve the problems encountered in the last two +Open edX Releases, it will still be difficult to maintain and keep track of the +edx-transifex-bot. The Open edX community will still struggle with debugging difficult +issues that stem from missing source code. + +Making a Transifex Project for each Repository +============================================== + +TCRIL pays for each Transifex Project. As translation support is provided for more repos, +the total cost will also increase. In addition, each Transifex Project will need to be +maintained separately, making debugging issues or tracking the progress of each Transifex +Project time-consuming. + +Next Steps +********** + +* Move translation files to a new repo +* Add GitHub Transifex App to openedx organization +* Connect i18n repo to Transifex via app +* Set up for each release branch +* Create/Copy Transifex Memory + +Locations +********* + +Dumps of the translation/localization files from Transifex for the Open edX Release +project already exist in a repository with the name of openedx/openedx-i18n. A new +repository named openedx/openedx-translations, will be similarly structured, but contain +the translation files for all repos within openedx. The GitHub Transifex app will be +installed in the openedx organization. Similar to how the Build-Test-Release Working +Group creates a new branch for each new named release of edx-platform, translation +releases can also be kept in branches corresponding to edx-platform releases. From d4f6b7a2535b7f4900648f60e9c86419f834269c Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 8 Aug 2022 14:58:54 -0400 Subject: [PATCH 02/49] docs: rename oep & reformat table --- ...oep-0058-proc-translations-management.rst} | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) rename oeps/processes/{oep-00XX-proc-translations-management.rst => oep-0058-proc-translations-management.rst} (72%) diff --git a/oeps/processes/oep-00XX-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst similarity index 72% rename from oeps/processes/oep-00XX-proc-translations-management.rst rename to oeps/processes/oep-0058-proc-translations-management.rst index 904671365..66872bd92 100644 --- a/oeps/processes/oep-00XX-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -1,27 +1,27 @@ -OEP-XX: Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot +OEP-58: Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot ################################################################################################################# -+---------------+-----------------------------------------------------------------------------------------------------------+ -| OEP | :doc:`OEP-XX ` | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Title | Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Last Modified | 2022-08-08 | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Authors | Carlos Muniz , | -| | Feanil Patel , | -| | Sarina Canelake | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Arbiter | | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Status | | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Type | Process | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Created | 2022-08-08 | -+---------------+-----------------------------------------------------------------------------------------------------------+ -| Resolution | | -+---------------+-----------------------------------------------------------------------------------------------------------+ +.. list-table:: + + * - OEP + - :doc:`OEP-58 ` + * - Title + - Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot + * - Last Modified + - 2022-08-08 + * - Authors + - + * Carlos Muniz + * Feanil Patel + * Sarina Canelake + * - Arbiter + - TBD + * - Status + - Accepted + * - Created + - 2022-08-08 +.. * - Resolution +.. - .. contents:: :local: From 2f97f835032afb1638096b6c6d8d5f6135fc576d Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 9 Aug 2022 15:20:41 -0400 Subject: [PATCH 03/49] docs: add Ned as Arbiter --- oeps/processes/oep-0058-proc-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 66872bd92..fc78bc700 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -15,7 +15,7 @@ OEP-58: Using the GitHub Transifex App and a Separate Translation Repo as a Repl * Feanil Patel * Sarina Canelake * - Arbiter - - TBD + - Ned Batchelder * - Status - Accepted * - Created From bd779ed6fe98dc3a485689ec72c1989474f6c7d0 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 10:17:18 -0400 Subject: [PATCH 04/49] docs: demote title to subtile and add new title New title is now: "Translations Management", which matches the file name. --- oeps/processes/oep-0058-proc-translations-management.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index fc78bc700..03ebeccac 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -1,12 +1,15 @@ -OEP-58: Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot -################################################################################################################# +OEP-58: Translations Management +############################### + +Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot +--------------------------------------------------------------------------------------------------------- .. list-table:: * - OEP - :doc:`OEP-58 ` * - Title - - Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot + - Translations Management * - Last Modified - 2022-08-08 * - Authors From 6f575d0dc87403a57e29c4d9e8ab5d2729c8cfb7 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 10:20:32 -0400 Subject: [PATCH 05/49] docs: change status to "Under Review" Also updates the last modified section. --- oeps/processes/oep-0058-proc-translations-management.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 03ebeccac..540678f80 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -11,7 +11,7 @@ Using the GitHub Transifex App and a Separate Translation Repo as a Replacement * - Title - Translations Management * - Last Modified - - 2022-08-08 + - 2022-08-10 * - Authors - * Carlos Muniz @@ -20,7 +20,7 @@ Using the GitHub Transifex App and a Separate Translation Repo as a Replacement * - Arbiter - Ned Batchelder * - Status - - Accepted + - Under Review * - Created - 2022-08-08 .. * - Resolution From 8826ec0236a99067949aceb021648871ede1b310 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 10:40:58 -0400 Subject: [PATCH 06/49] docs: add Type section to table Co-authored-by: Sarina Canelake --- oeps/processes/oep-0058-proc-translations-management.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 540678f80..4c72298e3 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -21,6 +21,8 @@ Using the GitHub Transifex App and a Separate Translation Repo as a Replacement - Ned Batchelder * - Status - Under Review + * - Type + - Process * - Created - 2022-08-08 .. * - Resolution From 89f22841dac505ac6d1a6a021ff6000af0c9c520 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 10:41:46 -0400 Subject: [PATCH 07/49] docs: add Review Period row to table Co-authored-by: Sarina Canelake --- oeps/processes/oep-0058-proc-translations-management.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 4c72298e3..4817e8753 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -25,6 +25,8 @@ Using the GitHub Transifex App and a Separate Translation Repo as a Replacement - Process * - Created - 2022-08-08 + * - Review Period + - TBD .. * - Resolution .. - From 58c39ac71d10efe5e877163fc603a17e41462291 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 10:42:41 -0400 Subject: [PATCH 08/49] docs: lowercase the 't' in tCRIL Co-authored-by: Sarina Canelake --- oeps/processes/oep-0058-proc-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 4817e8753..13ae43868 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -150,7 +150,7 @@ issues that stem from missing source code. Making a Transifex Project for each Repository ============================================== -TCRIL pays for each Transifex Project. As translation support is provided for more repos, +tCRIL pays for each Transifex Project. As translation support is provided for more repos, the total cost will also increase. In addition, each Transifex Project will need to be maintained separately, making debugging issues or tracking the progress of each Transifex Project time-consuming. From 0ae30b1ba8957aa8279820e2adf98be072db4818 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 12:02:35 -0400 Subject: [PATCH 09/49] docs: change Description section to Current State In addition, the first bullet point and sub-bullet point were combined to create a stronger first point. --- .../oep-0058-proc-translations-management.rst | 21 ++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 13ae43868..a0222d12c 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -56,18 +56,15 @@ suggest moving translations into their own repository, to make using the GitHub Transifex app more streamlined and straightforward, and in order to make organizing and using the up to date translations simpler. -Description -*********** - -* Edx-transifex-bot is a potential security issue - - * The edx-transifex-bot requires admin rights on Transifex in order to function. Admin - rights give access to private/sensitive information as well as the ability to - permanently delete translation and configuration files. At some point, the login to - the edx-transifex-bot user was lost, and without access to the scripts that the bot - uses to function, this edx-transifex-bot is a security issue we cannot control or - debug. - +Current State +************* + +* Edx-transifex-bot is a potential security issue: The edx-transifex-bot requires admin + rights on Transifex in order to function. Admin rights give access to private/sensitive + information as well as the ability to permanently delete translation and configuration + files. At some point, the login to the edx-transifex-bot user was lost, and without + access to the scripts that the bot uses to function, this edx-transifex-bot is a + security issue we cannot control or debug. * Edx-transifex-bot is a black box: Login/scripts for edx-transifex-bot cannot be found on GitHub or on wiki/secrets pages, and it is difficult to observe what work it is supposed to do, or whether it is doing it correctly. From a72f9018ca0ba45c5acc7bb3de66a90824f82e7a Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 16:32:35 -0400 Subject: [PATCH 10/49] docs: add widths to table --- oeps/processes/oep-0058-proc-translations-management.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index a0222d12c..0987333b2 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -5,6 +5,7 @@ Using the GitHub Transifex App and a Separate Translation Repo as a Replacement --------------------------------------------------------------------------------------------------------- .. list-table:: + :widths: 25 75 * - OEP - :doc:`OEP-58 ` From 228e3cd9e4815ac6f12ec7c4bc7b855206bf3fe9 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 16:34:18 -0400 Subject: [PATCH 11/49] docs: remove subtitle --- oeps/processes/oep-0058-proc-translations-management.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 0987333b2..3b2cb467d 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -1,9 +1,6 @@ OEP-58: Translations Management ############################### -Using the GitHub Transifex App and a Separate Translation Repo as a Replacement for the edx-transifex-bot ---------------------------------------------------------------------------------------------------------- - .. list-table:: :widths: 25 75 From 77dbb0ee3f731b25e47a25eb03fe089fe0edc610 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 16:42:24 -0400 Subject: [PATCH 12/49] docs: rearrange and rename sections A new section has been added named "Proposed Implementation". This section contains the content from 3 sections: "Next Steps", "Consequences", and "Locations". The "Next Steps" section header was removed, and its content place directly under "Proposed Implementation". The "Consequences" Section was renamed to "Pros & Cons". --- .../oep-0058-proc-translations-management.rst | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/processes/oep-0058-proc-translations-management.rst index 3b2cb467d..4c1af0f5d 100644 --- a/oeps/processes/oep-0058-proc-translations-management.rst +++ b/oeps/processes/oep-0058-proc-translations-management.rst @@ -103,19 +103,28 @@ Rationale maintenance burden and are more future proof of changes they might make since they maintain both the API and the github App. -Consequences -************ +Proposed Implementation +*********************** -Impact on Translators -===================== +* Move translation files to a new repo +* Add GitHub Transifex App to openedx organization +* Connect i18n repo to Transifex via app +* Set up for each release branch +* Create/Copy Transifex Memory + +Pros & Cons +=========== + +Con: Impact on Translators +-------------------------- As we approach the end of the translation upgrade process, we will need to tactically move from multiple transifex projects to a single project. This will require coordination with our translators to ensure that moving forward they are providing translations in the right place. -Impact on Site Operators -======================== +Con: Impact on Site Operators +----------------------------- Currently the translation files for any given service or library is stored at the same place as the code, this has generally simplified the deployment story in the past. With @@ -124,13 +133,24 @@ the old translations files, the relevant deployment tooling will need to be upda pull down the translations from the new repo as a part of the deployment process. This will impact both the old Ansible based tooling as well as any new docker based tooling. -Impact on Developers -==================== +Con: Impact on Developers +------------------------- While it won’t directly impact the day-to-day workflow of developers, due to the same reasons that we impact site operators(new translations location), we will have to update development tools as well. +Locations +========= + +Dumps of the translation/localization files from Transifex for the Open edX Release +project already exist in a repository with the name of openedx/openedx-i18n. A new +repository named openedx/openedx-translations, will be similarly structured, but contain +the translation files for all repos within openedx. The GitHub Transifex app will be +installed in the openedx organization. Similar to how the Build-Test-Release Working +Group creates a new branch for each new named release of edx-platform, translation +releases can also be kept in branches corresponding to edx-platform releases. + Rejected Alternatives ********************* @@ -149,23 +169,3 @@ tCRIL pays for each Transifex Project. As translation support is provided for mo the total cost will also increase. In addition, each Transifex Project will need to be maintained separately, making debugging issues or tracking the progress of each Transifex Project time-consuming. - -Next Steps -********** - -* Move translation files to a new repo -* Add GitHub Transifex App to openedx organization -* Connect i18n repo to Transifex via app -* Set up for each release branch -* Create/Copy Transifex Memory - -Locations -********* - -Dumps of the translation/localization files from Transifex for the Open edX Release -project already exist in a repository with the name of openedx/openedx-i18n. A new -repository named openedx/openedx-translations, will be similarly structured, but contain -the translation files for all repos within openedx. The GitHub Transifex app will be -installed in the openedx organization. Similar to how the Build-Test-Release Working -Group creates a new branch for each new named release of edx-platform, translation -releases can also be kept in branches corresponding to edx-platform releases. From 01cbfbab93d498d03c504a397bcffa44c9f3ff1b Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 10 Aug 2022 16:45:38 -0400 Subject: [PATCH 13/49] docs: move OEP-58 to Arch and rename with "arch" --- .../oep-0058-arch-translations-management.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename oeps/{processes/oep-0058-proc-translations-management.rst => architectural-decisions/oep-0058-arch-translations-management.rst} (100%) diff --git a/oeps/processes/oep-0058-proc-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst similarity index 100% rename from oeps/processes/oep-0058-proc-translations-management.rst rename to oeps/architectural-decisions/oep-0058-arch-translations-management.rst From 87ab993b31975472e682ef39c70eff77a9ae1860 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 11 Aug 2022 09:44:16 -0400 Subject: [PATCH 14/49] docs: expand Rejected Alternative question --- .../oep-0058-arch-translations-management.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 4c1af0f5d..47f277bfb 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -165,7 +165,12 @@ issues that stem from missing source code. Making a Transifex Project for each Repository ============================================== -tCRIL pays for each Transifex Project. As translation support is provided for more repos, -the total cost will also increase. In addition, each Transifex Project will need to be +As translation support is provided for more repos, the total cost will also increase. A +Transifex Project houses the content to be translated and needs to be created before any +content can be added for translation. Transifex Projects can only support 1 GitHub +repository each and need to be maintained separately. Maintaining a Transifex Project +involves adjusting configurations, adding new languages, assigning translators to +projects, or any other miscellaneous irregular tasks that would be time-consuming at a +larger scale. If we adda a Transifex Project, each Transifex Project will need to be maintained separately, making debugging issues or tracking the progress of each Transifex Project time-consuming. From 36385083c3d44a627966211ea74eefc612258bed Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 11 Aug 2022 09:51:45 -0400 Subject: [PATCH 15/49] docs: update Rejected Alternatives question 1 --- .../oep-0058-arch-translations-management.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 47f277bfb..23934b747 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -157,10 +157,12 @@ Rejected Alternatives Rewriting the Current Tooling for the New API ============================================= -While rewriting the current tooling may solve the problems encountered in the last two -Open edX Releases, it will still be difficult to maintain and keep track of the -edx-transifex-bot. The Open edX community will still struggle with debugging difficult -issues that stem from missing source code. +The source code for the edx-transifex-bot is missing. We could rewrite the current +tooling to try to solve the problems encountered in the last two Open edX releases and +upgrade to the new API, but this approach would require a full rewrite, potentially more +expensive than doing the rewrite in a way that Transifex more cleanly supports. It should +also be mentioned that GitHub discourages the use of bots and separate bot accounts; they +strongly recommend using GitHub Apps. Making a Transifex Project for each Repository ============================================== From 588e3a64b5626b09013846018624e8f14e10523f Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 11 Aug 2022 09:52:54 -0400 Subject: [PATCH 16/49] docs: change type to Architecture Decision --- .../oep-0058-arch-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 23934b747..8b414f7d6 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -20,7 +20,7 @@ OEP-58: Translations Management * - Status - Under Review * - Type - - Process + - Architecture Decision * - Created - 2022-08-08 * - Review Period From 973cf3ea89bacb71c02e09c0ca7af4436fee3d57 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 11 Aug 2022 10:05:10 -0400 Subject: [PATCH 17/49] fix: add correct file name reference for OEP --- .../oep-0058-arch-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 8b414f7d6..82813f167 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -5,7 +5,7 @@ OEP-58: Translations Management :widths: 25 75 * - OEP - - :doc:`OEP-58 ` + - :doc:`OEP-58 ` * - Title - Translations Management * - Last Modified From 116ded77da3577f9b3d7cf6bb86958699035c29b Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 6 Sep 2022 11:11:19 -0400 Subject: [PATCH 18/49] docs: change Pros & Cons section to Impacts --- .../oep-0058-arch-translations-management.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 82813f167..78771576c 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -112,19 +112,19 @@ Proposed Implementation * Set up for each release branch * Create/Copy Transifex Memory -Pros & Cons -=========== +Impacts +******* -Con: Impact on Translators --------------------------- +Impact on Translators +===================== As we approach the end of the translation upgrade process, we will need to tactically move from multiple transifex projects to a single project. This will require coordination with our translators to ensure that moving forward they are providing translations in the right place. -Con: Impact on Site Operators ------------------------------ +Impact on Site Operators +======================== Currently the translation files for any given service or library is stored at the same place as the code, this has generally simplified the deployment story in the past. With @@ -133,8 +133,8 @@ the old translations files, the relevant deployment tooling will need to be upda pull down the translations from the new repo as a part of the deployment process. This will impact both the old Ansible based tooling as well as any new docker based tooling. -Con: Impact on Developers -------------------------- +Impact on Developers +==================== While it won’t directly impact the day-to-day workflow of developers, due to the same reasons that we impact site operators(new translations location), we will have to update From 2836cde839e554414c7de5605d5a8ca9e8d1b235 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 6 Sep 2022 11:11:59 -0400 Subject: [PATCH 19/49] docs: change header level of Locations section --- .../oep-0058-arch-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 78771576c..c9e670b97 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -141,7 +141,7 @@ reasons that we impact site operators(new translations location), we will have t development tools as well. Locations -========= +********* Dumps of the translation/localization files from Transifex for the Open edX Release project already exist in a repository with the name of openedx/openedx-i18n. A new From 01195707fb305dfdc19baa9e921c70981f785de1 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 6 Sep 2022 11:57:53 -0400 Subject: [PATCH 20/49] docs: expand Proposed Implementation section --- .../oep-0058-arch-translations-management.rst | 41 ++++++++++++++++--- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index c9e670b97..90ed2c5c6 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -106,11 +106,42 @@ Rationale Proposed Implementation *********************** -* Move translation files to a new repo -* Add GitHub Transifex App to openedx organization -* Connect i18n repo to Transifex via app -* Set up for each release branch -* Create/Copy Transifex Memory +Move translation files to a new repo +==================================== + +Translation files (of types .mo and .po) currently exist amongst the code/documentation +they translate. We will move these Translation files from being amongst the +code/documentation to their own repo. For easier reintegration, Translation files will be +kept in the same directory structure as the code/documentation they translate. + +Add GitHub Transifex App to openedx organization +================================================ + +Transifex's GitHub app will need to be added to the openedx GitHub organization in order +to grant the app permissions to push/pull the Translation files. Currently, we manage the +push/pull permissions for the edx-transifex-bot through a number of groups. The Transifex +GitHub app is granted permission on a repository basis, and by moving all the +Translations to a single repository we reduce the number of user groups we will have to +manage. + +Connect i18n repository to Transifex via app +============================================ + +The Transifex webapp accepts configuration files for each Transifex project. By +connecting the single repository containing all Translation files, we only need to make a +single configuration file that allows the Transifex GitHub app to manage the Translation +files. Based on the Translation Working Group's instruction, we can set parameters that +automatically push and pull Translation files. + +Copy Transifex Memory +===================== + +As a last step we can save all the progress the Open edX translators have accomplished by +copying the Transifex Memory, the auto-translation feature that allows for Projects with +similar strings to be automatically translated, from the old projects to this new one. By +moving all the Translation Files to the same repository we can increase the reach of the +Transifex Memory feature to help translate similar strings across the entire +code/documentation base. Impacts ******* From 20b9e6c91c45d40a224dfdd288b743d884625cfd Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 6 Sep 2022 11:58:51 -0400 Subject: [PATCH 21/49] docs: replace repo with full word --- .../oep-0058-arch-translations-management.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 90ed2c5c6..543bd723c 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -95,10 +95,10 @@ Rationale Project has a lower cost: we pay by the project so we end up paying less, and by putting all translation files in the same place, it is easier to track translation progress, and debug translation issues. -* A repo that only contains text/binary files, and uses branches to separate translations - related to Open edX releases can make all interactions with translations very quick and - simple due to the ability to clone the branch of a specific release, with translations - organized by repository name. +* A repository that only contains text/binary files, and uses branches to separate + translations related to Open edX releases can make all interactions with translations + very quick and simple due to the ability to clone the branch of a specific release, + with translations organized by repository name. * By using an app that is maintained by Transifex the organization, we reduce the maintenance burden and are more future proof of changes they might make since they maintain both the API and the github App. @@ -161,8 +161,9 @@ Currently the translation files for any given service or library is stored at th place as the code, this has generally simplified the deployment story in the past. With this change, the translations files will move to their own repository. As we deprecate the old translations files, the relevant deployment tooling will need to be updated to -pull down the translations from the new repo as a part of the deployment process. This -will impact both the old Ansible based tooling as well as any new docker based tooling. +pull down the translations from the new repository as a part of the deployment process. +This will impact both the old Ansible based tooling as well as any new docker based +tooling. Impact on Developers ==================== @@ -177,8 +178,8 @@ Locations Dumps of the translation/localization files from Transifex for the Open edX Release project already exist in a repository with the name of openedx/openedx-i18n. A new repository named openedx/openedx-translations, will be similarly structured, but contain -the translation files for all repos within openedx. The GitHub Transifex app will be -installed in the openedx organization. Similar to how the Build-Test-Release Working +the translation files for all repositories within openedx. The GitHub Transifex app will +be installed in the openedx organization. Similar to how the Build-Test-Release Working Group creates a new branch for each new named release of edx-platform, translation releases can also be kept in branches corresponding to edx-platform releases. From 8f63bbec61c26cc158e28134fb884f2b10201293 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 10:26:37 -0400 Subject: [PATCH 22/49] docs: update oep-0058 Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake --- .../oep-0058-arch-translations-management.rst | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 543bd723c..42b786971 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -49,8 +49,8 @@ Decision To alleviate these issues, we will switch from using the edx-transifex-bot to the GitHub Transifex App, a stable app provided by Transifex. Benefits of this change include being -easier to maintain and solving a lot of the pain points. As part of this proposal, we -suggest moving translations into their own repository, to make using the GitHub +easier to maintain and solving a lot of the pain points detailed below. As part of this proposal, we +suggest moving translations into their own repository, to make using the GitHub Transifex app more streamlined and straightforward, and in order to make organizing and using the up to date translations simpler. @@ -87,7 +87,7 @@ Rationale * Upgrading from a bot (machine user) to an app/workflow is recommended by GitHub and makes the translation process more open source. * The GitHub Transifex Integration App is developed and supported by Transifex -* The Github Transifex App is very simple to configure and with many options. We can set +* The Github Transifex App is very simple to configure and has many options. We can set Transifex Projects to automatically upload/download translation files from a repository once the translations are reviewed and accepted. * Transifex only allows a one-to-one relationship between repositories and Transifex @@ -101,12 +101,12 @@ Rationale with translations organized by repository name. * By using an app that is maintained by Transifex the organization, we reduce the maintenance burden and are more future proof of changes they might make since they - maintain both the API and the github App. + maintain both the API and the GitHub App. Proposed Implementation *********************** -Move translation files to a new repo +Move Translation Files to a New Repo ==================================== Translation files (of types .mo and .po) currently exist amongst the code/documentation @@ -114,7 +114,7 @@ they translate. We will move these Translation files from being amongst the code/documentation to their own repo. For easier reintegration, Translation files will be kept in the same directory structure as the code/documentation they translate. -Add GitHub Transifex App to openedx organization +Add GitHub Transifex App to openedx Organization ================================================ Transifex's GitHub app will need to be added to the openedx GitHub organization in order @@ -124,7 +124,7 @@ GitHub app is granted permission on a repository basis, and by moving all the Translations to a single repository we reduce the number of user groups we will have to manage. -Connect i18n repository to Transifex via app +Connect the New Translation Repository to Transifex ============================================ The Transifex webapp accepts configuration files for each Transifex project. By @@ -158,7 +158,7 @@ Impact on Site Operators ======================== Currently the translation files for any given service or library is stored at the same -place as the code, this has generally simplified the deployment story in the past. With +place as the code, which has generally simplified the deployment story in the past. With this change, the translations files will move to their own repository. As we deprecate the old translations files, the relevant deployment tooling will need to be updated to pull down the translations from the new repository as a part of the deployment process. @@ -177,7 +177,7 @@ Locations Dumps of the translation/localization files from Transifex for the Open edX Release project already exist in a repository with the name of openedx/openedx-i18n. A new -repository named openedx/openedx-translations, will be similarly structured, but contain +repository named openedx/openedx-translations will be similarly structured, but it will contain the translation files for all repositories within openedx. The GitHub Transifex app will be installed in the openedx organization. Similar to how the Build-Test-Release Working Group creates a new branch for each new named release of edx-platform, translation @@ -196,15 +196,15 @@ expensive than doing the rewrite in a way that Transifex more cleanly supports. also be mentioned that GitHub discourages the use of bots and separate bot accounts; they strongly recommend using GitHub Apps. -Making a Transifex Project for each Repository +Making a Transifex Project for Each Repository ============================================== -As translation support is provided for more repos, the total cost will also increase. A +As translation support is provided for more repos, the effort to maintain the translations infrastructure increases. A Transifex Project houses the content to be translated and needs to be created before any content can be added for translation. Transifex Projects can only support 1 GitHub repository each and need to be maintained separately. Maintaining a Transifex Project involves adjusting configurations, adding new languages, assigning translators to projects, or any other miscellaneous irregular tasks that would be time-consuming at a -larger scale. If we adda a Transifex Project, each Transifex Project will need to be +larger scale. If we add a Transifex Project, each Transifex Project will need to be maintained separately, making debugging issues or tracking the progress of each Transifex Project time-consuming. From 224a5dfbee1198e8cbc9d1fd6d4a428b0f01f07d Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 10:35:32 -0400 Subject: [PATCH 23/49] docs: correct format to follow RST guidelines Update oeps/architectural-decisions/oep-0058-arch-translations-management.rst Co-authored-by: Sarina Canelake --- .../oep-0058-arch-translations-management.rst | 40 +++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 42b786971..099fcb165 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -49,10 +49,10 @@ Decision To alleviate these issues, we will switch from using the edx-transifex-bot to the GitHub Transifex App, a stable app provided by Transifex. Benefits of this change include being -easier to maintain and solving a lot of the pain points detailed below. As part of this proposal, we -suggest moving translations into their own repository, to make using the GitHub -Transifex app more streamlined and straightforward, and in order to make organizing and -using the up to date translations simpler. +easier to maintain and solving a lot of the pain points detailed below. As part of this +proposal, we suggest moving translations into their own repository, to make using the +GitHub Transifex app more streamlined and straightforward, and in order to make +organizing and using the up to date translations simpler. Current State ************* @@ -125,7 +125,7 @@ Translations to a single repository we reduce the number of user groups we will manage. Connect the New Translation Repository to Transifex -============================================ +=================================================== The Transifex webapp accepts configuration files for each Transifex project. By connecting the single repository containing all Translation files, we only need to make a @@ -169,7 +169,7 @@ Impact on Developers ==================== While it won’t directly impact the day-to-day workflow of developers, due to the same -reasons that we impact site operators(new translations location), we will have to update +reasons that we impact site operators (new translations location), we will have to update development tools as well. Locations @@ -177,11 +177,11 @@ Locations Dumps of the translation/localization files from Transifex for the Open edX Release project already exist in a repository with the name of openedx/openedx-i18n. A new -repository named openedx/openedx-translations will be similarly structured, but it will contain -the translation files for all repositories within openedx. The GitHub Transifex app will -be installed in the openedx organization. Similar to how the Build-Test-Release Working -Group creates a new branch for each new named release of edx-platform, translation -releases can also be kept in branches corresponding to edx-platform releases. +repository named openedx/openedx-translations will be similarly structured, but it will +contain the translation files for all repositories within openedx. The GitHub Transifex +app will be installed in the openedx organization. Similar to how the Build-Test-Release +Working Group creates a new branch for each new named release of edx-platform, +translation releases can also be kept in branches corresponding to edx-platform releases. Rejected Alternatives ********************* @@ -199,12 +199,12 @@ strongly recommend using GitHub Apps. Making a Transifex Project for Each Repository ============================================== -As translation support is provided for more repos, the effort to maintain the translations infrastructure increases. A -Transifex Project houses the content to be translated and needs to be created before any -content can be added for translation. Transifex Projects can only support 1 GitHub -repository each and need to be maintained separately. Maintaining a Transifex Project -involves adjusting configurations, adding new languages, assigning translators to -projects, or any other miscellaneous irregular tasks that would be time-consuming at a -larger scale. If we add a Transifex Project, each Transifex Project will need to be -maintained separately, making debugging issues or tracking the progress of each Transifex -Project time-consuming. +As translation support is provided for more repos, the effort to maintain the +translations infrastructure increases. A Transifex Project houses the content to be +translated and needs to be created before any content can be added for translation. +Transifex Projects can only support 1 GitHub repository each and need to be maintained +separately. Maintaining a Transifex Project involves adjusting configurations, adding new +languages, assigning translators to projects, or any other miscellaneous irregular tasks +that would be time-consuming at a larger scale. If we add a Transifex Project, each +Transifex Project will need to be maintained separately, making debugging issues or +tracking the progress of each Transifex Project time-consuming. From de686831b21f78ab235127c03ad17b2047aa0904 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 10:47:05 -0400 Subject: [PATCH 24/49] docs: condense sub-bullet point into parent --- .../oep-0058-arch-translations-management.rst | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 099fcb165..2639f9787 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -73,11 +73,9 @@ Current State uploading the translations did not work, and the script had to be tinkered for the new Transifex API in order to upload it in time for release. The files uploaded were still not in the preferred format because the new version of the Transifex API does not have - all the same features as the previous API version. - - * Running the script to upload translations requires Transifex admin rights, and there - are few people with Admin rights to Transifex and knowledge of the Transifex API. We - should use the solution provided by Transifex to solve this type of problem. + all the same features as the previous API version. There are few people with Admin + rights to Transifex and knowledge of the Transifex API; this could become a reccuring + problem with each Open edX release. Rationale ********* From cfac57dfbe8ab9f8dc2b2047c10789ecb86aa283 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 10:51:12 -0400 Subject: [PATCH 25/49] docs: split Rationale section into two sections --- .../oep-0058-arch-translations-management.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 2639f9787..9ef5145ee 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -77,8 +77,8 @@ Current State rights to Transifex and knowledge of the Transifex API; this could become a reccuring problem with each Open edX release. -Rationale -********* +Rationale for migrating to the Transifex GitHub App +*************************************************** * This is an upgrade of a system we use regularly, but do not want to have to maintain regularly. @@ -88,6 +88,13 @@ Rationale * The Github Transifex App is very simple to configure and has many options. We can set Transifex Projects to automatically upload/download translation files from a repository once the translations are reviewed and accepted. +* By using an app that is maintained by Transifex the organization, we reduce the + maintenance burden and are more future proof of changes they might make since they + maintain both the API and the GitHub App. + +Rationale for consolidating translations files centrally +******************************************************** + * Transifex only allows a one-to-one relationship between repositories and Transifex Projects. Organizing all of the translation files into one repository and one Transifex Project has a lower cost: we pay by the project so we end up paying less, and by @@ -97,9 +104,6 @@ Rationale translations related to Open edX releases can make all interactions with translations very quick and simple due to the ability to clone the branch of a specific release, with translations organized by repository name. -* By using an app that is maintained by Transifex the organization, we reduce the - maintenance burden and are more future proof of changes they might make since they - maintain both the API and the GitHub App. Proposed Implementation *********************** From 8c1a57a864a24532659a3b4da9e7dc56e258bb8c Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 10:54:00 -0400 Subject: [PATCH 26/49] docs: standardize to "translation files" --- .../oep-0058-arch-translations-management.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 9ef5145ee..a0fa5937c 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -112,28 +112,28 @@ Move Translation Files to a New Repo ==================================== Translation files (of types .mo and .po) currently exist amongst the code/documentation -they translate. We will move these Translation files from being amongst the -code/documentation to their own repo. For easier reintegration, Translation files will be +they translate. We will move these translation files from being amongst the +code/documentation to their own repo. For easier reintegration, translation files will be kept in the same directory structure as the code/documentation they translate. Add GitHub Transifex App to openedx Organization ================================================ Transifex's GitHub app will need to be added to the openedx GitHub organization in order -to grant the app permissions to push/pull the Translation files. Currently, we manage the +to grant the app permissions to push/pull the translation files. Currently, we manage the push/pull permissions for the edx-transifex-bot through a number of groups. The Transifex GitHub app is granted permission on a repository basis, and by moving all the -Translations to a single repository we reduce the number of user groups we will have to -manage. +translation files to a single repository we reduce the number of user groups we will have +to manage. Connect the New Translation Repository to Transifex =================================================== The Transifex webapp accepts configuration files for each Transifex project. By -connecting the single repository containing all Translation files, we only need to make a -single configuration file that allows the Transifex GitHub app to manage the Translation +connecting the single repository containing all translation files, we only need to make a +single configuration file that allows the Transifex GitHub app to manage the translation files. Based on the Translation Working Group's instruction, we can set parameters that -automatically push and pull Translation files. +automatically push and pull translation files. Copy Transifex Memory ===================== @@ -141,7 +141,7 @@ Copy Transifex Memory As a last step we can save all the progress the Open edX translators have accomplished by copying the Transifex Memory, the auto-translation feature that allows for Projects with similar strings to be automatically translated, from the old projects to this new one. By -moving all the Translation Files to the same repository we can increase the reach of the +moving all the translation files to the same repository we can increase the reach of the Transifex Memory feature to help translate similar strings across the entire code/documentation base. From 00611212e70603bef8b59f625871eb61f6350a41 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 16:04:50 -0400 Subject: [PATCH 27/49] docs: clarify GH app permissions behavior --- .../oep-0058-arch-translations-management.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index a0fa5937c..12e50ed02 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -119,12 +119,12 @@ kept in the same directory structure as the code/documentation they translate. Add GitHub Transifex App to openedx Organization ================================================ -Transifex's GitHub app will need to be added to the openedx GitHub organization in order -to grant the app permissions to push/pull the translation files. Currently, we manage the -push/pull permissions for the edx-transifex-bot through a number of groups. The Transifex -GitHub app is granted permission on a repository basis, and by moving all the -translation files to a single repository we reduce the number of user groups we will have -to manage. +The Transifex GitHub app will need to be added to the openedx GitHub organization in +order to grant the app permissions to push/pull the translation files. Currently, we +manage the push/pull permissions for the edx-transifex-bot through a number of GitHub +user groups. The Transifex GitHub app once installed in an organization, is granted +permissions to push/pull on a repository basis, and by moving all the translation files +to a single repository we eliminate separate translations user groups. Connect the New Translation Repository to Transifex =================================================== From b9e5abcf82e2eba7714277791670e9d03a3d128a Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 16:20:52 -0400 Subject: [PATCH 28/49] docs: update final implementation step --- .../oep-0058-arch-translations-management.rst | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 12e50ed02..f5ad25bd3 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -135,15 +135,18 @@ single configuration file that allows the Transifex GitHub app to manage the tra files. Based on the Translation Working Group's instruction, we can set parameters that automatically push and pull translation files. -Copy Transifex Memory -===================== +Copy Transifex Memory and Combine Translators +============================================== -As a last step we can save all the progress the Open edX translators have accomplished by -copying the Transifex Memory, the auto-translation feature that allows for Projects with -similar strings to be automatically translated, from the old projects to this new one. By -moving all the translation files to the same repository we can increase the reach of the -Transifex Memory feature to help translate similar strings across the entire -code/documentation base. +As a last step we will reorganize the openedx Transifex organization by combining +translators and reviewers across Transifex projects into the new project associated with +the new repository. In addition, we can save all the progress the Open edX translators +have accomplished by copying the Transifex Memory, the auto-translation feature that +allows for Projects with similar strings to be automatically translated, from the old +projects to this new one. Once older projects are made redundnant by the new project, +they will be deprecated. By moving all the translation files to the same repository we +can increase the reach of the Transifex Memory feature to help translate similar strings +across the entire code/documentation base. Impacts ******* From 59bf579623333ed9171b99989eaf009e1406e51d Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 16:37:50 -0400 Subject: [PATCH 29/49] docs: clarify Translation WG instruction --- .../oep-0058-arch-translations-management.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index f5ad25bd3..9808a7210 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -132,8 +132,9 @@ Connect the New Translation Repository to Transifex The Transifex webapp accepts configuration files for each Transifex project. By connecting the single repository containing all translation files, we only need to make a single configuration file that allows the Transifex GitHub app to manage the translation -files. Based on the Translation Working Group's instruction, we can set parameters that -automatically push and pull translation files. +files. Based on the Translation Working Group's instruction on acceptable +translation/review percentages, we can set parameters that automatically push and pull +translation files. Copy Transifex Memory and Combine Translators ============================================== From b50102069acc3d03a4cf45a52c52691ee4caec26 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 8 Sep 2022 16:53:24 -0400 Subject: [PATCH 30/49] docs: add example to Move Translation Files sect --- .../oep-0058-arch-translations-management.rst | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 9808a7210..877d426c0 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -113,8 +113,15 @@ Move Translation Files to a New Repo Translation files (of types .mo and .po) currently exist amongst the code/documentation they translate. We will move these translation files from being amongst the -code/documentation to their own repo. For easier reintegration, translation files will be -kept in the same directory structure as the code/documentation they translate. +code/documentation to their own repo. For example, a translation file for the openedx +repository edx-platform_ located at ``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` +would be moved to the new repository with the name openedx-translations_ located at +``openedx-translation/edx-platform/conf/locale/en/LC_MESSAGES/django.po``. For easier +reintegration, translation files will be kept in the same directory structure as the +code/documentation they translate. + +.. _edx-platform: https://github.com/openedx/edx-platform +.. _openedx-translations: https://github.com/openedx/openedx-translation Add GitHub Transifex App to openedx Organization ================================================ From c0d275568e51d82cd918733f3b28237b71039a7b Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Fri, 16 Sep 2022 13:28:17 -0400 Subject: [PATCH 31/49] docs: split bullet point and add link to PR Take one large bullet point about the state and break it up into two bullet points that are more understandable. The link added includes more information about the formatting issues during the Nutmeg Release. --- .../oep-0058-arch-translations-management.rst | 24 ++++++++++--------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 877d426c0..f569695eb 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -68,14 +68,16 @@ Current State supposed to do, or whether it is doing it correctly. * The translations for the Open edX Maple Release were never uploaded to Transifex, because the automation handled by the edx-transifex-bot never uploaded it. -* The translations for the Open edX Nutmeg Release were nearly not uploaded because the - edx-transifex-bot did not automatically upload it. In addition, the script for - uploading the translations did not work, and the script had to be tinkered for the new - Transifex API in order to upload it in time for release. The files uploaded were still - not in the preferred format because the new version of the Transifex API does not have - all the same features as the previous API version. There are few people with Admin - rights to Transifex and knowledge of the Transifex API; this could become a reccuring - problem with each Open edX release. +* The underlying library and Transifex API (V2) are being deprecated. This has led to + insconsistent behavior by our tooling when we try to automatically manage translations. + See `this pull request`_ for more details. +* We have a complex process for managing translations for the named releases. As a + result, the black box nature of the transifex bot and the deprecation of the underlying + tooling, this has become more laborious to keep running. Especially because there are + few people with Admin rights to Transifex and knowledge of the Transifex API; this + could become a reccuring problem with each Open edX release. + +.. _this pull request: https://github.com/openedx/edx-platform/pull/30567 Rationale for migrating to the Transifex GitHub App *************************************************** @@ -97,9 +99,9 @@ Rationale for consolidating translations files centrally * Transifex only allows a one-to-one relationship between repositories and Transifex Projects. Organizing all of the translation files into one repository and one Transifex - Project has a lower cost: we pay by the project so we end up paying less, and by - putting all translation files in the same place, it is easier to track translation - progress, and debug translation issues. + Project has a lower labor cost: projects are managed separately so we end up spending + less time tracking translation progress, and debuging translation issues when all + translation files are put in the same place. * A repository that only contains text/binary files, and uses branches to separate translations related to Open edX releases can make all interactions with translations very quick and simple due to the ability to clone the branch of a specific release, From b0a0a06fdf2ebfc70ed49bcebe65daf19671d86a Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Fri, 16 Sep 2022 14:49:50 -0400 Subject: [PATCH 32/49] docs: add translation labor cost --- .../oep-0058-arch-translations-management.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index f569695eb..6040d0914 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -101,7 +101,8 @@ Rationale for consolidating translations files centrally Projects. Organizing all of the translation files into one repository and one Transifex Project has a lower labor cost: projects are managed separately so we end up spending less time tracking translation progress, and debuging translation issues when all - translation files are put in the same place. + translation files are put in the same place. By decreasing the number of projects we + need to maintain, we can add more content like the MFE translations. * A repository that only contains text/binary files, and uses branches to separate translations related to Open edX releases can make all interactions with translations very quick and simple due to the ability to clone the branch of a specific release, From 4587332c320b30bc9ac8b61b40ea2c1eb2df40f2 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Fri, 16 Sep 2022 15:12:22 -0400 Subject: [PATCH 33/49] docs: add new repo translations instructions We will create new instructions for developers on how to enable translations for a new service/repo when it comes online. --- .../oep-0058-arch-translations-management.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 6040d0914..0d19684fe 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -186,7 +186,8 @@ Impact on Developers While it won’t directly impact the day-to-day workflow of developers, due to the same reasons that we impact site operators (new translations location), we will have to update -development tools as well. +development tools as well. In addion, we will create new instructions for developers on +how to enable translations for a new service/repo when it comes online. Locations ********* From 8fcb11e49020443dfaf1744063aa02b3e4537c4f Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Fri, 23 Sep 2022 11:53:05 -0400 Subject: [PATCH 34/49] docs: add Final State section The final state section illustrates how all the systems will interact after the implementation of this OEP. It includes a description of a tool that will enable the correct placement of translation files back into their selected repositories. --- .../oep-0058-arch-translations-management.rst | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 0d19684fe..635e84b7c 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -9,7 +9,7 @@ OEP-58: Translations Management * - Title - Translations Management * - Last Modified - - 2022-08-10 + - 2022-09-23 * - Authors - * Carlos Muniz @@ -189,6 +189,32 @@ reasons that we impact site operators (new translations location), we will have development tools as well. In addion, we will create new instructions for developers on how to enable translations for a new service/repo when it comes online. +Final State +*********** + +Repositories that generate translation files will have their translation files generated +and pushed to the openedx-translation repository via a github workflow. Once the +translation files from edx-platform and other repositories are moved to the +openedx-translations repository, the Transifex GitHub App will link a Transifex project +of a name such as "Open edX Translations" to the openedx-translations repository. A +configuration file naming the files that are to be translated and the trigger that pulls +translation files back into will be created in the openedx/translations repository. This +link will allow for the Transifex GitHub App to automatically manage the push/pull of the +translation files without the need for human intervention. + +When it comes time to cut an Open Release, a new branch will be formed in the +openedx-translations repository for this release. This new branch will correspond with +other release branches such as those found in edx-platform. It is my recommendation that +a tool in the form of a python library is written to enable the placement of the +translation files kept in openedx-translations into the repositories the translation +files are formed from. This tool will manage the placement of translation files through +editable configuration files kept in the repositories that have translation files kept in +openedx-translations. The configuration file will support options that allow for the +concatenation, reorganization, and reformatting of translation files as they are copied +to their locations amongst the code. The configuration file will also support selecting +which languages to be included in an Open edX deployment. The tool will have to be +used/ran as part of the setup of a repository, whether for development or deployment. + Locations ********* From e00b25397e9339daa3724a31ab05de83ba01d640 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 26 Sep 2022 09:54:03 -0400 Subject: [PATCH 35/49] docs: update language to be clearer Using "committed via a pull request" instead of "pushed". --- .../oep-0058-arch-translations-management.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 635e84b7c..93fef8801 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -9,7 +9,7 @@ OEP-58: Translations Management * - Title - Translations Management * - Last Modified - - 2022-09-23 + - 2022-09-26 * - Authors - * Carlos Muniz @@ -193,14 +193,14 @@ Final State *********** Repositories that generate translation files will have their translation files generated -and pushed to the openedx-translation repository via a github workflow. Once the -translation files from edx-platform and other repositories are moved to the -openedx-translations repository, the Transifex GitHub App will link a Transifex project -of a name such as "Open edX Translations" to the openedx-translations repository. A -configuration file naming the files that are to be translated and the trigger that pulls -translation files back into will be created in the openedx/translations repository. This -link will allow for the Transifex GitHub App to automatically manage the push/pull of the -translation files without the need for human intervention. +and committed via a pull request to the openedx-translation repository via a github +workflow. Once the translation files from edx-platform and other repositories are moved +to the openedx-translations repository, the Transifex GitHub App will link a Transifex +project of a name such as "Open edX Translations" to the openedx-translations repository. +A configuration file naming the files that are to be translated and the trigger that +pulls translation files back into will be created in the openedx/translations repository. +This link will allow for the Transifex GitHub App to automatically manage the push/pull +of the translation files without the need for human intervention. When it comes time to cut an Open Release, a new branch will be formed in the openedx-translations repository for this release. This new branch will correspond with From a550fc8fa1aa6ab9002112a858ebfc3f3581a007 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 26 Sep 2022 10:29:07 -0400 Subject: [PATCH 36/49] docs: add Transifex github app & config links Adding links changed the lengths of some of the section titles. In addition more consistent reference to the app has been applied. --- .../oep-0058-arch-translations-management.rst | 98 ++++++++++--------- 1 file changed, 52 insertions(+), 46 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 93fef8801..8e8ae47f3 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -47,12 +47,12 @@ point. Decision ******** -To alleviate these issues, we will switch from using the edx-transifex-bot to the GitHub -Transifex App, a stable app provided by Transifex. Benefits of this change include being -easier to maintain and solving a lot of the pain points detailed below. As part of this -proposal, we suggest moving translations into their own repository, to make using the -GitHub Transifex app more streamlined and straightforward, and in order to make -organizing and using the up to date translations simpler. +To alleviate these issues, we will switch from using the edx-transifex-bot to the +`Transifex GitHub App`_, a stable app provided by Transifex. Benefits of this change +include being easier to maintain and solving a lot of the pain points detailed below. As +part of this proposal, we suggest moving translations into their own repository, to make +using the `Transifex GitHub App`_ more streamlined and straightforward, and in order to +make organizing and using the up to date translations simpler. Current State ************* @@ -79,20 +79,20 @@ Current State .. _this pull request: https://github.com/openedx/edx-platform/pull/30567 -Rationale for migrating to the Transifex GitHub App -*************************************************** +Rationale for migrating to the `Transifex GitHub App`_ +****************************************************** * This is an upgrade of a system we use regularly, but do not want to have to maintain regularly. * Upgrading from a bot (machine user) to an app/workflow is recommended by GitHub and makes the translation process more open source. -* The GitHub Transifex Integration App is developed and supported by Transifex -* The Github Transifex App is very simple to configure and has many options. We can set - Transifex Projects to automatically upload/download translation files from a repository - once the translations are reviewed and accepted. +* The `Transifex GitHub App`_ is developed and supported by Transifex +* The `Transifex GitHub App`_ is very simple to configure and has many options. We can + set Transifex Projects to automatically upload/download translation files from a + repository once the translations are reviewed and accepted. * By using an app that is maintained by Transifex the organization, we reduce the maintenance burden and are more future proof of changes they might make since they - maintain both the API and the GitHub App. + maintain both the API and the `Transifex GitHub App`_. Rationale for consolidating translations files centrally ******************************************************** @@ -126,25 +126,25 @@ code/documentation they translate. .. _edx-platform: https://github.com/openedx/edx-platform .. _openedx-translations: https://github.com/openedx/openedx-translation -Add GitHub Transifex App to openedx Organization -================================================ +Add `Transifex GitHub App`_ to openedx Organization +=================================================== -The Transifex GitHub app will need to be added to the openedx GitHub organization in +The `Transifex GitHub App`_ will need to be added to the openedx GitHub organization in order to grant the app permissions to push/pull the translation files. Currently, we manage the push/pull permissions for the edx-transifex-bot through a number of GitHub -user groups. The Transifex GitHub app once installed in an organization, is granted +user groups. The `Transifex GitHub App`_ once installed in an organization, is granted permissions to push/pull on a repository basis, and by moving all the translation files to a single repository we eliminate separate translations user groups. Connect the New Translation Repository to Transifex =================================================== -The Transifex webapp accepts configuration files for each Transifex project. By -connecting the single repository containing all translation files, we only need to make a -single configuration file that allows the Transifex GitHub app to manage the translation -files. Based on the Translation Working Group's instruction on acceptable -translation/review percentages, we can set parameters that automatically push and pull -translation files. +The Transifex web-app accepts a `Transifex GitHub Integration configuration file`_ for +each Transifex project. By connecting the single repository containing all translation +files, we only need to make a single `Transifex GitHub Integration configuration file`_ +that allows the `Transifex GitHub App`_ to manage the translation files. Based on the +Translation Working Group's instruction on acceptable translation/review percentages, we +can set parameters that automatically push and pull translation files. Copy Transifex Memory and Combine Translators ============================================== @@ -174,8 +174,8 @@ Impact on Site Operators ======================== Currently the translation files for any given service or library is stored at the same -place as the code, which has generally simplified the deployment story in the past. With -this change, the translations files will move to their own repository. As we deprecate +place as the code, which has generally simplified the deployment story in the past. With +this change, the translations files will move to their own repository. As we deprecate the old translations files, the relevant deployment tooling will need to be updated to pull down the translations from the new repository as a part of the deployment process. This will impact both the old Ansible based tooling as well as any new docker based @@ -195,12 +195,13 @@ Final State Repositories that generate translation files will have their translation files generated and committed via a pull request to the openedx-translation repository via a github workflow. Once the translation files from edx-platform and other repositories are moved -to the openedx-translations repository, the Transifex GitHub App will link a Transifex +to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex project of a name such as "Open edX Translations" to the openedx-translations repository. -A configuration file naming the files that are to be translated and the trigger that -pulls translation files back into will be created in the openedx/translations repository. -This link will allow for the Transifex GitHub App to automatically manage the push/pull -of the translation files without the need for human intervention. +A `Transifex GitHub Integration configuration file`_ naming the files that are to be +translated and the trigger that pulls translation files back into will be created in the +openedx/translations repository. This link will allow for the `Transifex GitHub App`_ to +automatically manage the push/pull of the translation files without the need for human +intervention. When it comes time to cut an Open Release, a new branch will be formed in the openedx-translations repository for this release. This new branch will correspond with @@ -208,12 +209,13 @@ other release branches such as those found in edx-platform. It is my recommendat a tool in the form of a python library is written to enable the placement of the translation files kept in openedx-translations into the repositories the translation files are formed from. This tool will manage the placement of translation files through -editable configuration files kept in the repositories that have translation files kept in -openedx-translations. The configuration file will support options that allow for the -concatenation, reorganization, and reformatting of translation files as they are copied -to their locations amongst the code. The configuration file will also support selecting -which languages to be included in an Open edX deployment. The tool will have to be -used/ran as part of the setup of a repository, whether for development or deployment. +an editable translation tool configuration file kept in the repositories that have +translation files kept in openedx-translations. The editable translation tool +configuration file will support options that allow for the concatenation, reorganization, +and reformatting of translation files as they are copied to their locations amongst the +code. The editable translation tool configuration file will also support selecting which +languages to be included in an Open edX deployment. The tool will have to be used/ran as +part of the setup of a repository, whether for development or deployment. Locations ********* @@ -221,10 +223,11 @@ Locations Dumps of the translation/localization files from Transifex for the Open edX Release project already exist in a repository with the name of openedx/openedx-i18n. A new repository named openedx/openedx-translations will be similarly structured, but it will -contain the translation files for all repositories within openedx. The GitHub Transifex -app will be installed in the openedx organization. Similar to how the Build-Test-Release -Working Group creates a new branch for each new named release of edx-platform, -translation releases can also be kept in branches corresponding to edx-platform releases. +contain the translation files for all repositories within openedx. The +`Transifex GitHub App`_ will be installed in the openedx organization. Similar to how the +Build-Test-Release Working Group creates a new branch for each new named release of +edx-platform, translation releases can also be kept in branches corresponding to +edx-platform releases. Rejected Alternatives ********************* @@ -236,7 +239,7 @@ The source code for the edx-transifex-bot is missing. We could rewrite the curre tooling to try to solve the problems encountered in the last two Open edX releases and upgrade to the new API, but this approach would require a full rewrite, potentially more expensive than doing the rewrite in a way that Transifex more cleanly supports. It should -also be mentioned that GitHub discourages the use of bots and separate bot accounts; they +also be mentioned that GitHub discourages the use of bots and separate bot accounts; theyq strongly recommend using GitHub Apps. Making a Transifex Project for Each Repository @@ -246,8 +249,11 @@ As translation support is provided for more repos, the effort to maintain the translations infrastructure increases. A Transifex Project houses the content to be translated and needs to be created before any content can be added for translation. Transifex Projects can only support 1 GitHub repository each and need to be maintained -separately. Maintaining a Transifex Project involves adjusting configurations, adding new -languages, assigning translators to projects, or any other miscellaneous irregular tasks -that would be time-consuming at a larger scale. If we add a Transifex Project, each -Transifex Project will need to be maintained separately, making debugging issues or -tracking the progress of each Transifex Project time-consuming. +separately. Maintaining a Transifex Project involves adjusting configuration files, +adding new languages, assigning translators to projects, or any other miscellaneous +irregular tasks that would be time-consuming at a larger scale. If we add a Transifex +Project, each Transifex Project will need to be maintained separately, making debugging +issues or tracking the progress of each Transifex Project time-consuming. + +.. _Transifex GitHub App: https://github.com/apps/transifex-integration +.. _Transifex GitHub Integration configuration file: https://docs.transifex.com/transifex-github-integrations/github-tx-ui#linking-a-specific-project-with-a-github-repository \ No newline at end of file From f7260860c7a5462c73a4c0de81b53469eaa23de4 Mon Sep 17 00:00:00 2001 From: Ned Batchelder Date: Wed, 28 Sep 2022 14:52:15 -0400 Subject: [PATCH 37/49] fix: spelling errors --- .../oep-0058-arch-translations-management.rst | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 8e8ae47f3..5af4bb939 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -69,13 +69,13 @@ Current State * The translations for the Open edX Maple Release were never uploaded to Transifex, because the automation handled by the edx-transifex-bot never uploaded it. * The underlying library and Transifex API (V2) are being deprecated. This has led to - insconsistent behavior by our tooling when we try to automatically manage translations. + inconsistent behavior by our tooling when we try to automatically manage translations. See `this pull request`_ for more details. * We have a complex process for managing translations for the named releases. As a - result, the black box nature of the transifex bot and the deprecation of the underlying + result, the black box nature of the Transifex bot and the deprecation of the underlying tooling, this has become more laborious to keep running. Especially because there are few people with Admin rights to Transifex and knowledge of the Transifex API; this - could become a reccuring problem with each Open edX release. + could become a recurring problem with each Open edX release. .. _this pull request: https://github.com/openedx/edx-platform/pull/30567 @@ -100,7 +100,7 @@ Rationale for consolidating translations files centrally * Transifex only allows a one-to-one relationship between repositories and Transifex Projects. Organizing all of the translation files into one repository and one Transifex Project has a lower labor cost: projects are managed separately so we end up spending - less time tracking translation progress, and debuging translation issues when all + less time tracking translation progress, and debugging translation issues when all translation files are put in the same place. By decreasing the number of projects we need to maintain, we can add more content like the MFE translations. * A repository that only contains text/binary files, and uses branches to separate @@ -117,7 +117,7 @@ Move Translation Files to a New Repo Translation files (of types .mo and .po) currently exist amongst the code/documentation they translate. We will move these translation files from being amongst the code/documentation to their own repo. For example, a translation file for the openedx -repository edx-platform_ located at ``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` +repository `edx-platform`_ located at ``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` would be moved to the new repository with the name openedx-translations_ located at ``openedx-translation/edx-platform/conf/locale/en/LC_MESSAGES/django.po``. For easier reintegration, translation files will be kept in the same directory structure as the @@ -154,7 +154,7 @@ translators and reviewers across Transifex projects into the new project associa the new repository. In addition, we can save all the progress the Open edX translators have accomplished by copying the Transifex Memory, the auto-translation feature that allows for Projects with similar strings to be automatically translated, from the old -projects to this new one. Once older projects are made redundnant by the new project, +projects to this new one. Once older projects are made redundant by the new project, they will be deprecated. By moving all the translation files to the same repository we can increase the reach of the Transifex Memory feature to help translate similar strings across the entire code/documentation base. @@ -166,7 +166,7 @@ Impact on Translators ===================== As we approach the end of the translation upgrade process, we will need to tactically -move from multiple transifex projects to a single project. This will require +move from multiple Transifex projects to a single project. This will require coordination with our translators to ensure that moving forward they are providing translations in the right place. @@ -186,14 +186,14 @@ Impact on Developers While it won’t directly impact the day-to-day workflow of developers, due to the same reasons that we impact site operators (new translations location), we will have to update -development tools as well. In addion, we will create new instructions for developers on +development tools as well. In addition, we will create new instructions for developers on how to enable translations for a new service/repo when it comes online. Final State *********** Repositories that generate translation files will have their translation files generated -and committed via a pull request to the openedx-translation repository via a github +and committed via a pull request to the openedx-translation repository via a GitHub workflow. Once the translation files from edx-platform and other repositories are moved to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex project of a name such as "Open edX Translations" to the openedx-translations repository. @@ -239,7 +239,7 @@ The source code for the edx-transifex-bot is missing. We could rewrite the curre tooling to try to solve the problems encountered in the last two Open edX releases and upgrade to the new API, but this approach would require a full rewrite, potentially more expensive than doing the rewrite in a way that Transifex more cleanly supports. It should -also be mentioned that GitHub discourages the use of bots and separate bot accounts; theyq +also be mentioned that GitHub discourages the use of bots and separate bot accounts; they strongly recommend using GitHub Apps. Making a Transifex Project for Each Repository @@ -248,7 +248,7 @@ Making a Transifex Project for Each Repository As translation support is provided for more repos, the effort to maintain the translations infrastructure increases. A Transifex Project houses the content to be translated and needs to be created before any content can be added for translation. -Transifex Projects can only support 1 GitHub repository each and need to be maintained +Transifex Projects can only support one GitHub repository each and need to be maintained separately. Maintaining a Transifex Project involves adjusting configuration files, adding new languages, assigning translators to projects, or any other miscellaneous irregular tasks that would be time-consuming at a larger scale. If we add a Transifex @@ -256,4 +256,4 @@ Project, each Transifex Project will need to be maintained separately, making de issues or tracking the progress of each Transifex Project time-consuming. .. _Transifex GitHub App: https://github.com/apps/transifex-integration -.. _Transifex GitHub Integration configuration file: https://docs.transifex.com/transifex-github-integrations/github-tx-ui#linking-a-specific-project-with-a-github-repository \ No newline at end of file +.. _Transifex GitHub Integration configuration file: https://docs.transifex.com/transifex-github-integrations/github-tx-ui#linking-a-specific-project-with-a-github-repository From 9474f40f80f9bd89b8033b999f07a90634503a81 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 29 Sep 2022 14:57:18 -0400 Subject: [PATCH 38/49] docs: add changes based on @sarina's review Distributes the Final State section amongst the implementation section. Comes up with name for tool: openedx-atlas. --- .../oep-0058-arch-translations-management.rst | 54 +++++++++---------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 5af4bb939..4b8ccd981 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -123,6 +123,17 @@ would be moved to the new repository with the name openedx-translations_ located reintegration, translation files will be kept in the same directory structure as the code/documentation they translate. +Repositories that generate translation files will have their translation files generated +and committed via a pull request to the openedx-translation repository via a GitHub +workflow. Once the translation files from edx-platform and other repositories are moved +to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex +project of a name such as "Open edX Translations" to the openedx-translations repository. +A `Transifex GitHub Integration configuration file`_ naming the files that are to be +translated and the trigger that pulls translation files back into will be created in the +openedx/translations repository. This link will allow for the `Transifex GitHub App`_ to +automatically manage the push/pull of the translation files without the need for human +intervention. + .. _edx-platform: https://github.com/openedx/edx-platform .. _openedx-translations: https://github.com/openedx/openedx-translation @@ -159,6 +170,21 @@ they will be deprecated. By moving all the translation files to the same reposit can increase the reach of the Transifex Memory feature to help translate similar strings across the entire code/documentation base. +Get Translations Back for Deployment/Development +================================================ + +A new python library, called openedx-atlas, will be created. This will enable the +placement of the translation files kept in openedx-translations into locally cloned +repositories for development and containers containing the code translation files are +formed from. This tool will manage the placement of translation files through an editable +atlas configuration file (atlas-config.yml) kept in the repositories that have +translation files kept in openedx-translations. The atlas-config.yml file will support +options that allow for the concatenation, reorganization, and reformatting of translation +files as they are copied to their locations amongst the code. The atlas-config.yml file +will also support selecting which languages to be included in an Open edX deployment. The +tool will have to be used/ran as part of the setup of a repository, whether for +development or deployment. + Impacts ******* @@ -189,34 +215,6 @@ reasons that we impact site operators (new translations location), we will have development tools as well. In addition, we will create new instructions for developers on how to enable translations for a new service/repo when it comes online. -Final State -*********** - -Repositories that generate translation files will have their translation files generated -and committed via a pull request to the openedx-translation repository via a GitHub -workflow. Once the translation files from edx-platform and other repositories are moved -to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex -project of a name such as "Open edX Translations" to the openedx-translations repository. -A `Transifex GitHub Integration configuration file`_ naming the files that are to be -translated and the trigger that pulls translation files back into will be created in the -openedx/translations repository. This link will allow for the `Transifex GitHub App`_ to -automatically manage the push/pull of the translation files without the need for human -intervention. - -When it comes time to cut an Open Release, a new branch will be formed in the -openedx-translations repository for this release. This new branch will correspond with -other release branches such as those found in edx-platform. It is my recommendation that -a tool in the form of a python library is written to enable the placement of the -translation files kept in openedx-translations into the repositories the translation -files are formed from. This tool will manage the placement of translation files through -an editable translation tool configuration file kept in the repositories that have -translation files kept in openedx-translations. The editable translation tool -configuration file will support options that allow for the concatenation, reorganization, -and reformatting of translation files as they are copied to their locations amongst the -code. The editable translation tool configuration file will also support selecting which -languages to be included in an Open edX deployment. The tool will have to be used/ran as -part of the setup of a repository, whether for development or deployment. - Locations ********* From e9a2c7039a881eb801a61f441df7e19a6c3fd0f7 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 1 Nov 2022 10:34:16 -0400 Subject: [PATCH 39/49] style: correct formatting --- .../oep-0058-arch-translations-management.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 4b8ccd981..5e5bd3458 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -117,8 +117,9 @@ Move Translation Files to a New Repo Translation files (of types .mo and .po) currently exist amongst the code/documentation they translate. We will move these translation files from being amongst the code/documentation to their own repo. For example, a translation file for the openedx -repository `edx-platform`_ located at ``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` -would be moved to the new repository with the name openedx-translations_ located at +repository `edx-platform`_ located at +``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` would be moved to the new +repository with the name openedx-translations_ located at ``openedx-translation/edx-platform/conf/locale/en/LC_MESSAGES/django.po``. For easier reintegration, translation files will be kept in the same directory structure as the code/documentation they translate. @@ -165,9 +166,9 @@ translators and reviewers across Transifex projects into the new project associa the new repository. In addition, we can save all the progress the Open edX translators have accomplished by copying the Transifex Memory, the auto-translation feature that allows for Projects with similar strings to be automatically translated, from the old -projects to this new one. Once older projects are made redundant by the new project, -they will be deprecated. By moving all the translation files to the same repository we -can increase the reach of the Transifex Memory feature to help translate similar strings +projects to this new one. Once older projects are made redundant by the new project, they +will be deprecated. By moving all the translation files to the same repository we can +increase the reach of the Transifex Memory feature to help translate similar strings across the entire code/documentation base. Get Translations Back for Deployment/Development @@ -192,9 +193,9 @@ Impact on Translators ===================== As we approach the end of the translation upgrade process, we will need to tactically -move from multiple Transifex projects to a single project. This will require -coordination with our translators to ensure that moving forward they are providing -translations in the right place. +move from multiple Transifex projects to a single project. This will require coordination +with our translators to ensure that moving forward they are providing translations in the +right place. Impact on Site Operators ======================== From 17190addc511fa8c0eb71e04f90a311e1fbf57c8 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 1 Nov 2022 16:14:34 -0400 Subject: [PATCH 40/49] docs: update OEP based on prototype and newinfo --- .../oep-0058-arch-translations-management.rst | 59 ++++++++++--------- 1 file changed, 32 insertions(+), 27 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 5e5bd3458..54e5c28ca 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -35,14 +35,17 @@ OEP-58: Translations Management Context ******* -There are problems with the current method of managing translations via the -edx-transifex-bot. The edx-transifex-bot uses a soon to be deprecated (Nov 30, 2022) -version of the Transifex API, and requires admin rights on Transifex in order to -function. In addition, it is difficult to track why some PRs are merged by the bot, and -some are not, and where the bot is creating and merging PRs. Most recently, it was -discovered that the translations were not uploading properly but it has been impossible -to debug exactly why. In the week before the Nutmeg release, this was a significant pain -point. +The current method of managing and organizing translations files is flawed, +semi-unsupported, and confusing. For example: the edx-transifex-bot performs the +automated upload/download of translation files to/from GitHub and Transifex. It uses a +soon to be deprecated (Nov 30, 2022) version of the Transifex API and requires admin +rights on Transifex in order to function. In addition because it runs on legacy +infrastructure originally provided by a community member and is no longer supported, it +is difficult to track why some PRs are merged by the bot, and some are not, and where the +bot is creating and merging PRs. Most recently, it was discovered that the translations +were not uploading properly but it has been impossible for most members of the Open edX +community to debug exactly why. In the week before the Nutmeg release, this was a +significant pain point. Decision ******** @@ -63,20 +66,22 @@ Current State files. At some point, the login to the edx-transifex-bot user was lost, and without access to the scripts that the bot uses to function, this edx-transifex-bot is a security issue we cannot control or debug. -* Edx-transifex-bot is a black box: Login/scripts for edx-transifex-bot cannot be found - on GitHub or on wiki/secrets pages, and it is difficult to observe what work it is - supposed to do, or whether it is doing it correctly. +* edx-transifex-bot is a black box for most of the community: the code for the + edx-transifex-bot is in the `ecommerce-scripts`_ repository but it is impossible for + most of the community to observe the work it is doing, or whether it is doing it + correctly. * The translations for the Open edX Maple Release were never uploaded to Transifex, because the automation handled by the edx-transifex-bot never uploaded it. * The underlying library and Transifex API (V2) are being deprecated. This has led to inconsistent behavior by our tooling when we try to automatically manage translations. See `this pull request`_ for more details. * We have a complex process for managing translations for the named releases. As a - result, the black box nature of the Transifex bot and the deprecation of the underlying - tooling, this has become more laborious to keep running. Especially because there are - few people with Admin rights to Transifex and knowledge of the Transifex API; this - could become a recurring problem with each Open edX release. + result, the black box nature of the edx-transifex-bot and the deprecation of the + underlying tooling, this has become more laborious to keep running. Especially because + there are few people with Admin rights to Transifex and knowledge of the Transifex API; + this could become a recurring problem with each Open edX release. +.. _ecommerce-scripts: https://github.com/openedx/ecommerce-scripts/tree/master/transifex .. _this pull request: https://github.com/openedx/edx-platform/pull/30567 Rationale for migrating to the `Transifex GitHub App`_ @@ -105,8 +110,8 @@ Rationale for consolidating translations files centrally need to maintain, we can add more content like the MFE translations. * A repository that only contains text/binary files, and uses branches to separate translations related to Open edX releases can make all interactions with translations - very quick and simple due to the ability to clone the branch of a specific release, - with translations organized by repository name. + very quick and simple due to the ability to clone and sparse-checkout the branch of a + specific release and the directory (repository name) with translation files. Proposed Implementation *********************** @@ -128,7 +133,7 @@ Repositories that generate translation files will have their translation files g and committed via a pull request to the openedx-translation repository via a GitHub workflow. Once the translation files from edx-platform and other repositories are moved to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex -project of a name such as "Open edX Translations" to the openedx-translations repository. +project of a name such as "openedx-translations" to the openedx-translations repository. A `Transifex GitHub Integration configuration file`_ naming the files that are to be translated and the trigger that pulls translation files back into will be created in the openedx/translations repository. This link will allow for the `Transifex GitHub App`_ to @@ -178,10 +183,10 @@ A new python library, called openedx-atlas, will be created. This will enable th placement of the translation files kept in openedx-translations into locally cloned repositories for development and containers containing the code translation files are formed from. This tool will manage the placement of translation files through an editable -atlas configuration file (atlas-config.yml) kept in the repositories that have -translation files kept in openedx-translations. The atlas-config.yml file will support +atlas configuration file (atlas.yml) kept in the repositories that have +translation files kept in openedx-translations. The atlas.yml file will support options that allow for the concatenation, reorganization, and reformatting of translation -files as they are copied to their locations amongst the code. The atlas-config.yml file +files as they are copied to their locations amongst the code. The atlas.yml file will also support selecting which languages to be included in an Open edX deployment. The tool will have to be used/ran as part of the setup of a repository, whether for development or deployment. @@ -234,12 +239,12 @@ Rejected Alternatives Rewriting the Current Tooling for the New API ============================================= -The source code for the edx-transifex-bot is missing. We could rewrite the current -tooling to try to solve the problems encountered in the last two Open edX releases and -upgrade to the new API, but this approach would require a full rewrite, potentially more -expensive than doing the rewrite in a way that Transifex more cleanly supports. It should -also be mentioned that GitHub discourages the use of bots and separate bot accounts; they -strongly recommend using GitHub Apps. +The source code for the edx-transifex-bot can be found in `ecommerce-scripts`_. We could +rewrite the current tooling to try to solve the problems encountered in the last two Open +edX releases and upgrade to the new API, but this approach is a patch-up job that will +not address several other issues mentioned and would have to be undertaken by the +community member with exclusive access to the legacy infrastructure currently running the +edx-transifex-bot. Making a Transifex Project for Each Repository ============================================== From c83942425193bf7c569c6b1781af0cc4b304d51a Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Thu, 3 Nov 2022 14:10:05 -0400 Subject: [PATCH 41/49] fix: remove most mentions of transifex-client deprecation Ecommerce-scripts/transifex was updated to remove transifex-client as a dependency in April 2022. While this still caused issues during Maple and Nutmeg, it will no longer be an issue. --- .../oep-0058-arch-translations-management.rst | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 54e5c28ca..5518f78cb 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -35,17 +35,15 @@ OEP-58: Translations Management Context ******* -The current method of managing and organizing translations files is flawed, -semi-unsupported, and confusing. For example: the edx-transifex-bot performs the -automated upload/download of translation files to/from GitHub and Transifex. It uses a -soon to be deprecated (Nov 30, 2022) version of the Transifex API and requires admin -rights on Transifex in order to function. In addition because it runs on legacy -infrastructure originally provided by a community member and is no longer supported, it -is difficult to track why some PRs are merged by the bot, and some are not, and where the -bot is creating and merging PRs. Most recently, it was discovered that the translations -were not uploading properly but it has been impossible for most members of the Open edX -community to debug exactly why. In the week before the Nutmeg release, this was a -significant pain point. +The current method of managing and organizing translations files is overly complicated +and unavailable to the majority of the Open edX community. For example: the +edx-transifex-bot performs the automated upload of english translation source files to +Transifex and download of translation files to GitHub. It currently runs on legacy +infrastructure originally provided by a community member and is difficult to track why +some PRs are merged by the bot, and some are not, and where the bot is creating and +merging PRs. Most recently, it was discovered that the translations were not uploading +properly but it has been impossible for most members of the Open edX community to debug +exactly why. In the week before the Nutmeg release, this was a significant pain point. Decision ******** @@ -64,19 +62,20 @@ Current State rights on Transifex in order to function. Admin rights give access to private/sensitive information as well as the ability to permanently delete translation and configuration files. At some point, the login to the edx-transifex-bot user was lost, and without - access to the scripts that the bot uses to function, this edx-transifex-bot is a - security issue we cannot control or debug. + access to infrastructure that the bot uses to function, this edx-transifex-bot is a + security issue most of the Open edX community cannot control or debug. * edx-transifex-bot is a black box for most of the community: the code for the edx-transifex-bot is in the `ecommerce-scripts`_ repository but it is impossible for most of the community to observe the work it is doing, or whether it is doing it - correctly. + correctly. In addition, there is no documentation for these important scripts. * The translations for the Open edX Maple Release were never uploaded to Transifex, because the automation handled by the edx-transifex-bot never uploaded it. -* The underlying library and Transifex API (V2) are being deprecated. This has led to - inconsistent behavior by our tooling when we try to automatically manage translations. - See `this pull request`_ for more details. +* The underlying transifex-client library and Transifex API (V2) are being deprecated on + November 30th, 2022. Prior to removing the transifex-client as a dependency, this led + to inconsistent behavior by our tooling when we try to automatically manage + translations. See `this pull request`_ for more details. * We have a complex process for managing translations for the named releases. As a - result, the black box nature of the edx-transifex-bot and the deprecation of the + result, the black box nature of the edx-transifex-bot and the unclear state of the underlying tooling, this has become more laborious to keep running. Especially because there are few people with Admin rights to Transifex and knowledge of the Transifex API; this could become a recurring problem with each Open edX release. From fc07406ecfd9930b136b3119cd4fddabcce549ef Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 8 Nov 2022 13:17:18 -0500 Subject: [PATCH 42/49] fix: update OEP-58 based on suggestion Co-authored-by: Jillian --- .../oep-0058-arch-translations-management.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 5518f78cb..037981414 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -229,7 +229,7 @@ repository named openedx/openedx-translations will be similarly structured, but contain the translation files for all repositories within openedx. The `Transifex GitHub App`_ will be installed in the openedx organization. Similar to how the Build-Test-Release Working Group creates a new branch for each new named release of -edx-platform, translation releases can also be kept in branches corresponding to +edx-platform, translation releases will also be kept in branches corresponding to edx-platform releases. Rejected Alternatives From 3b40a066395e67100ffe2f0837d9b8b87df9202c Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 7 Nov 2022 14:17:15 -0500 Subject: [PATCH 43/49] feat: add review period --- .../oep-0058-arch-translations-management.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 037981414..3b86b2fb7 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -9,7 +9,7 @@ OEP-58: Translations Management * - Title - Translations Management * - Last Modified - - 2022-09-26 + - 2022-11-07 * - Authors - * Carlos Muniz @@ -24,7 +24,7 @@ OEP-58: Translations Management * - Created - 2022-08-08 * - Review Period - - TBD + - 2022-11-07 - 2022-11-21 .. * - Resolution .. - From 1d27a59028f8ee3d4c3bf69dbb9a3e936ad0a260 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 8 Nov 2022 16:44:23 -0500 Subject: [PATCH 44/49] fix: change Transifex Memory to Translation Memory --- .../oep-0058-arch-translations-management.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 3b86b2fb7..7190a59d3 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -162,18 +162,18 @@ that allows the `Transifex GitHub App`_ to manage the translation files. Based o Translation Working Group's instruction on acceptable translation/review percentages, we can set parameters that automatically push and pull translation files. -Copy Transifex Memory and Combine Translators -============================================== +Copy Transifex's Translation Memory and Combine Translators +=========================================================== As a last step we will reorganize the openedx Transifex organization by combining translators and reviewers across Transifex projects into the new project associated with the new repository. In addition, we can save all the progress the Open edX translators -have accomplished by copying the Transifex Memory, the auto-translation feature that -allows for Projects with similar strings to be automatically translated, from the old -projects to this new one. Once older projects are made redundant by the new project, they -will be deprecated. By moving all the translation files to the same repository we can -increase the reach of the Transifex Memory feature to help translate similar strings -across the entire code/documentation base. +have accomplished by copying the Transifex's Translation Memory, the auto-translation +feature that allows for Projects with similar strings to be automatically translated, +from the old projects to this new one. Once older projects are made redundant by the new +project, they will be deprecated. By moving all the translation files to the same +repository we can increase the reach of the Transifex's Translation Memory feature to +help translate similar strings across the entire code/documentation base. Get Translations Back for Deployment/Development ================================================ From 620e6b324e0ce8907bfe1daa9f1217522dee29da Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 8 Nov 2022 16:56:37 -0500 Subject: [PATCH 45/49] fix: add clarification on impact on developers --- .../oep-0058-arch-translations-management.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 7190a59d3..bc8a02dfa 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -215,10 +215,11 @@ tooling. Impact on Developers ==================== -While it won’t directly impact the day-to-day workflow of developers, due to the same -reasons that we impact site operators (new translations location), we will have to update -development tools as well. In addition, we will create new instructions for developers on -how to enable translations for a new service/repo when it comes online. +While it won’t directly impact the day-to-day workflow of developers (unless you are +developing or testing with translation files), due to the same reasons that we impact +site operators (new translations location), we will have to update development tools as +well. In addition, we will create new instructions for developers on how to enable +translations for a new service/repo when it comes online. Locations ********* From 7a06a162f144ec2408ec1af0a44f0fb905adc669 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 9 Nov 2022 13:55:11 -0500 Subject: [PATCH 46/49] fix: update based on suggestion --- .../oep-0058-arch-translations-management.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index bc8a02dfa..cc6c7c18e 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -37,7 +37,7 @@ Context The current method of managing and organizing translations files is overly complicated and unavailable to the majority of the Open edX community. For example: the -edx-transifex-bot performs the automated upload of english translation source files to +edx-transifex-bot performs the automated upload of English translation source files to Transifex and download of translation files to GitHub. It currently runs on legacy infrastructure originally provided by a community member and is difficult to track why some PRs are merged by the bot, and some are not, and where the bot is creating and @@ -58,7 +58,7 @@ make organizing and using the up to date translations simpler. Current State ************* -* Edx-transifex-bot is a potential security issue: The edx-transifex-bot requires admin +* edx-transifex-bot is a potential security issue: The edx-transifex-bot requires admin rights on Transifex in order to function. Admin rights give access to private/sensitive information as well as the ability to permanently delete translation and configuration files. At some point, the login to the edx-transifex-bot user was lost, and without @@ -118,8 +118,8 @@ Proposed Implementation Move Translation Files to a New Repo ==================================== -Translation files (of types .mo and .po) currently exist amongst the code/documentation -they translate. We will move these translation files from being amongst the +Translation files (of types .mo, .po, and .json) currently exist amongst the code and +documentation they translate. We will move these translation files from being amongst the code/documentation to their own repo. For example, a translation file for the openedx repository `edx-platform`_ located at ``edx-platform/conf/locale/en/LC_MESSAGES/django.po`` would be moved to the new @@ -140,7 +140,7 @@ automatically manage the push/pull of the translation files without the need for intervention. .. _edx-platform: https://github.com/openedx/edx-platform -.. _openedx-translations: https://github.com/openedx/openedx-translation +.. _openedx-translations: https://github.com/openedx/openedx-translations Add `Transifex GitHub App`_ to openedx Organization =================================================== @@ -209,7 +209,7 @@ place as the code, which has generally simplified the deployment story in the pa this change, the translations files will move to their own repository. As we deprecate the old translations files, the relevant deployment tooling will need to be updated to pull down the translations from the new repository as a part of the deployment process. -This will impact both the old Ansible based tooling as well as any new docker based +This will impact both the old Ansible based tooling as well as any new Docker based tooling. Impact on Developers From e86e67da63cad9659a14464b0617f97ec2062cc2 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Mon, 14 Nov 2022 12:28:45 -0500 Subject: [PATCH 47/49] feat: add Follow-up Work link This signifes that after the review period, the status will change to Provisional and questions related to the implementation of this OEP-58 in order to reach the Accepted status will be answered in the Follow-up Work page linked here. --- .../oep-0058-arch-translations-management.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index cc6c7c18e..95fe6e981 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -25,6 +25,8 @@ OEP-58: Translations Management - 2022-08-08 * - Review Period - 2022-11-07 - 2022-11-21 + * - References + - `Follow-up Work `_ .. * - Resolution .. - From 4fbff9e96e4aea17d00489bc565972a0ebb36d8e Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Wed, 16 Nov 2022 09:42:49 -0500 Subject: [PATCH 48/49] fix: apply suggestions from review --- .../oep-0058-arch-translations-management.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 95fe6e981..11eb46d0b 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -18,7 +18,7 @@ OEP-58: Translations Management * - Arbiter - Ned Batchelder * - Status - - Under Review + - Under Review (=> Provisional) * - Type - Architecture Decision * - Created @@ -52,10 +52,10 @@ Decision To alleviate these issues, we will switch from using the edx-transifex-bot to the `Transifex GitHub App`_, a stable app provided by Transifex. Benefits of this change -include being easier to maintain and solving a lot of the pain points detailed below. As -part of this proposal, we suggest moving translations into their own repository, to make -using the `Transifex GitHub App`_ more streamlined and straightforward, and in order to -make organizing and using the up to date translations simpler. +include being easier to maintain and solving a lot of the pain points detailed below. In +addition, translation files will be moved into their own repository. This will make using +the `Transifex GitHub App`_ more streamlined and straightforward, and facilitate the +organization of translation files on Transifex. Current State ************* From eeaf460790333de0c5f1601b43a722d41a0989b4 Mon Sep 17 00:00:00 2001 From: Carlos Muniz Date: Tue, 22 Nov 2022 11:55:37 -0500 Subject: [PATCH 49/49] feat: add notes from threads & fix structure With the conclusion of the review period, conversation threads have been summarized and added where necessary to provide more clarity. In addition, the structure of the OEP has been reorganized to be mapped into the normal OEP template. --- .../oep-0058-arch-translations-management.rst | 125 ++++++++++-------- 1 file changed, 67 insertions(+), 58 deletions(-) diff --git a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst index 11eb46d0b..69d8becf1 100644 --- a/oeps/architectural-decisions/oep-0058-arch-translations-management.rst +++ b/oeps/architectural-decisions/oep-0058-arch-translations-management.rst @@ -9,7 +9,7 @@ OEP-58: Translations Management * - Title - Translations Management * - Last Modified - - 2022-11-07 + - 2022-11-22 * - Authors - * Carlos Muniz @@ -18,7 +18,7 @@ OEP-58: Translations Management * - Arbiter - Ned Batchelder * - Status - - Under Review (=> Provisional) + - Provisional * - Type - Architecture Decision * - Created @@ -27,15 +27,24 @@ OEP-58: Translations Management - 2022-11-07 - 2022-11-21 * - References - `Follow-up Work `_ -.. * - Resolution -.. - .. contents:: :local: :depth: 1 -Context -******* +Abstract +******** + +We will switch from using the edx-transifex-bot to the `Transifex GitHub App`_, a stable +app provided by Transifex. Benefits of this change include being easier to maintain and +solving a lot of the pain points detailed below. In addition, translation files will be +moved into their own repository. This will make using the `Transifex GitHub App`_ more +streamlined and straightforward, and facilitate the organization of translation files on +Transifex. A CLI tool will be developed to move translation files from the repository +that contains them to where they are needed for development or deployment. + +Motivation +********** The current method of managing and organizing translations files is overly complicated and unavailable to the majority of the Open edX community. For example: the @@ -47,18 +56,8 @@ merging PRs. Most recently, it was discovered that the translations were not upl properly but it has been impossible for most members of the Open edX community to debug exactly why. In the week before the Nutmeg release, this was a significant pain point. -Decision -******** - -To alleviate these issues, we will switch from using the edx-transifex-bot to the -`Transifex GitHub App`_, a stable app provided by Transifex. Benefits of this change -include being easier to maintain and solving a lot of the pain points detailed below. In -addition, translation files will be moved into their own repository. This will make using -the `Transifex GitHub App`_ more streamlined and straightforward, and facilitate the -organization of translation files on Transifex. - Current State -************* +============= * edx-transifex-bot is a potential security issue: The edx-transifex-bot requires admin rights on Transifex in order to function. Admin rights give access to private/sensitive @@ -82,11 +81,8 @@ Current State there are few people with Admin rights to Transifex and knowledge of the Transifex API; this could become a recurring problem with each Open edX release. -.. _ecommerce-scripts: https://github.com/openedx/ecommerce-scripts/tree/master/transifex -.. _this pull request: https://github.com/openedx/edx-platform/pull/30567 - Rationale for migrating to the `Transifex GitHub App`_ -****************************************************** +====================================================== * This is an upgrade of a system we use regularly, but do not want to have to maintain regularly. @@ -99,9 +95,11 @@ Rationale for migrating to the `Transifex GitHub App`_ * By using an app that is maintained by Transifex the organization, we reduce the maintenance burden and are more future proof of changes they might make since they maintain both the API and the `Transifex GitHub App`_. +* Transifex has a very robust notification system, and errors occurring there will notify + the Transifex Admins and the Translation Working Group. Rationale for consolidating translations files centrally -******************************************************** +======================================================== * Transifex only allows a one-to-one relationship between repositories and Transifex Projects. Organizing all of the translation files into one repository and one Transifex @@ -114,8 +112,8 @@ Rationale for consolidating translations files centrally very quick and simple due to the ability to clone and sparse-checkout the branch of a specific release and the directory (repository name) with translation files. -Proposed Implementation -*********************** +Specification +************* Move Translation Files to a New Repo ==================================== @@ -133,17 +131,14 @@ code/documentation they translate. Repositories that generate translation files will have their translation files generated and committed via a pull request to the openedx-translation repository via a GitHub workflow. Once the translation files from edx-platform and other repositories are moved -to the openedx-translations repository, the `Transifex GitHub App`_ will link a Transifex -project of a name such as "openedx-translations" to the openedx-translations repository. +to the `openedx-translations`_ repository, the `Transifex GitHub App`_ will link a Transifex +project of a name such as "openedx-translations" to the `openedx-translations`_ repository. A `Transifex GitHub Integration configuration file`_ naming the files that are to be translated and the trigger that pulls translation files back into will be created in the openedx/translations repository. This link will allow for the `Transifex GitHub App`_ to automatically manage the push/pull of the translation files without the need for human intervention. -.. _edx-platform: https://github.com/openedx/edx-platform -.. _openedx-translations: https://github.com/openedx/openedx-translations - Add `Transifex GitHub App`_ to openedx Organization =================================================== @@ -180,20 +175,22 @@ help translate similar strings across the entire code/documentation base. Get Translations Back for Deployment/Development ================================================ -A new python library, called openedx-atlas, will be created. This will enable the -placement of the translation files kept in openedx-translations into locally cloned -repositories for development and containers containing the code translation files are -formed from. This tool will manage the placement of translation files through an editable -atlas configuration file (atlas.yml) kept in the repositories that have -translation files kept in openedx-translations. The atlas.yml file will support -options that allow for the concatenation, reorganization, and reformatting of translation -files as they are copied to their locations amongst the code. The atlas.yml file -will also support selecting which languages to be included in an Open edX deployment. The -tool will have to be used/ran as part of the setup of a repository, whether for -development or deployment. - -Impacts -******* +A new CLI tool called `openedx-atlas`_ will be created to enable the placement of the +translation files kept in `openedx-translations`_ into locally cloned repositories for +development and containers containing the code translation files are formed from. This +tool will manage the placement of translation files through an editable atlas +configuration file (atlas.yml) kept in the repositories that have translation files kept +in openedx-translations. The atlas.yml file will support options that allow for the +concatenation, reorganization, and reformatting of translation files as they are copied +to their locations amongst the code. The atlas.yml file will also support selecting which +languages to be included in an Open edX deployment. The tool will have to be used/ran as +part of the setup of a repository, whether for development or deployment. The +`openedx-atlas`_ tool can also be run without configuration files through CLI parameters +that override atlas.yml. This tool is still in development, and while the language the +tool is written in may change, the commands and purpose will not change. + +Impact +****** Impact on Translators ===================== @@ -221,19 +218,23 @@ While it won’t directly impact the day-to-day workflow of developers (unless y developing or testing with translation files), due to the same reasons that we impact site operators (new translations location), we will have to update development tools as well. In addition, we will create new instructions for developers on how to enable -translations for a new service/repo when it comes online. +translations for a new service/repo when it comes online. Information about the use of +the `openedx-atlas`_ CLI tool will also be added to READMEs and Makefiles as necessary. Locations -********* +========= + +Dumps of the translation/localization files from Transifex for the Open edX Releases +already exist in the repository `openedx-i18n`_. A new repository named +`openedx-translations`_ will have a similar role, but it will contain the translation +files for all languages and for all repositories within the GitHub openedx organization. +This OEP will deprecate `openedx-i18n`_ since it will contain strings for all languages, +repositories, and will follow the regular minor/major release schedule. -Dumps of the translation/localization files from Transifex for the Open edX Release -project already exist in a repository with the name of openedx/openedx-i18n. A new -repository named openedx/openedx-translations will be similarly structured, but it will -contain the translation files for all repositories within openedx. The -`Transifex GitHub App`_ will be installed in the openedx organization. Similar to how the -Build-Test-Release Working Group creates a new branch for each new named release of -edx-platform, translation releases will also be kept in branches corresponding to -edx-platform releases. +The `Transifex GitHub App`_ will be installed in the openedx organization. +Similar to how the Build-Test-Release Working Group creates a new branch for each new +named release of edx-platform, translation releases will also be kept in branches +corresponding to edx-platform releases. Rejected Alternatives ********************* @@ -252,14 +253,22 @@ Making a Transifex Project for Each Repository ============================================== As translation support is provided for more repos, the effort to maintain the -translations infrastructure increases. A Transifex Project houses the content to be +translations infrastructure increases. A Transifex Project contains the content to be translated and needs to be created before any content can be added for translation. Transifex Projects can only support one GitHub repository each and need to be maintained separately. Maintaining a Transifex Project involves adjusting configuration files, -adding new languages, assigning translators to projects, or any other miscellaneous -irregular tasks that would be time-consuming at a larger scale. If we add a Transifex -Project, each Transifex Project will need to be maintained separately, making debugging -issues or tracking the progress of each Transifex Project time-consuming. +adding new languages, assigning translators to projects, as well as any other +miscellaneous irregular tasks that would be time-consuming at a larger scale. If we add a +Transifex Project, each Transifex Project will need to be maintained separately, making +debugging issues or tracking the progress of each Transifex Project time-consuming. In +addition, the Transifex editor does not support editing strings across multiple projects +making it extremely time consuming for users to translate strings from many projects. +.. _ecommerce-scripts: https://github.com/openedx/ecommerce-scripts/tree/master/transifex +.. _edx-platform: https://github.com/openedx/edx-platform +.. _openedx-atlas: https://github.com/openedx/openedx-atlas +.. _openedx-i18n: https://github.com/openedx/openedx-i18n +.. _openedx-translations: https://github.com/openedx/openedx-translations +.. _this pull request: https://github.com/openedx/edx-platform/pull/30567 .. _Transifex GitHub App: https://github.com/apps/transifex-integration .. _Transifex GitHub Integration configuration file: https://docs.transifex.com/transifex-github-integrations/github-tx-ui#linking-a-specific-project-with-a-github-repository