Tool: translation compiler

[ivan-mogilko]

As a step of decoupling game compilation procedure from the Editor, we need a stand-alone tool that is run from the command-line, parses a translation source (TRS) file and writes a compiled translation (TRA) file in binary format.
Should be written in the similar line with the existing tools, in C++ (#1262, #1264, #1269).

Name suggestion: trac (TRAnslation Compiler).

NOTE: this task is exclusively in writing this tool, adjusting Editor is a separate task, so no need to concern yourself with that if you are doing this. In fact it's best to assume that Editor will not be present to ensure tool's work result has no reliance on it.

Input:

• a TRS file (AGS translation source);
• path to output file;

Output

• a TRA file (AGS compiled translation)

---

Details

The algorithm is relatively simple, most of TRS is just pairs of text lines that form a key/value map, with a few exceptions (comments, options).
Current implementation of reading and writing TRS in the Editor may be found here:
https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Types/Translation.cs
Implementation of the TRA writer in the Editor may be found here: https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Editor/Components/TranslationsComponent.cs#L92
User documentation of TRS format:
https://github.com/adventuregamestudio/ags-manual/wiki/Translations

The existing Tool code in the current master branch:
https://github.com/adventuregamestudio/ags/tree/master/Tools
MSVS solution for standalone tools:
https://github.com/adventuregamestudio/ags/blob/master/Solutions/Tools.sln

[fernewelten]

In my opinion, we should take this project as an opportunity to go standard:

• If AGS uses a standard format for translatable strings then translators can profit from ready-made tools and ecosystems to handle them.
• Standard formats provide annotations for strings, e.g. to say where they were pulled from so that translators get better context when deciding how they should be translated. We might make use of that.
• Standard formats are more robust against involuntary breaking. They usually have some sort of rudimentary markup. So if something becomes garbled then the reading module can point out just where it is garbled. In contrast, if a spurious line break gets inserted into our current translation files, then a human must check the all the lines one-by-one to find out just where the source/translation alternation has gone out of sync.
• Professional translators can translate standard files in their accustomed translation environment, which also means that they can more easily build on pre-existing work.

[fernewelten]

• Many open-source projects use the GNU "gettext" (PO) format, which has the advantage that you don't need tools for translating the files: A normal text editor is entirely sufficient.

  It's common, but it isn't formally standardized, so the GNU documentation is authoritative: https://www.gnu.org/software/gettext/manual/html_node/index.html

• All the professional general translating environments (SDL Trados, memoq, Wordfast, swordfish, OmegaT etc. etc.) can read from and write into an XML based standard called XLIFF . It's become a universal standard for exchanging translatable files. However, from the vantage point of end users its disadvantage is that you need a tool to process the files. If you edit the files per hand, you run at least as much risk of mucking up everything as with our current .tra/.trs files.

  Standardized here: http://docs.oasis-open.org/xliff/xliff-core/v2.1/xliff-core-v2.1.html

[ivan-mogilko]

@fernewelten this particular task is only to separate existing logic from the editor so that it could be run separately, that's it really. It's also meant for ags3 branch too and work with existing project sources.

To elaborate, this is a part of parent task of decoupling certain functions from the Editor to be able to run game compilation without GUI and .NET.

Any changes to source format should be a separate feature task, and considered whether it's best to add to ags3 or ags4.

Of course the separation of this procedure from the editor may make it easier to transition to the new format.

[ivan-mogilko]

I think there were couple of more tickets related to translation system (unfortunately we do not have a label for these) , perhaps it makes sense reviewing them and writing a summary ticket with problems & proposals.
EDIT: ah, I was wrong, either these were discussed on forums only, or in comments to other tickets, but I cannot find what I expected in our tracker.

[fernewelten]

I'd suggest:

• changing the Editor to export into PO format (exclusively)
• writing trac in such a way that accepts PO files (exclusively) and generates .TRS files
• writing a separate .TRA-to-PO (sic!) converter for legacy support.

In one go, i.e., as a single, unified task.

[ivan-mogilko]

I cannot do this within this task, because this is a part of parent problem of decoupling compilation procedure from the editor. This does not assume any change in formats and has to maintain full compatibility with the existing 3.* projects and 3.* editor.

Any further changes has to be a separate task, as these involve larger changes to all parts of the project: editor, tools, and maybe engine too (I am not sure what will be engine's data from your explanation).
It probably also requires doing a good research first, as I have no idea about PO or translation formats in general.

[fernewelten]

The .TRS format wouldn't (and shouldn't) change in any way.

A .PO file isn't anything spectacular at all. To quote from the docs:

A PO file is made up of many entries, each entry holding the relation between an original untranslated string and its corresponding translation. ... One PO file entry has the following schematic structure [white space and lines starting with '#' are optional]:

white-space
#  translator-comments
#. extracted-comments
#: reference…
#, flag…
#| msgid "previous-untranslated-string"
msgid "untranslated-string"
msgstr "translated-string"

That is, from the vantage point of the translator:

• Look for a line that leads with msgid and translate it on the next line leading with msgstr.
• Put any notes you might have about this specific translation in front of this pair, with a leading # .
• Lines starting with #. might give you context for the translation.
• Ignore all other lines that lead with #.
• Repeat until you are done.

From the vantage point of the generating program:

• If you have human-readable comments that show where the original is from, put them after #..
• If you have any technical references and such, put them after #:.
• Generate a line msgid "ORIGINAL"
• Generate a line msgstr "" for the translator to fill in.
• Repeat until you are done.

From the vantage point of the reading program (in the simplest case):

• Ignore all white lines and all lines beginning with #.
• Find a line that begins with msgid .
• The following non-white line must begin with msgstr (or else, complain about it).
• Do your thing with the pair you found.
• Repeat until you are done.

[ivan-mogilko]

@fernewelten i'm sorry, but I will make this tool as planned in any case for ags3 branch, because the plan was to fist of all have a set of tools working with existing projects.

There's another reason, I believe we are rather close to achieving this milestone (stand-alone tool chain for the game compilation), and I won't like to distract from the set path right now.

If you think PO format is beneficial then we may open a new task for doing what's necessary to transit there. Perhaps it will be aimed at ags4 branch, but this may be argued to be done in ags3 too.

The .TRS format wouldn't (and shouldn't) change in any way.

I'm not sure if I understand... could be that you mistake TRS and TRA? TRS is the current translation source (basically a txt file), TRA is a compiled data for the engine. From your explanation I assumed you meant replacing TRS with PO. Or was I wrong perhaps?