Multi-language/I18n support

[proddy]

Support multi-locale for Telnet console commands, MQTT, SysLog messages and the Web. Since there is not enough memory to hold all the translations it will have to be different builds (DE, NL, UK). String literals will be stored in translation files and built at compile time.

@MichaelDvP @bbqkees what you vote for this feature?

[bbqkees]

Most Gateways go to German speaking countries so it makes perfect sense to provide the interface in their own language. I get some questions about it regularly.
If there is not enough memory to store everything, we should at least provide a language selector at the first install which will then download and install the new firmware.
So we could just have the UK build as default with a selector to install another language.

[MichaelDvP]

Yes, that would be nice. What about a changeable language-file in the spiffs? There is a lot of space left and you have only one software.

[proddy]

That's a good idea. Store the localizations in SPIFFS and download to PROGMEM when EMS-ESP boots.

[proddy]

In order of importance

• Web UI
• MQTT topics and payload
• Console commands (serial and telnet)
• Logging information (to console & syslog)

For the web I had a look at react-intl, i18next-react, polyglot and this one called lingui-js but they'll all quite big libraries and there is just not enough progmem to store all the translations to have it real-time on the ESP8266. react-intl is the right one here (14k zipped).

[proddy]

First localized version of the UI is looking promising.

[proddy]

with the extra 160K on the ESP32 I can easily store the strings in SPIFFS and load them dynamically

[proddy]

Going to look at this again.

[proddy]

started working on this again. It's easier now since we moved React to use hooks. I have selectable languages for EN and DE and starting to translate the UI. I've been thinking how to store the translations and ideally, it would be great if this was in a custom JSON file the user can upload and replace, but for now I'm hardcoding it.

@MichaelDvP one thing that struct me is the amount of memory we have left on the ESP32. I remember the countless long weeks I used to spend optimizing every heap byte on the ESP8266 with version 2 so it wouldn't run out of memory and crash. On the ESP32 we have still 250KB of RAM and Flash both available so I was wondering what difference it would make if we ran the web pages not in PROGMEM and also all the F_() text literals in normal Memory instead of Flash. It may be faster ! I'll run some tests and see.....

[proddy]

The web code is 300KB so needs to stay in flash, but its cached by the browser anyway.

[proddy]

I finished the coding on this. You can switch real-time between UK and DE and the Web UI and Device Entities change. Problem is I've run out of Flash memory so now need to go back to the drawing board to see how to squeeze this into the tiny 4MB Flash the E32 and S32 boards have. On BBQKees's latest prototype it works like a dream since it has an embedded ESP32 chip with 16MB flash!

@MichaelDvP I might need your expertise here to hack the IDF partition tables.

[proddy]

I managed to squeeze in the languages into the 4MB flash and make it dynamic. So you can adjust the language in the WebUI from the sign-on screen and also change the language of the device entity names from the Settings without re-starting the ESP32. A few notes

• I've done German (DE) for starters. It's most likely all wrong as I used google translate, sorry
• The WebUI language translations are in interface/src/i18n/de and interface/src/i18n/en. The English (en) language is leading and any other language file should match. To create new languages simply add a new folder.
• Translations to the web can be added and tested real-time by running npm run standalone from the interface directory. No need to upload to the ESP32.
• I've only translated the login screen and the Dashboard so far in the web. The rest is easy but time-consuming. I'll add bit by bit.
• The languages in the EMS-ESP core are stored in system.cpp as const char * const languages[] = {EMSESP_LOCALE_EN, EMSESP_LOCALE_DE};. This can be expanded but the sequence is important. So 0 for EN, 1 for DE, etc...
• The core files are located in a single file called src/locale_translations.h. I've missed some of the German translations and have marked these with a comment // need DE.
• I made a few fixes and optimizations in the code as I re-looked into code again. It's nice to be back! For example the way we store device entities and manage the numerical operations like div2, div10 etc. This does mean I've probably introduced a few more bugs but we'll fix along the way just like old times
• When we have DE done we can think of adding Dutch NL and Swedish SE. I think we can safely add up to 5 more languages without blowing the memory, If we do, we'll have to fix the language at compile time,

The code is at https://github.com/emsesp/EMS-ESP32/tree/v3.5.0

@bbqkees @MichaelDvP can you give it a whirl and see if it works?

[pierafal]

I can help with polish translation if you like. I didn't have time to go through the whole threat yet, but if you could send me some details of what is needed, i'll try

[MichaelDvP]

@pierafal Great, there are two files to edit.

• interface/src/i18n/pl/index.ts for the web, you have to replace the english texts by polish.
• src/locale_translations.h for entities, you have to append the polish translation as , F("translation") to each list before the closing bracket, Scheme: MAKE_PSTR_LIST(name, F(en), F(de), F(nl), F(se), F(pl))

If you want to check the we translations you can use the mock-api, but for the entities you need to compile and flash to emsesp.

[proddy]

@MichaelDvP let's create a section in the 'For Developers' section on the wiki stating the steps needed to add a new translation.

[MichaelDvP]

I've made a PR to the doc with (hopefully) all steps to add a new language. Please check.

[bbqkees]

Which languages do we want to implement in this first round?
If we have EN/DE/NL/SE/PL we already covered the most important ones.
Others can wait I guess. CZ/HU/IT/FR would be nice but not a lot of activity over there.
On the docs you mentioned a Finnish translation but this is not a country I shipped to recently.

[proddy]

It wasn't Finnish (my mistake) but Norwegian! Viking_NO requested it on https://discord.com/channels/816637840644505620/816638754772353095/1004403792529346732

[proddy]

Polish (PL) translation is underway but it'll take a few weeks and will be incomplete. We may decide to comment that language out to avoid confusion when we push 3.5.0 to dev

[MichaelDvP]

Ok, i think remove only the selections.
Maybe it's easier to collect the translations in a spreadsheet and generate the headerfile from that.
Than we can sort the columns and it's better see missing translations.

(Edit: Sheet removed, please use newer sheet below)

[MichaelDvP]

This is the web translations as table:
web.xlsx
I'll make a PR with pl and no selections disabled.

[proddy]

that's great if you have a script that can quicky output the locale string data

[MichaelDvP]

It's only export csv and three replace operations in editor,
Insert a empty column after tag and export csv to have tag;;shorttext;en,de,nl,se
replace ;;by,F(", replace ; by "), F("and replace \r\nby ")\r\nMAKE_PSTR_LIST(

[MichaelDvP]

I've tested the way back, maybe it's easier if we leave line start/end as fields, and replace only multiple semicolon.

BTW: in this spreadsheet missing translation could be seen much easier, @mvjt could you please check lines 353-356 and 407, 408.
@pierafal You could add the translation to this sheet if you like, no handling of brackets and quotation marks.

(Edit: sheet removed)

[mvjt]

I've tested the way back, maybe it's easier if we leave line start/end as fields, and replace only multiple semicolon.

BTW: in this spreadsheet missing translation could be seen much easier, @mvjt could you please check lines 353-356 and 407, 408. @pierafal You could add the translation to this sheet if you like, no handling of brackets and quotation marks.

locale_translations.xlsx

Hi @MichaelDvP , the Swedish translations for 353-356 and 407,408 are there. You're missing the NL translations. So the Swedish ones are now in the NL column.

[MichaelDvP]

Thanks for checking, @bbqkees could you please take a look.
Updated list:
locale_translations.xlsx

[proddy]

Kees is on holiday so I'll see if I can add the NL ones.

[tp1de]

I tried to do my best to adjust some German translations (marked in yellow). I have to admit that I do not understand all parameters and some descriptions are not unique .....
locale_translations.xlsx

[MichaelDvP]

Thanks Thomas,
i've added most of them, some WW are not needed since the entities are prefixed with dhw (planned to translate to WW). For a few others i stick to wording from my german Buderus manual (Antpendelzeit, Einschaltdifferenz, etc.)
I've also added the web-translations to this xlsx, for NL the later introduced help texts are missing (en-text).
I'll make a PR to dev soon with this texts..
locale_translations.xlsx

[pierafal]

hi, i've done pretty much of the web translations but i see some issues. Please see the web.xlsx, a color cells in column F requires some clarification to understand the context. Can someone please review that and comment?
The issues:

1. different inflections of words. I.e. In english you have 1 "Second" and for plural you have "Seconds". In Polish you have 1 "Sekunda", 2-4 "Sekundy", and for more than 4 you have i.e. 5 "Sekund". So we have more than 1 inflections for plurals.
2. In english You can have separate words like "System", and "Log" that can work together as a "System Log". In Polish, you can translate separate words respectively "System" and "Dziennik", but when you put these straight together you'll have "System dziennik" and it sounds very weird - inflection needs to be used to have "Dziennik systemowy" which is correct form.
3. i do not see words "official" and "development" to be translated, from the Version Check.

What do you think of creating a screen shots of each WEB page in english to have a context of what we're translating. IMHO a list of words is not enough to create a good translation. We need to understand the context
web.xlsx

[MichaelDvP]

Yes it's not easy to understand the context. For web translations it is best to use the mock-api and check the webpage with your translations to see the context.

1. Thiis is pluralization described here, for poland i find Category one/few/many/others here (see definition of d on top)
   I think for secondes this should be {num} Sekun{{da|dy|d|d}} expanding to 1 Sekunda, few Sekundy, many Secund, other Secund. I don't know what difference is in many/others.
2. Yes, the LOG is only used in combination with System. This is set on many pages for Status and settings: Network Status, Network settings, Mqtt Status, Mqtt Settings, etc. It is a bit redundant because the page headline is System and the tabs are named System Status, System Log. @proddy Any idea how to handle this?
3. Yes, actually only THE_LATEST is translated, official and version is are fixed.

If you don't compile yourself, the v3.5 is now in dev, you can download the bin and test the EN page and check the context.
For mqtt you'll find the options here. I think you should not translate clean session, retain flag and response-topic , as they are special terms in mqtt configuration, but translate the always set . The options for single topics gives:

• normal publish is a long json with all parameters: thermostat_data:{"hc1":{"seltemp":21, .....}
• single topic publishes a value for each parameter: thermostat_data/hc1/seltemp:21
• publish to command-topic is: thermostat/hc1/seltemp:21 This is ONLY usefull for a mqtt-broker that is a endpoint and do not republish (like ioBroker-mqtt (if set)). A standard mosquitto will republish this to subscribed ems-esp as command to set seltemp to 21 and this ends up in chaos.

Problem loading is shown on a red snackbar on the web ifsomething goes wrong with loading data.
offset, factor are settings for configuring anlaog sensors.

[MichaelDvP]

Tested the pluralisation, it has to be {{zero|one|two|few|many|other}} and added this to xls and web.
I have added this so far to the dev build, @pierafal you can test a bin here, but i also made a PR, the missing translation make a fallback to EN.

Updated list with web and entities: locale_translations.xlsx

[MichaelDvP]

I've found that the EN-fallback in web only works only for a missing index-file, not for single word missing. So i set missing texts to english and marked them red in the xlsx.
locale_translations.xlsx

[proddy]

I've added the missing NL ones

[pierafal]

Guys, sorry but i'm not sure what should I do to use mock-api. I was expecting doing some tranlslation in a human readible form like xls. I'm still ready to work on that but I'd need more detailed instruction.

[proddy]

Guys, sorry but i'm not sure what should I do to use mock-api. I was expecting doing some tranlslation in a human readible form like xls. I'm still ready to work on that but I'd need more detailed instruction.

did you edit the files in the repo or update the XLS? Sorry, can't remember...

[pierafal]

I edited the xls file

[proddy]

ok, are you able to use check out the source file from the repo, make the edits and push back as a PR? If you've never worked in github before we can always do it manually from the XLS you created but it'll take some time

https://emsesp.github.io/docs/#/Coding?id=adding-a-new-language

[MichaelDvP]

@pierafal Your web translations from xls file are already implemented in the dev. Only a few words are missing, see here. You can flash the dev and check, no need for the mock-api.

Next is to add translations of entities in locale_translations. You can use the xls or edit a local copy of locale_translations.h

[proddy]

the last changes to support Polish were added. Closing this now.