Reloaded matomo data gives yamerl error

[achedeuzot]

First of all, thanks for this amazing elixir module ! I've been using it for a few months and it's very useful.

I think there's something strange with one of the latest matomo updates: after downloading the data (both ua_inspector.download.databases and ua_inspector.download.short_code_maps), I get the following error:

[error] GenServer UAInspector.ShortCodeMap.DeviceBrands terminating
** (stop) bad return value: {:yamerl_exception, [{:yamerl_parsing_error, :error, 'Invalid Unicode character at byte #3304', :undefined, :undefined, :invalid_unicode, :undefined, [byte: 3304]}]}
Last message: {:"$gen_cast", :reload}
State: %{}

It started on 11th Feb 2019 between 01:00 and 02:00am CET.

I tried clearing my "cache" and redownloading and I get the same error. Any ideas in what causes this ?

EDIT: I think I found the origin of the issue at this commit from 2 days ago (the special character is ö): matomo-org/device-detector@b64c80b#diff-301e1dcec83d084222343ca8c90095b1R335

EDIT 2: If anyone stumbles on the same issue, my temporary fix is to lock the commit just before the one mentioned above so the downloaded DB is OK:

remote_database = "https://raw.githubusercontent.com/matomo-org/device-detector/6ddc1650c5cb2af870a6d9dca473894e3c6791b8/regexes"
remote_shortcode = "https://raw.githubusercontent.com/matomo-org/device-detector/6ddc1650c5cb2af870a6d9dca473894e3c6791b8"

[mneudert]

Thank you very much for opening this issue and even providing the source of the error and a workaround ❤️

The problem seems to be the file being locally saved as ISO-8859 instead of the required UTF-8. This would be the "shell scripty" workaround after the download was done:

iconv -f ISO-8859-15 -t UTF-8 short_codes.device_brands.yml > short_codes.device_brands.yml.new
mv short_codes.device_brands.yml.new short_codes.device_brands.yml 

I will work on a proper fix asap.

---

Just for reference this should be the full workaround configuration required:

remote_database = "https://raw.githubusercontent.com/matomo-org/device-detector/6ddc1650c5cb2af870a6d9dca473894e3c6791b8/regexes"
remote_shortcode = "https://raw.githubusercontent.com/matomo-org/device-detector/6ddc1650c5cb2af870a6d9dca473894e3c6791b8"

config :ua_inspector,
  remote_path: [
    bot: "#{remote_database}",
    browser_engine: "#{remote_database}/client",
    client: "#{remote_database}/client",
    device: "#{remote_database}/device",
    os: "#{remote_database}",
    short_code_map: "#{remote_shortcode}",
    vendor_fragment: "#{remote_database}"
  ]

[mneudert]

I have just now published the release v0.19.2 including a fix for this issue to hex. If you (or anyone else coming across this) requires a published fix for an older version like v0.18.0 please just drop me a note here and I will prepare one. Of course also please comment if the release does not work as intended in any way.

💙 💚 ❤️ 💛 💜