Detect Zawgyi (Burmese pseudo-Unicode)

[brawer]

Could Osmose detect Burmese text in Zawgyi encoding?

Zawgyi is a historic font encoding that looks like Unicode, and can therefore be passed around as UTF-8, but abuses certain Burmese codepoints in a non-standard way. When rendered as a Unicode string, Zawgyi-encoded text will show up with garbled characters. Although Zawgyi is on the decline, there’s still a fair amount of Zawgyi-encoded text in OpenStreetMap as of early 2025. A current list of bad strings in OSM is found by this tool.

This tool is written in Go and calls Google’s probabilistic open-source Zawgyi detector, plus a custom open-source Zawgyi-to-Unicode converter. Perhaps the above encoder could upload its findings to Osmose via the XML API. But actually, it should be trivial to implement a new analyzer from scratch as a Python plugin for Osmose. Zawgyi dection and conversion also exist in Python, published as PyPI packages myanmartools respectively PyICU. A plug-in for Osmose might look roughly like this. What do you think, would you be OK with this? What error class should be used, and how to get an ID for it?

import myanmartools
import PyICU

class TagFix_ZawgyiBurmese(Plugin):

    def init(self, logger):
        Plugin.init(self, logger)
        self.detector = myanmartools.ZawgyiDetector()
        self.converter = PyICU.Transliterator.createInstance('Zawgyi-my')

    def node(self, data, tags):
        err = []
        for key, value in tags.items():
            if not any(0x1000 <= ord(c) <= 0x109F for c in value):
                continue
            score = self.detector.get_zawgyi_probability(value)
            if score < 0.8:
                continue
            fixed_value = self.converter.transliterate(value)
            if value == fixed_value:
                continue
            err.append({
               "class": ????, "subclass": ????,
               "text": T_("Value contains Zawgyi-encoded Burmese"),
               "fix": {key: fixed_value},
            })
        return err

    def way(self, data, tags, nds):
        return self.node(data, tags)

    def relation(self, data, tags, members):
        return self.node(data, tags)

###########################################################################                              
from plugins.Plugin import TestPluginCommon

class Test(TestPluginCommon):
    def test(self):
        a = TagFix_ZawgyiBurmese(None)
        a.init(None)
        for name in [
                     u"foo",
                     u"",
                     u"ဘားအံ",
                     u"ကျိုက်မရော အဝေးပြေးလမ်း",
                    ]:
            assert not a.node(None, {"name": name}), name
        for name in [
                     u"ေအာင္ခ်မ္းသာလမ္း",
                     u"သင္​းပယ္​",
                    ]:
            self.check_err(a.node(None, {"addr:street": name}), name)

[frodrigo]

Yes. This check could be integrated into Osmose.

Please, could you open a PR with this code ?

Item 5070 could be fine. With class=50706.

https://osmose.openstreetmap.fr/en/issues/open?item=5070

Please add a only_for. I see that not documented, but look for example in the code, to limit the usage.

Maybe also limit the check to some tags ?