What about Han unification? #2208
Comments
|
Is it possible to add some sort of regional specific selector (based on country or bbox) as cartocss extension? I think that's the basic facility required for using different glyph forms for their respective regions when preferred language is unspecified. The default rendering should be using the region-specific glyph forms, assuming |
|
I'm not sure, but the biggest issues for the map might be between Traditional Chinese and Japanese. Maybe also for traditional (Taiwan) vs. simplified (Mainland China) Chinese. The map for Korea has almost no place names with Chinese characters, They are all in Hangeul and there is no overlap with Chinese or Japanese. However, some objects in Korea have Hanja names too. It usually is in the name:zh tag, which is wrong, but there is no better proposal. See this discussion https://lists.openstreetmap.org/pipermail/talk-ko/2015-October/000228.html |
Not within CartoCSS as it exists now. If it got added we could consider using it. It's possible to do something with more complicated SQL queries, but this gets messy, and is even messier without defining functions in PostgreSQL, which we avoid. |
Chinese-specific tiles would require modifying the style, so modifying the font list for a better rendering is pretty easy for someone to do once they've started modifications. fwiw, I think we're stuck with the Han unification problems with current technologies. |
|
Unicode 9.0.0 core specification http://www.unicode.org/versions/Unicode9.0.0/UnicodeStandard-9.0.pdf has an implementation guideline about language information in plain text and especially Han unification: Chapter 5.10. |
I've read over it and it doesn't help much. In the situations they describe there is implicit language information by the reader being either Japanese or Chinese and having corresponding fonts. With server-side rendering (and most client-side) the fonts are supplied so none of the scenarios are what we have. I do note that "plain text remains legible in the absence [of format specifications]" |
And: "The goal and methods of Han unification were to ensure that the text remained legible." "There should never be any confusion in Unicode, because the distinctions between the unified characters are all within the range of stylistic variations that exist in each country." |
|
Some improvements may be possible based on 5e5fb3b by reducing JP coverage to the Noto/Source Han Sans subset font, and padding it up with a SC or TC variant that has all the glyphs. This way all characters used by Japanese would be made Japanese, while the rest can be left to be written in the Chinese ways. Correction for my comment in #2608: It appears that Japanese, for example, don't quite use the character "门": https://ja.wiktionary.org/wiki/%E9%97%A8. Since it's possibly still in the subset file (according to the source han sans readme the subset still covers all the JIS X characters), someone may have to do some font editing to kick it out. |
@Artoria2e5 If I understand you correctly, the proposal with the region-specific subsets is not a solution for our problem, right? |
|
I’ve made some further investigations and updated the issue description (“first comment”). It seems to me that the only reliable way to support this is having in the OSM database itself the information about the language that was used in the name tag. |
|
@sommerluk Taking subsets can be a good enough solution as you can isolate characters not (usually) used by one region and give it a writing style from a region that commonly uses it. The region subset files can appear quite a bit too inclusive for any given region though. Name tagging is the ideal solution around this. |
This could be paired nicely with Mapnik if Mapnik were extended to dynamically read this value in from the database and pass it to harfbuzz. I've sketched out how that could work at mapnik/mapnik#3655 (comment). |
|
Using a name_lang=[lang code] tag or similar would solve the Han problem and the duplication of name tags by because the name:[local lang code]=[local name] tag could be omitted. |
|
@springmeyer Thanks! |
|
I’ve written a proposal for language information tagging at https://wiki.openstreetmap.org/wiki/Proposed_features/Language_information_for_name Feedback is welcome. |
|
@sommerluk I have thought about something similar, but limited to multilingual names: https://wiki.openstreetmap.org/wiki/User:Nebulon42/Multilingual_names Maybe something there is of value for this problem. Or vice versa :) |
|
@nebulon42 Thanks! I did not know your proposal. Great work! The syntax is essentially the same: semicolon-separated list of the language codes that are already used for name:= Additional to your proposal, I simply admit single-language values. Would you consider that my proposal is a superset of your proposal and would be enouth to also serve your purposes? |
Yes, I drafted it some time ago but did not have the time to push it further.
Definitely. If you have the time and energy to push this further I really appreciate that. If you need some help please tell me. If anything on my Wiki page suits your needs for the proposal like the example renderings etc. please do not hesitate to use it. I saw that there is some progress on the Mapnik side. If there is anything that needs to be done for CartoCSS, please create an issue and I will try to get it into the next release. |
|
Yes, that sounds good. The RFC at the tagging mailing list is done. The multilingual name processing is added as use case. Overall, the proposal is still quite short, also because my english is not so good. Hopefully that’s not an obstacle… About Mapnik and CartoCSS: The most important part is getting support for controling locl via a property in the stylesheet. As far as I know, that’s not done yet. Once it’s done, will it be automatically available in CartoCSS, or is it necesarry to add in manually? |
|
If it's only about the property then adding it to https://github.com/mapnik/mapnik-reference for 3.1.0 or which version it is released in would be sufficient. Then carto needs to used that updated reference in a new version and it is available. |
|
The voting for a language tag is now open at https://wiki.openstreetmap.org/wiki/Proposed_features/Language_information_for_name Support is welcome ;-) |
To both questions: I don’t know, neither did I found a good answer searching on the web. |
|
Working on Step 2 above here: mapnik/mapnik#3655 (comment) For Step 1, my feeling is that for the zh-hant locale this problem is so pervasive that an OSM data based approach isn't realistic - essentially any character that uses the 辶 radical is affected including common names of linear features like 大道(Boulevard), 步道(Trail), etc. (before/after using my
For the region-based approach, I'm skeptical that a polygon-based approach will result in an elegant implementation; what about a raster/bitmap solution? For example, a GeoTiff where each pixel encodes an 8-bit value corresponding to BCP47 language tag that can be sampled for every symbolizer - for my own uses I would probably implement this in-memory directly in the mapnik C++ code, but for OSM Carto I'm not sure how this would fit into the tile rendering path. A bitmap based approximation based on Z14 tiles might be detailed enough and would be 16384x16384 px, which is a reasonable size. |
|
Some good news re Han Unification: Mapnik merged mapnik/mapnik#4493 which accepts a
|
This is not necessarily a barrier - moving to a carto fork would not be out of the question. Ultimately, of course, support for manually selecting font A demonstration how this could be done was shown and discussed in https://imagico.de/blog/en/rethinking-name-labeling-in-openstreetmap-map-styles/ which does not, however, include the code of generating the language_regions data from OSM data. |
|
@mapmeld I see you're being modest about your contributions to mapnik! It's great to somebody taking an interest in mapnik development. Most of the commits these days seem to be code maintenance rather fixes. If you do have a functional mapnik development environment, it might be interesting to look at this issue, which is indirectly related to the current topic. We can probably find examples where ascenders / descenders mess up the placement of text in shield boxes. |
|
The use of admin boundaries to determine language display preferences has been discussed in another context (community forum?). It would be useful to review options for processing the admin boundaries using the new flex input. In #4431 a new table was created for admin boundaries. But this deliberately removed overlaps so that only the highest priority admin boundary way was included. This is probably not useful for the current application. As I read the current flex input, "boundaries" are loaded into both the roads and polygon tables, with "admin level" boundaries being read from roads and others from the polygon table. Is this because "admin" boundaries can't be guaranteed to be closed, and so it is only safe to treat them as a collection of ways? It would useful to have the admin boundaries stored as a set of nested rings, but I imagine that it will never be as neat as this. |
|
The original (unsplit) admin polygons are not suitable for efficient on-the-fly point in polygon lookup. And given the nature of the problem it would definitely not be necessary for the language lookup data to be updated in realtime. |
|
I wonder whether we can use a generated column to perform the job of your If the function returned "label to use" and Inefficiency of the polygon lookup is less of an issue if it is only happening when the object is added/changed. The generated column would need to be rebuilt if the "language polygons" changed, but, as you say, this would not be done in real time (not least to avoid abuse). |
|
My hope is that we could pilot this in a few areas, before bringing in every design concern (updatable admin boundaries, borders). Currently there's too much uncertainty about whether this will happen at all. We'd be in a better state if the pipeline exists, and we can compare real world tiles somewhere. Based on when I was looking up test cases, the number of pixels changed on real world tiles may be... underwhelming |
|
Changes can be made step by step but there needs to be a viable strategy. |
|
I'm re-reading your post and taking a look at the Noto CJK docs recently. A I didn't mention it in this thread yet, I gave in Option 2 of https://github.com/mapmeld/osm2pgsql-cjk some research on the |
|
My two cents on polygons is this: the polygons do not need to be very precise in practice. Only the parts that border another Sinosphere/glyph-style region actually matter, and even within this subset of points, the maritime polygon vertices are going to affect way fewer labels than the land vertices. So really the parts that require "reasonably" high resolution are:
That's it. The rest can use very crude hand-simplified polygons. |
That is true for CJK - in other cases it might be necessary to modify the fonts to be single language.
We definitively do not want to increase maintenance burden on the style by hand designing and shipping a spatial data set - even if very low detail. And this also would not be compatible with our practice (as also mandated by the Guidelines for new tile layers) to use OSM data where possible. And substantial long term mismatches between spatially based language assignments and the administrative boundaries as mapped in OSM would just be a source of complaints and disputes in an already sensitive domain. So - for production use - i do not consider this viable. Nothing, however, speaks against starting development of this with a placeholder data set - as long as you keep in mind that this will eventually need to be tackled. And please everyone keep in mind that this issue - despite its title - is generically about solving the issue that different scripts should be used for different languages and that we have no strait away method to determine the language of the name tag. In my proof-of-concept demonstration i combined this with a move away from showing the generic name tag everywhere - which would be a much bigger strategic change here, but which - as @mapmeld also hints at in his contemplations - naturally ties with practical solutions to this issue since those will often involve analyzing and possibly matching the various name tags. |
The SC TC HK font variants represent the glyph for Chinese characters adopted by local standard-setting bodies in China, Taiwan, Hong Kong respectively. Note that this difference is not mainly about simplified vs traditional, despite their name. And thus for example the glyph for a Traditional Chinese character in SC set based on China standard could be different from TC/HK set and that represent how the standard-setting body in China think that character should be written. (Although there are more nuances behind the situation). But given majority of Simplified users are most likely accustomed to the Chinese standard on how the characters should look, it might make sense to use the SC font even if Simplified characters appear in places outside China. But how to determine a name with Han character in Vancouver or Los Angeles is intended to be rendered using which glyph, I have no idea.
For the purpose of name in Hong Kong, I don't think it is an established custom for people to add zh-Hant name as it is the default and would simply duplicate with name:zh. And name:zh-Hans I think are mostly added by Simplified Chinese users visiting/arriving at the city. So I don't think evaluating whether they match the Han character in default name tag is a good way to figure out whether the name is Simplified or not. |
|
An update: I have a test branch with Simplified-only font + data from Liaoning (near China/DPRK border). I made red text in the same Carto selectors to see what's changed and what's remaining: 
Aside from the tagging / post-upload querying which was discussed earlier:
|
|
I put a server with the Simplified Chinese demo online at |
In the San Francisco Bay Area, it was not too long ago that Toishanese rendered in Traditional (Hong Kong) characters would’ve been the closest thing to a local default. However, that’s definitely no longer the case, especially outside of San Francisco Chinatown proper. Some city governments use Traditional (Hong Kong) Chinese, while some public transportation systems use Traditional Chinese in simplified fonts, and private businesses are an unruly mix of both traditional and simplified usage, depending on the owner’s background or clientele. In OSM, Anyways, this is just one data point from a non-native speaker. Others in the OSM community might be able to provide more insight about tagging and groundtruth, but it would probably be considered somewhat off-topic for this issue tracker. |
[This description is regularly updated to summarize the current state of discussion in the comments.]
What is the problem of Han unification for openstreetmap-carto?
The problem of Han unification is a general problem that is independent of any specific font!
Unicode encodes abstract characters (“meanings of signs”). It does not encode glyphs (“specific graphical representations of an abstract character”).
There are three Han scripts: The Chinese Han script, the Japanese Han script and the Korean Han script. According to the initials it is abbreviated “CJK scripts”.
A wide variety of abstract characters is shared between the CJK scripts.
There are glyphs that have the same appearance in all CJK scripts. There are other glyphs that are different in all CJK scripts.
Nevertheless, native language speakers expect to see the glyphs they are used to seeing (language-specific glyphs). Different to the Unicode consortium, they consider that the different glyph form makes also a difference in the meaning of the sign. They feel that the other glyph forms are a foreign language.
Furthermore, Chinese Han has two different script variants: simplified (People’s Republic of China) and traditional (Hong Kong, Macao, Taiwan). So it’s not enough to know the language, but you also have to know the script variant.
Furthermore, even Traditional Chinese Han glyphs are usually rendered differently in three different regions (Hong Kong, Macao, Taiwan). So it’s not enough to know the language and the script variant, but you also have to know the target region.
It is not possible with plain Unicode to distinguish these forms. (IVD does not help with Han unification.)
Good CJK fonts provide all these glyphs for all language variants. Via an OpenType feature, you can access the glyph variant that you need.
openstreetmap-carto uses yet the Noto fonts, which do support all Han target languages, Han target variants and Han target regions (except Macao).
The problem is how to make the choice between all the available glyph forms.
Web pages solve the problem by using HTML
langattribute. It contains an IETF language tag (BCP-47), which can provide information about language, script and region. So the rendering engine can easily choose the appropriate default glyph for this language-script-combination because it knows the target language and the target script and the target region.openstreetmap-carto has currently no knowledge about the target language or the target script or the target region (in the “name” key) and no region-specific rendering rules.
openstreetmap-carto policy is however to display text in the native language.
Question: How to render CJK names and other text in the native language?
Does this problem also exist in other regions of the world?
Yes.
There are four variants of the Cyrillic alphabet: Russian, Bulgarian, Serbian and Macedonian, the style can be selected by locl .
The character 'LATIN CAPITAL LETTER ENG' (U+014A) has different shapes in African and European languages, the style can be selected by locl .
The Syriac script (ISO 15924: Syrc/135) has the variants “Syriac (Eastern Variant), ISO 15924 Syrn / 136” and “Syriac (Western Variant), ISO 15924 Syrj / 137” and “Syriac (Estrangelo variant), ISO 15924 Syre / 138”. The current Noto version provides one single font file for syriac, the style can be selected by locl . However, for the future there are plans to split it up into three different font files: one for each style.
Technically, these problems are almost identical to the problem of Han unification.
Are there other problems that have the same technical base?
What is the current situation at openstreetmap-carto?
If we default to Chinese glyph forms, then also Japanese city names will be rendered with Chinese glyph forms, and Japanese people will feel like it is a Chinese map. If we default to Japanese glyph forms, then also Korean city names will be rendered with Japanese glyph forms, and Korean people will feel like it is a Japanese map…
Current defaults:
What is necessary for a better solution?
1. Knowledge about the target language/script/region of each label
namewithname:jp,name:zh… does not work. Example: The node http://www.openstreetmap.org/node/25248662 (english: Beijing) hasname=北京市andname:ja=北京市andname:zh=北京市. They are identical. We cannot reliably determine the language of thenamevalue.namevalue in the OSM database itself. As a separate tag that specifies the language code of the language that was used in thenamevalue. Furthermore, for Chinese rendering, an information about the region (Taiwan, Hong Kong, Macau, People’s Republic of China) is necessary.default_languageand this key is yet used on more than 200 relations in the OSM database.2. A way to get Mapnik actually render the correct localized fonts
The text was updated successfully, but these errors were encountered: