FreeType <= 2.6.1 on OSM rendering server causes Noto CJK too much bold

[littlebtc]

#2349, #2391 Followup.

As the patch that enabled Noto CJK on Standard layer is up, I wrote this issue.

Take Tokyo, Japan for example, you can see the CJK fonts is way too bold compared to ref numbers:

IMHO we can use lighter variant of Noto CJK for both regular and bold font.
My personal suggestion:

• Noto Sans CJK Regular -> Noto Sans CJK DemiLight
• Noto Sans CJK Bold -> Noto Sans CJK Medium

The following is a screenshot I made to compare the font weight:

But actual Mapnik rendering may be compared too. Can someone do it in Tokyo, Seoul and Taipei?

[littlebtc]

Unifont (left) vs Noto CJK (right).
Taken when rendering is updating.

[vholten]

I think it wood be a good idea to find out what exactly causes the boldness of this font. I don't think it is caused by the weight of the font.
In #2349 (comment), @sommerluk wrote

This might be a configuration problem. At least for me it renders fine (=regular) locally

and in #2349 (comment) @pnorman wrote

So it looks like the difference comes from different versions of the font, not anything we can change on the stylesheet. The same for the boldness in Chinese

Apparently both of them were able to render Noto Sans CJK without excessive boldness. It would be useful to know which versions of the font they installed. My guess is that the hinted / unhinted versions of the font could explain the difference, but this has to be tested.

[sommerluk]

Unfortounally, I’m currently busy with other stuff, so I can’t investigate further on this. However, to generate the screenshot at #2349 (comment) I was using the package that comes with openSUSE Leap 42:

noto-sans-cjkjp - Noto CJKjp Sans Serif Font

Version:
20151002-1.2

/usr/share/fonts/truetype/NotoSansCJKjp-Black.otf
/usr/share/fonts/truetype/NotoSansCJKjp-Bold.otf
/usr/share/fonts/truetype/NotoSansCJKjp-DemiLight.otf
/usr/share/fonts/truetype/NotoSansCJKjp-Light.otf
/usr/share/fonts/truetype/NotoSansCJKjp-Medium.otf
/usr/share/fonts/truetype/NotoSansCJKjp-Regular.otf
/usr/share/fonts/truetype/NotoSansCJKjp-Thin.otf

[vholten]

Thanks. Since you used the otf version, a possibility is that the OTF versus TTC versions of the font could explain the non-bold / bold difference.

[pnorman]

Apparently both of them were able to render Noto Sans CJK without excessive boldness. It would be useful to know which versions of the font they installed. My guess is that the hinted / unhinted versions of the font could explain the difference, but this has to be tested

One was Ubuntu 14.04 packaged, the other was the latest version of Noto from git. Both should have used hinted if available.

[vholten]

I have further investigated the issue. I have compared the TTC and OTF versions of the font, and both render the same (too bold in my year-old version of Kosmtik). There are no unhinted versions of the CJK fonts as far as I can see, only hinted ones (even Google's Noto-unhinted.zip contains hinted CJK fonts).

So the fonts are not the problem. Instead, it appears the problem is caused by the FreeType rasterizer that Mapnik uses. Some background: OpenType fonts come in two flavors: TrueType-based, with extension ttf, and PostScript-based (also called CFF), with extension otf. All Noto fonts are TrueType except the CJK fonts, which are CFF. That is why we are seeing the boldness problem only with the CJK font and not with the other scripts.

In June 2013, FreeType 2.5 switched to Adobe's engine for rendering CFF fonts. This engine contains a feature called "stem darkening" which makes fonts thicker at small sizes. Many users found that this feature makes fonts much too dark, which is caused by incorrect gamma correction. For this reason, stem darkening was switched off in FreeType 2.6.2, released on 28 Nov 2015.

This then is the reason for the too bold fonts: when a version of FreeType from 2.5 to 2.6.1 is used, which have stem darkening turned on. For those using Kosmtik, node-mapnik was upgraded from FreeType 2.6.1 to 2.6.5 on 9 Sep 2016 with node-mapnik 3.5.14. I have just installed a fresh copy of Kosmtik with node-mapnik 3.5.14 and verified that the CJK fonts look OK there.

On the rendering servers, which use Ubuntu 16.04.1 LTS, the installed version of FreeType is probably the packaged version which is 2.6.1 and has the stem darkening turned on. So to solve the problem on those servers, installing a newer version of FreeType would probably be sufficient.

Comparison old / new version of Kosmtik and node-mapnik:

[mboeringa]

In June 2013, FreeType 2.5 switched to Adobe's engine for rendering CFF fonts. This engine contains a feature called "stem darkening" which makes fonts thicker at small sizes.

Sounds very much like the horrible "Enhance thin lines" option Adobe made default for Adobe Reader, and that ruins elegant (cartographic) designs with thin lines when displayed in its default ON status by making them bolder and over-emphasizing them:
https://forums.adobe.com/thread/666344

[pnorman]

This then is the reason for the too bold fonts: when a version of FreeType from 2.5 to 2.6.1 is used, which have stem darkening turned on. For those using Kosmtik, node-mapnik was upgraded from FreeType 2.6.1 to 2.6.5 on 9 Sep 2016 with node-mapnik 3.5.14. I have just installed a fresh copy of Kosmtik with node-mapnik 3.5.14 and verified that the CJK fonts look OK there.

I had initially doubted this could explain it since I had a less bold rendering with Noto when I used fonts from git, but I realized that I was probably getting Unifont there.

[littlebtc]

Thanks for investigtion. I have changed the title.

[littlebtc]

Unfortunately it seems no official backports of FreeType available for Ubuntu Xenial.

• Request a backport?
• Use PPA Packages?
• Manually convert fonts to TTF?

[pnorman]

• Request a backport?
• Use PPA Packages?

yakkety has 2.6.3, so I'd prefer a backport over backporting ourself with a PPA. Unfortunately libfreetype6 has a huge reverse depends list including some core components so I don't think this is practical.

• Manually convert fonts to TTF?

That might be the best option.

[sommerluk]

@vholten Thanks for your investigative work and the detailed description!

[littlebtc]

Simply converts to (OpenType) TTF does not work, I am trying otfcc solution to convert the TTF.

[vholten]

• Manually convert fonts to TTF?

I would advise against this. TTF and CFF are fundamentally different formats and the conversion is not lossless. In particular, we would probably lose all hinting. There is a reason that the CJK fonts were only released as CFF. As explained in notofonts/noto-fonts#249,

There are currently no plans to release TrueType versions of the Noto Sans CJK fonts. One huge advantage of the OpenType/CFF versions is that they have superior hinting.

If installing a more recent version of FreeType on the rendering servers is not practical, there is another solution. The "stem darkening" feature can be turned off in all versions of FreeType, including 2.6.1. This is done using the FT_Property_Set function as described at https://www.freetype.org/freetype2/docs/reference/ft2-cff_driver.html#no-stem-darkening(cff) .

If that is also not possible, maybe the FreeType 2.6.1 package could be rebuilt from source, changing the line driver->no_stem_darkening = FALSE; in cffobjs.c to driver->no_stem_darkening = TRUE;

[littlebtc]

I tried to convert the OTF file for a whole day but still are not able to find a good solution.

Converting to TTF is not a good way to go.

So there are two possible patch point:

• Mapnik, add FT_Property_Set to Here
• FreeType

IMHO patching FreeType is a little scary. What do you think @pnorman ?

[littlebtc]

It is 3AM in Taiwan now, so I have to sleep. I will try to patch Mapnik tomorrow to see the result.

🌙

[pnorman]

This is done using the FT_Property_Set function as described at https://www.freetype.org/freetype2/docs/reference/ft2-cff_driver.html#no-stem-darkening(cff) .

I tried setting FREETYPE_PROPERTIES but it did nothing, and it looks like it was added in 2.7.

Neither patching FreeType nor Mapnik is within the scope of what we can do here. I'm revising #2409 to remove the mention of converting the CJK fonts.

[littlebtc]

So IMHO, the most practical way to fix the boldness in rendering server is to use lighter variant of Noto Sans CJK until we are able to upgrade FreeType to 2.7 and newer. (see below)

This may cause some confusion for tile designer, but I think it is the best way for everyone.

[littlebtc]

Wait, I am wrong, maybe Stem darkening is too much bold that changing the font weight will not be so effective.

[littlebtc]

I have tried to install FreeType 2.7 PPA from here, which overrides libfreetype6 for Ubuntu Xenial in my Docker image. It really makes the rendering of Noto CJK normal.

But I don't know what else can happen if just upgrade the libfreetype6.

---

And it seems I cannot request backport for freetype as Ubuntu Yakkety is not yet released.
Frustrated day. :(

[pnorman]

And it seems I cannot request backport for freetype as Ubuntu Yakkety is not yet released.
Frustrated day. :(

Operations is trying it in the PPA. The problem with doing an official backport is that we'd need to test the rdepends packages

[vholten]

Wait, I am wrong, maybe Stem darkening is too much bold that changing the font weight will not be so effective.

I still hope we can solve this by updating FreeType, but if not, here are renders of CJK Thin, Light, DemiLight, and Regular, with an old version of FreeType:

Thin works quite well. However, if we use it, the font will look OK on the rendering servers but not on systems with a recent FreeType.

[pnorman]

Operations did a backport from yakkity

old

new

This issue remains open because we need to document the freetype requirement

[vholten]

Great! I thank the operations team for their quick response.
Is the fix now live on the rendering servers?

[tomhughes]

Yes - backport is in our PPA at https://launchpad.net/~osmadmins/+archive/ubuntu/ppa and has been installed on all three servers and renderd restarted.

[littlebtc]

Huge thanks!!
👍

I have checked the rendering, all goes well. :)

[sommerluk]

@tomhughes Thanks a lot for the quick solution!