Some entities get wrongly encoded, when using entities_processNumerical=true

[agorum]

Type of report

Bug

Provide detailed reproduction steps (if any)

Open JSFiddle to see the problem: https://jsfiddle.net/veyd1gjn/10/

There is an Emoji in there 👍 and the config.entities_processNumerical is set to true.

When switching to Sourcecode of the Editor and back, the Emoji is destroyed, showing ��

In Sourcecode view it is encoded to: "��"

The correct value would be "👍"

Maybe, it is because the number is larger than 0xFFFF ?

Expected result

The expected result, that the encoded entity is correctly encoded and decoded.

Actual result

The entity is destroyed. From 👍 to ��

Other details

• Browser: Chrome
• OS: Windows 10
• CKEditor version: 4.5.0+
• Installed CKEditor plugins: entities

[Comandeer]

It seems that it's connected with the fact that we use String#charCodeAt() in the entities plugin instead of String#codePointAt() and a regex without the u flag. Due to that, the emoji is converted to its surrogate pair.

The quick workaround involves two changes inside the entities plugin:

1. ckeditor4/plugins/entities/plugin.js

   Line 109 in bb0a805

   return config.entities_processNumerical == 'force' || !entitiesTable[ character ] ? '&#' + character.charCodeAt( 0 ) + ';'

   – charCodeAt should be replaced by the codePointAt,

2. ckeditor4/plugins/entities/plugin.js

   Line 146 in bb0a805

   entitiesRegex = new RegExp( entitiesRegex, 'g' );

   – the g flag should be replaced by the gu flags.

However, this fix works only if there is no need to support IEs. As CKEditor 4 supports these browsers, we need to think of a more proper fix that will work also there.

@agorum, I've updated your original report to contain info about the entities plugin.

[agorum]

Thank you very much, that did the trick!

Internet Explorer is not necessary for us anymore.

As a default fix, I would suggest to implement a config-option to turn on this behavior (with the warning, that IEs are not supported, when turning on this option). What do you think?

[Comandeer]

It sounds reasonable. However, I'm not sure if simpler approach wouldn't be even better: check if codePointAt() method is available and use it, otherwise fall back to the older charCodAt() one; the same for u flag.

[agorum]

Sounds good!

[CKEditorBot]

Closed in #5306