Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support "true" extended grapheme cluster #46

Closed
k-takata opened this issue Nov 18, 2014 · 2 comments
Closed

Support "true" extended grapheme cluster #46

k-takata opened this issue Nov 18, 2014 · 2 comments
Labels
todo

Comments

@k-takata
Copy link
Owner

Currently \X is defined as (?>\P{M}\p{M}*). This definition is the same as Perl 5.10. However the definition was changed as of Perl 5.12.

See pp.15-18 of this slide. The slide says Onigmo supports legacy grapheme clusters.

See also: UAX #29: Unicode Text Segmentation
@k-takata k-takata added the todo label Nov 18, 2014
@mtsmfm mtsmfm mentioned this issue Oct 9, 2016
Merged
@k-takata
Copy link
Owner Author

Related: /\X/ (extended grapheme cluster) can't pass unicode.org's GraphemeBreakTest

k-takata added a commit that referenced this issue Dec 1, 2016
@k-takata
Import Ruby r56949 (Issue #46)
dde0c43 
Regexp supports Unicoe 9.0.0's \X

* meta character \X matches Unicode 9.0.0 characters with some workarounds
  for UTR #51 Unicode Emoji, Version 4.0 emoji zwj sequences.
  [Feature #12831] [ruby-core:77586]

The term "character" can have many meanings bytes, codepoints, combined
characters, and so on. "grapheme cluster" is highest one of such words,
which means user-perceived characters.
Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION specifies how to
handle grapheme clusters (extended grapheme cluster).
But some specs aren't updated to current situation because Unicode Emoji
is rapidly extended without well definition.
It breaks the precondition of UTR#29 "Grapheme cluster boundaries can be
easily tested by looking at immediately adjacent characters". (the
sentence will be removed in the next version)
Though some of its detail are described in Unicode Technical Report #51
UNICODE EMOJI but it is not merged into UTR#29 yet.

http://unicode.org/reports/tr29/
http://unicode.org/reports/tr51/
http://unicode.org/Public/emoji/4.0/
k-takata added a commit that referenced this issue Dec 1, 2016
@k-takata
Fix \X on UTF-16/32 (Issue #46)
bbb8b4c
k-takata added a commit that referenced this issue Dec 1, 2016
@k-takata
\X should match \x00D\x0A also on non-Unicode encodings (Issue #46)
ca16697
k-takata added a commit that referenced this issue Dec 1, 2016
@k-takata
Use add_ctype_to_cc_by_range() instead of add_code_range()
67fa14b 
(Issue #46)
k-takata added a commit that referenced this issue Dec 1, 2016
@k-takata
Use raw string if a cclass contains only one character
d3a6585 
(Issue #46)
@k-takata
Copy link
Owner Author

k-takata commented Dec 2, 2016

Fixed.

@k-takata k-takata closed this as completed Dec 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
todo
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@k-takata