fix: Convert numbers to hexadecimal inside Unicode escape sequences

[graphemecluster]

Feel free to suggest a better comment that you feel comfortable with.

For simplicity, this PR doesn’t distinguish between Unicode and Unicode property escape sequences. It works by the fact that, among the currently allowed LoneUnicodePropertyNameOrValue, only gc, sc and scx start with a lowercase character and none of them are purely a~f. Don’t hesitate to close this PR if you would like to achieve this by separating ENCLOSED_TOKEN to P_TOKEN and U_TOKEN instead.

Only minimal test cases are written, I’d love it if you could take over and include more.

[slevithan]

Nice catch! And thanks for the PR!

It works by the fact that, among the currently allowed LoneUnicodePropertyNameOrValue, only gc, sc and scx start with a lowercase character and none of them are purely a~f.

That's clever, but relying on this will lead to confusing errors. Ex: If given \p{${12}} it will generate \p{c} which is correctly an error (thanks to JS's case sensitivity for Unicode properties), but it looks a lot like the valid \p{C}.

Don’t hesitate to close this PR if you would like to achieve this by separating ENCLOSED_TOKEN to P_TOKEN and U_TOKEN instead.

That's a good idea (since I don't love relying on the generated error), but if I decide to change this I can do it in a follow up diff to avoid holding back your PR.

Only minimal test cases are written, I’d love it if you could take over and include more.

No problem. Will do!

[slevithan]

I've now added additional tests and split ENCLOSED_TOKEN into two values. Thanks again!

Also, I realized that converting to hex within enclosed \p{…} and \P{…} would have introduced a bug since you can do e.g. regex`\p{C${12}}`. That would have given /\p{Cc}/v, which is not an appropriate result but doesn't throw because it's a valid regex.

---

among the currently allowed LoneUnicodePropertyNameOrValue, only gc, sc and scx start with a lowercase character

There are a few others starting with a lowercase letter that JS supports:

• \p{cntrl} - An alias for general category Cc/Control.
• \p{digit} - An alias for general category Nd/Decimal_Number.
• \p{punct} - An alias for general category P/Punctuation.
• p{space} - An alias for binary property White_Space, NOT general category Z/Separator.

As far as I know, that's all of them.

[slevithan]

Published in v4.3.3.