Regex not matching malformed UTF-8

[fonsp]

I noticed this behaviour on 1.7.0 (it throws an error on 1.6.5, see #25997 for prior discussion):

julia> match(r".+", "hello\x80world")
RegexMatch("hello")

I expected . to match any character, including invalid characters.

In JS (V8 engine), which also allows arbitrary data in strings, the result is:

> let c = new TextDecoder().decode(new Uint8Array([0x80]))

> /.+/.exec(`hello${c}world`)
[ "hello�world" ]

This came up in JuliaWeb/HTTP.jl#796

[vtjnash]

In V8, which does not allow arbitrary data in strings, it has lied to you and changed your string to only contain valid characters:

> new TextEncoder().encode(c)
Uint8Array(3) [239, 191, 189]

> c.charAt(0) == '\ufffd'
true

PCRE2 made the decision to not consider their definition of any character to include bytes that are not characters (#39524)

[fonsp]

Just learned something new, thanks Jameson!

Let's close then, I suppose that PCRE2 has a good reason for that decision.