Fix handling of empty matches in iterators in UTF mode #36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
November 7, 2023 22:48
Following an empty match, the iterator (Matches or CaptureMatches) advances the last_end position so as not to return the same match twice. However, if the regex uses UTF mode, the position passed to find_at_with_match_data or captures_read_at is required to be a UTF-8 character boundary, so last_end must be advanced by a whole UTF-8 character, not just one byte. Determining whether or not the regex is using UTF mode requires PCRE2_INFO_ALLOPTIONS (checking the config is not enough.)
These tests will fail if the iterator does not correctly advance the end position (by either a byte or a whole character, as appropriate) following an empty match.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
find_iterandcaptures_iterfunctions iterate over the distinct, non-overlapping matches within the subject string.Normally, this means the iterator will start searching at the end position of the previous match. However, if the previous match was zero-width, we want to advance by an additional character so we don't return the same match again.
In byte mode, this just means incrementing the position by 1. In "UTF" mode, however, we need to advance to the next UTF-8 character boundary.
Note that to determine whether the regex is in UTF mode, we can't simply check the
utfflag of itsconfig, because UTF mode can also be turned on by including(*UTF)in the pattern itself. So we need to invokepcre2_pattern_info_8to check which mode is actually used.(I also considered that, instead of adding a
utffield toRegex, theRegex::buildfunction could modify the savedconfig. It's possible that might work but it seems fragile.)