[red-knot] Hand-written MDTest parser

[sharkdp]

Summary

Replaces our existing Markdown test parser with a fully hand-written parser. I tried to fix this bug using the old approach and kept running into problems. Eventually this seemed like the easier way. It's more code (+50 lines, excluding the new test), but I hope it's relatively straightforward to understand, compared to the complex interplay between the byte-stream-manipulation and regex-parsing that we had before.

I did not really focus on performance, as the parsing time does not dominate the test execution time, but this seems to be slightly faster than what we had before (executing all MD tests; debug):

Command	Mean [s]	Min [s]	Max [s]	Relative
this branch	2.775 ± 0.072	2.690	2.877	1.00
main	2.921 ± 0.034	2.865	2.967	1.05 ± 0.03

closes #15923

Test Plan

One new regression test.

[sharkdp]

This is something that we could remove. It seems strange that we trim trailing newlines of code blocks. But I didn't want to change the behavior in this PR.

[sharkdp]

This message could be improved/adjusted according to the logic in the if condition, but I didn't want to introduce and behavioral changes here.

[MichaReiser]

This is great. No more regex :)

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[MichaReiser]

Oh, I wasn't aware that eat_char2 is only available on the ruff_python_parser Cursor.

[sharkdp]

Oh. And I didn't notice that eat_char2 was available somewhere. My implementation is slightly different:

pub fn eat_char2(&mut self, c1: char, c2: char) -> bool {
        if self.first() == c1 && self.second() == c2 {
            self.bump();
            self.bump();
            true
        } else {
            false
        }
    }

vs the existing:

pub(super) fn eat_char2(&mut self, c1: char, c2: char) -> bool {
        let mut chars = self.chars.clone();
        if chars.next() == Some(c1) && chars.next() == Some(c2) {
            self.bump();
            self.bump();
            true
        } else {
            false
        }
    }

Should I do anything about this?

[MichaReiser]

I think it's fine. The second might be slightly faster but then, it's used rarely.