[bug] libxml 2.9.13 breaks HTML4 parser recovery from ill-formed < character

[flavorjones]

Summary

Nokogiri v1.13.2 shipped libxml 2.9.13. That version of libxml2 introduced a behavior change to how the HTML4 parser recovers when it sees a bare (ill-formed) < character (one that is not part of a start tag).

I've opened an issue upstream at https://gitlab.gnome.org/GNOME/libxml2/-/issues/339

Immediate next steps

• add test coverage for recovery from < in an HTML4 document to the Nokogiri test suite
• apply a patch to libxml2 reverting https://gitlab.gnome.org/GNOME/libxml2/-/commit/798bdf13f6964a650b9a0b7b4b3a769f6f1d509a
  • fix: revert libxml2 regression with HTML4 recovery #2462
  • fix: revert libxml2 regression with HTML4 recovery (v1.13.x branch) #2463
• ship v1.13.3 🤷

Less-urgent next steps

• test: add coverage for entities flavorjones/loofah#227
• prioritize CI should test against major downstream consumers #2293 to test loofah and rails-html-sanitizer (at least) against upstream libxml2/lbxslt

[flavorjones]

OK, I have a repro which seems common across the rails-html-sanitizer failures as well as my day job CI failures:

      it "handles < character" do
        input = %{<div> this < that </div>}
        expected = %{<div> this &lt; that </div>}
        actual = Loofah.scrub_fragment(input, :escape)
        assert_equal(expected, actual.to_html)
      end

with nokogiri v1.13.1, this passes. with nokogiri v1.13.2:

Expected: "<div> this &lt; that </div>"
  Actual: "<div> this </div>"

[flavorjones]

Without Loofah, here's the core problem:

# nokogiri 1.13.1
$ ruby -rnokogiri -e 'pp Nokogiri::HTML4::Document.parse("<div> this < that </div>")'
#(Document:0x3c {
  name = "document",
  children = [
    #(DTD:0x50 { name = "html" }),
    #(Element:0x64 {
      name = "html",
      children = [
        #(Element:0x78 {
          name = "body",
          children = [
            #(Element:0x8c {
              name = "div",
              children = [ #(Text " this < that ")]
              })]
          })]
      })]
  })

# nokogiri 1.13.2
$ ruby -rnokogiri -e 'pp Nokogiri::HTML4::Document.parse("<div> this < that </div>")'
#(Document:0x3c {
  name = "document",
  children = [
    #(DTD:0x50 { name = "html" }),
    #(Element:0x64 {
      name = "html",
      children = [
        #(Element:0x78 {
          name = "body",
          children = [
            #(Element:0x8c { name = "div", children = [ #(Text " this ")] })]
          })]
      })]
  })

[flavorjones]

I've opened an issue upstream: https://gitlab.gnome.org/GNOME/libxml2/-/issues/339

I'm going to explore reverting the related commits in a patch to see if I can get a fast-follow release of Nokogiri for y'all.

[flavorjones]

I've updated this issue's description with a punch list of next steps.

[flavorjones]

v1.13.3 has been released to address this: https://github.com/sparklemotion/nokogiri/releases/tag/v1.13.3