normalize method can't handle URLs with punycoded TLD

[walro]

[12] pry(main)> Twingly::URL::Normalizer.normalize("http://xn--80aesdcplhhhb0k.xn--p1ai/")
=> []

Page loads fine in Chrome though, whois works fine too.

[walro]

It was pointed out that the TLD is really funky here.

[dentarg]

Strange, xn--p1ai (or рф) is in the public suffix list

I tested too, with public_suffix 1.5.1, got the same as above.

Don't really know of public_suffix work, if that above should be enough to support xn--p1ai or if they are missing something.

Could it be an encoding issue?

[walro]

Could it be an encoding issue?

Yeah, to get it working (with public-suffix) we need to go from punycode back to utf-8:

irb(main):005:0> PublicSuffix.valid?("xn--80aesdcplhhhb0k.xn--p1ai")
=> false
irb(main):006:0> PublicSuffix.valid?("domain.рф")
=> true

Public suffix won't add support for it: weppos/publicsuffix-ruby#24

We could use https://github.com/mmriis/simpleidn (I found other, even less maintained, alternatives too) to do this ourselves.

[jage]

We should analyze our data and how many punycode TLDs do we have.