Add fingerprint for blockpages in MY

[hellais]

In https://github.com/TheTorProject/ooni-explorer/issues/96 @kaerumy pointed out a blockpage for Malaysia: https://explorer.ooni.torproject.org/measurement/20160802T205955Z_AS4788_3omRbM1JA9BYIMF5O5uiKEsdmUqy4kdunnKn7exzBlM2ebboDh?input=http:%2F%2Fwww.sarawakreport.org.

We should be adding a fingerprint for it to the data processing pipeline.

In more recent times this is done via DNS by resolving to the IP 175.139.142.25.

See:
https://explorer.ooni.io/measurement/20160817T033110Z_AS4788_jk5ghw4QwieT2JOFiIqto9Z2LzCFhP05v3U0sCcaetBr50NxuU?input=http:%2F%2Fwww.sarawakreport.org%2Ftag%2F1mdb

[kaerumy]

Does a fingerpint need to be added for each verified blocked site? If so should I provide a list known list of Inputs?

[hellais]

@kaerumy only if the blockpages are different, it would be useful though to have a list of the known blocked website if you have that available so I can verify that.

[kaerumy]

@hellais two more blocked sites confirmed by web connectivity tests

• Medium
  • AS4878 https://explorer.ooni.torproject.org/measurement/20160803T222511Z_AS4788_Pyv6z2tLMnilg00tJmElKqTESkwzoi3U48gdLvm049SpchSek6?input=https:%2F%2Fmedium.com
  • AS4818
    https://explorer.ooni.torproject.org/measurement/20160804T013556Z_AS4818_tJWqtRTFvZQoX7MkH0sO8x51WebyfPHthnPAPWdSns63OughMd?input=https:%2F%2Fmedium.com
• Asia Sentinel
  • AS4878
    https://explorer.ooni.torproject.org/measurement/20160803T222511Z_AS4788_Pyv6z2tLMnilg00tJmElKqTESkwzoi3U48gdLvm049SpchSek6?input=http:%2F%2Fwww.asiasentinel.com
  • AS4818 https://explorer.ooni.torproject.org/measurement/20160804T013556Z_AS4818_tJWqtRTFvZQoX7MkH0sO8x51WebyfPHthnPAPWdSns63OughMd?input=http:%2F%2Fwww.asiasentinel.com

Will add some more known blocked sites to the test list so they can be picked up by OONI

[kaerumy]

The following on global test lists are also detected by ooni tests and redirected to Malaysia Communications and Multimedia Commission (MCMC) notice page:

Porn:
http://adultfriendfinder.com
http://www.ashemaletube.com
http://www.hustler.com
http://www.livejasmin.com
http://www.sex.com
http://www.worldsex.com
http://xhamster.com
http://xvideos.com
http://youjizz.com

Gambling:
http://www.belmont.ag
http://www.betfair.com
http://www.carnivalcasino.com
http://www.spinpalace.com

http://sinarproject.org/en/projects/measuring-and-detecting-network-interference/updates/porn-and-online-gambling-sites-blocked-by-mcmc

[anadahz]

I'm not sure what what would be an accurate fingerprint for the blockpage, at first I thought about this:
Laman sesawang ini tidak dapat diakses di Malaysia but after a quick search it seems that it triggers a number of false positives.

It seems that the blockpage in AS4788, AS4818 and AS17971 have the following unique fingerprint:
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMTE0Mzk3ODk1N

[kaerumy]

This fingerprint will also fail if the test url includes additional page specific url and not just the domain. In this case, you will get a 404 from that server because, the notice server is badly configured. You will also get 404 if they haven't configured the vhost yet on newly added domains.

What is common however so far on 3 ISPs and 4 networks (3rd ISP tests for Malaysia should show in today's reports) is that for this notice server the IP address returned is always: 175.139.142.25 regardless of network or ISP DNS server.

The headers for this server serving notices is also consistent, it is always IIS/8.4 and ASP.NET

headers: {Content-Type: text/html, Date: 'Thu, 18 Aug 2016 08:54:06 GMT', Server: Microsoft-IIS/8.5,
      X-Powered-By: ASP.NET, content-encoding: ''}

http://www.sarawakreport.org
https://explorer.ooni.torproject.org/measurement/20160817T222431Z_AS4788_kPemLiRZuzSWmiw8ww2Orj7lpba3Zqm3YNdwLttj9M7pcgEjEq?input=http:%2F%2Fwww.sarawakreport.org

vs

http://www.sarawakreport.org/tag/1mdb
https://explorer.ooni.torproject.org/measurement/20160817T033110Z_AS4788_jk5ghw4QwieT2JOFiIqto9Z2LzCFhP05v3U0sCcaetBr50NxuU?input=http:%2F%2Fwww.sarawakreport.org%2Ftag%2F1mdb

[anadahz]

The 404 errors will be cached by the pipeline's filter http-diff (as in your example).
However since this is a very common bad practice to break the internet without the user even realizing that is censored it makes sense to create a global filter for that enumerates http-diff blockages with 404 errors.

Using the IIS/8.4 and ASP.NET headers will trigger a big number of false positives.

[kaerumy]

What I mean is that the following query:

ip: 175.139.142.25 AND headers: {Server: Microsoft-IIS/8.5, X-Powered-By: ASP.NET}

Is unique on 4 different networks, when there is an MCMC notice or 404. The block directive on how to block sites seems to be universal across ISPs ie. for X in domain, redirect to 175.139.142.25 which is an IIS/8.5 webserver.

I'm not sure how the fingerprinting works, but if it's by conditions, there should not be any false positives, this is a specific server IP and type of web server that all blocked sites of this type are redirected to.

[darkk]

Xref: citizenlab/blockpages#1 — where I mined MY blockpage from the data.

[anadahz]

Pushed a pull request with the most common blockpage fingerprint: #68
This fingerprint will detect and show all the ooniprobe reports withn blockpages in AS4788, AS4818 and AS17971.

@kaerumy in your #35 (comment) the blocked report shows a 403 HTTP error response which is not (currently) supported by the pipeline.

[hellais]

According to @kaerumy we should be using as a fingerprint for MY the fact that DNS resolves to the IP 175.139.142.25.

See:
https://explorer.ooni.io/measurement/20160817T033110Z_AS4788_jk5ghw4QwieT2JOFiIqto9Z2LzCFhP05v3U0sCcaetBr50NxuU?input=http:%2F%2Fwww.sarawakreport.org%2Ftag%2F1mdb

[hellais]

Added these to: #289

[hellais]

Added these to: #289