Privacy Badger triggers 404 errors on web servers.

[InternalError503]

Hello

I am contacting you today about an issue a few users have encountered running your addon, Your
addon caused a few users to get banned from my server by its anti-404 error detection system.

Your addon triggered enough 404 errors in a time period that caused the user to receive a 403
forbidden message and be permanently unable to access my service, This system is designed to
prevent bad robots, spiders and other nefarious actions not regular users and done its job correctly
so no fault by the system in place.

The issue comes from this section of code

policyCheck.js line:145

function checkPrivacyPolicy(host, callback) {
  let success = false;
  let policyUrl = "https://" + host + "/.well-known/dnt-policy.txt";
  if (!policyHashesExist()) {
    console.debug("Not checking privacy policy because there are no acceptable hashes!");
    callback(success);
    return;
  }

  // Cookies are stripped from all DNT policy responses in onExamineResponse.
  new Request({
    url: policyUrl,
    contentType: "application/json",
    onComplete: function(response) {
      let success = false;
      let status = Number(response.status);
      if (status >= 200 && status < 300) {
        success = isValidPolicy(response.text);
      }
      callback(success);
    },
    anonymous: true
  }).get();
}

The request that triggered the issue was /.well-known/dnt-policy.txt, So as the user browsed the site the 404 errors built up then triggered the detection system, You need to only check once per host not 50+ times in under 2 minutes as it did in this case.

This may have triggered other systems on other web servers causing users of your addon the inability to access a web service temporally or permanently without warning or knowing the cause and or why.

I hope you will address this issue before more innocent users encounter it.

[ztefn]

I'd like to add a "me too". Saw this in my iThemes Security logs:

Host or User Lockout    2016-06-05 04:09:23 xxx.114.242.219
404 Error   2016-06-05 04:09:22 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:22 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:21 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:20 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:19 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:19 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:18 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:18 xxx.114.242.219     /.well-known/dnt-policy.txt
404 Error   2016-06-05 04:09:16 xxx.114.242.219     /.well-known/dnt-policy.txt

If this is indeed an innocent user and not a robot, it's now locked out from my website thanks to this crappy add-on trying to request a useless file every second or less!

[InternalError503]

@ztefn

I can confirm in localhost the issue when using the iThemes Security plugin with privacy badger, Same issue with my server system for 404 detection.

This could be used to fingerprint privacy badger users or track them across them web as they leave a trail of 404 errors for /.well-known/dnt-policy.txt funny how a anti-tracking\privacy addon makes the users easily identifiable from the masses.

One could also detect the 404 for /.well-known/dnt-policy.txt and show a carefully crafted webpage to that user claiming there version of privacy badger is outdated and offer them a payload all because it can be fingerprinted by 404 errors for /.well-known/dnt-policy.txt

[mario-areias]

Hi @cooperq

I was trying to have a look on this, but I wasn't able to see where that many requests came from. The function @InternalError503 specified was supposed to be used either when a origin is blocked (so then we check if there is a DNT policy in place) or to recheck if any of the origins blocked in the past added the DNT policy at some point. In both scenarios only one request should be done by each origin and repeated requested should only happen after days.

So I am really have no idea what's causing this behaviour. I would really appreciate If you could give some direction in where the problems is, so I can fix it.

[InternalError503]

@mario-areias

My assumption on how this issue works is when you load a page for example:
The index.html includes links to the following items.

• Two css style sheets
• Three javascript files.
• Ten images.

The request the index.html makes is including itself, 16 individual requests, With each of these requests privacy badger checks for /.well-known/dnt-policy.txt

Each of the requests in the index.html all load within milliseconds of each other, So the requests for /.well-known/dnt-policy.txt are milliseconds of each other as well causing 16 404 errors on the server due to the file not existing.

Now as you browse the other pages attached to index.html more and more 404 errors build up in such a quick succession the issue with 404 error detection systems occur.

This is all a guess as i have not looked that far into how PB works or its code and i could be way off the mark and wrong.

[cooperq]

This should be fixed by the move to web extensions, we will only check the domain once per week. Won't fix in the legacy branch, but I will accept pull requests!