Security 2021 custom metrics

[GJFR]

Progress on HTTPArchive/almanac.httparchive.org#2150.

• robots.txt
• /.well-known/security.txt

Also includes renaming and extending ecommerce custom metric for well-known URLs as per HTTPArchive/almanac.httparchive.org#2211.

[rviscomi]

Is this ready for review?

[max-ostapenko]

@GJFR Could you please add, as ,emtioned in #211

// privacy
parseResponse('/.well-known/gpc.json', r => {
  return r.text().then(text => {
    let data = {
      'gpc': null
    };
    let gpc_data = JSON.parse(text);
    if (typeof gpc_data.gpc == 'boolean') {
      data.gpc = gpc_data.gpc;
    }
    return data;
  });
}),

[GJFR]

@max-ostapenko Your code has been added 👍

@rviscomi Thank you for your comments! I've updated the code. I'm gonna do a quick check and tweak the robots.txt data collecting. Will mark as ready for review ASAP.

[SaptakS]

❤️

[GJFR]

WPT test runs:

• Test google.com
• Test facebook.com
• Test github.com
• Test www.ieee.org

[SaptakS]

WPT tests look good to me.

[GJFR]

I filtered on all keywords discussed in this thread.

A few thoughts:

• If we are not that interested in User-agent entries in itself, we could remove reported user-agents that do not have any matched Disallow paths to reduce clutter.
• As shown in the test results below: author seems a popular false positive for auth. I'm sure we will encounter similar false positives in the crawl data. In my opinion, we should make the filter more strict in the querying stage and not on the custom metric level for two reasons:
  • We could overtighten the filter and miss out on endpoints. It's safer to try out more strict filters when we have the crawl data.
  • It's more practical to put as less filtering logic at custom metric level as possible. For example, if later on this metric will be used in a longitudinal analysis spanning multiple crawls, it will be more difficult to get it right if the custom metric filter has been adjusted between crawls. Not that familiar with the database/querying costs, though, so can't say whether it will be worth the (limited?) additional cost.

WPT test runs:

• google.com
• facebook.com
• github.com
• ieee.org

[rviscomi]

LGTM thanks everyone!