Account for 64 KiB limit for sending beacon data

[westonruter]

This comes out of a discovery from #1311 (comment):

Well, it turns out that fetch() can now be used as a replacement for navigator.sendBeacon() when the the keepalive property is set (supported by all browsers). It seems that uBlock may block navigator.sendBeacon() for the sake of privacy, so using fetch() may be more reliable.

According to the spec, I'm just seeing that such keepalive requests have a POST body have a size limit of 64 KiB (65,536 bytes):

If the sum of contentLength and inflightKeepaliveBytes is greater than 64 kibibytes, then return a network error.

This is an aggregated limit, so if multiple keepalive fetch() requests are being made, then as soon as one is added which goes over the limit, then a network error will occur. Apparently for navigator.sendBeacon() there is also a limit of 64 KiB for all pending beacons on the page, just for one beacon.

I was not aware of this restriction on sending beacon data, but it makes sense. The browser shouldn't be sending huge payloads on pages that the user has closed. Nevertheless, this is a complication for Optimization Detective because a URL Metric can contain a lot of data. When there are 100 images on a page, this results in a URL Metric JSON body of that is ~53,371 bytes or 83% of 64 KiB limit. When there are 200 images, then the URL Metric is ~106,024 bytes, or 166% of 64 KiB limit. However, this 64 KiB limit is not for a single beacon but for all beacons combined. Therefore, if there are various analytics providers attempting to also send a lot of data, then this 64 KiB limit could be breached even when only 50% (32 KiB) of the limit is being used.

So this PR starts to address this issue by:

• Measure the length of the JSON that is pending being sent and include the length and the budget percentage in the console logs when debugging is enabled.
• If the size is larger than 64 KiB then the request is aborted entirely and, when debugging is enabled, an error is shown in the console.
• If the size is 50% or greater of the 64 KiB limit, then a warning is used in the console when the debug message is logged for sending the URL Metric.
• Add a TODO for how we might start to reduce the size of the JSON.

We can reduce the precision in the numbers to reduce the size somewhat. When there are 100 images, rounding the numbers to the nearest whole number reduces the size from 83% down to 68% of the budget. Other strategies could be employed, like coming up with a compression algorithm to replace common JSON strings. But we can explore that later as we start monitoring how close we are getting to the maximum. I need to look at my own site to see how close the captured URL metrics are to the 64 KiB limit for some real world data. Update: See #1851 (comment).

Screenshots

When there are 3 images:

When there are 100 images:

When there are 200 images:

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: westonruter <[email protected]>
Co-authored-by: felixarntz <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[westonruter]

I updated the Optimization Detective Admin UI plugin to add a METER element to show how large the URL Metrics are in relation to the 64 KiB limit:

The METER has a tooltip which says:

25,009 bytes (39% of 64 KiB limit)

Here is the page in question: https://weston.ruter.net/2009/12/12/google-tts/

This page has a larger URL Metric size because I got a lot of comments on this old post, so there are 42 images on the page (due to the Gravatars).

[felixarntz]

LGTM!

[westonruter]

@felixarntz I added one more thing here (6f96d20) to enforce this 64 KiB limit on the server in the REST API when processing a URL Metric storage request. Since no URL Metric should be larger than 64 KiB it makes sense to me that we should just plain not even accept it. This would be a way to harden against malicious payloads to pollute the od_url_metics post type, sent not by our detect.js but by some bad actor's fetch() or even via cURL.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.27%. Comparing base (68c6deb) to head (6f96d20).
Report is 19 commits behind head on trunk.

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk    #1851      +/-   ##
==========================================
+ Coverage   65.97%   66.27%   +0.29%     
==========================================
  Files          88       88              
  Lines        6895     6956      +61     
==========================================
+ Hits         4549     4610      +61     
  Misses       2346     2346              

Flag	Coverage Δ
multisite	66.27% <100.00%> (+0.29%)	⬆️
single	37.85% <0.00%> (-0.33%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[felixarntz]

@westonruter Thanks, that makes sense. LGTM!