feat: improve estimated disk usage

[morph-dev]

What was wrong?

We underestimate the disk usage. See #1653 for details.

How was it fixed?

Based on the analysis in #1653, I added fixed number of bytes per content item that should be used when estimating disk usage (together with already counted number of bytes for content_id, content_key and content_value).

This way, the data that is actually store in the db (content_size = content_id.len() + content_key.len() + content_value.len()) doesn't have to change and we can easily update our estimate in the future if we need to.

How was it tested?

I had local trin client that was running for some time (~20h) last week. Storage was configured to use 1GB of data, and all 3 subnetworks are enabled (state and history had 500MB capacity each). The actual size of the DB was 1447 MB (45% than desired).

Starting trin client with this PR would trigger pruning at startup. Pruning would run until our estimated disk usage is below 1GB. This however doesn't affect the actual disk usage. To do so, we would have to run "VACUUM".

Before running this code, I made a copy of db. That way I was able to re-run the code, and confirm that results are stable.

Run 1: Prune + Vacuum

I started new trin client (with this PR) and observed that:

• estimated disk usage was:
  • 1063 MB for state + 526 MB for history (compared to 500 MB each before) = 1590 MB (compared to actual 1447 MB)
• pruning executed at startup
• estimated disk usage after pruning: 498 MB for state + 495 MB for history (both under specified capacity)
  • the actual DB size didn't decrease because we don't use Vacuum
• I stop trin and run Vacuum manually
  • the db size decreased to 785 MB

Run 2: Vacuum + Prune + Vacuum

I run Vacuum on the original DB which decreased db size to 1154 MB.
Then I repeated steps from the first run and observed that:

• The estimated disk usage was the same (as expected)
• Pruning ended with the same estimated usage
• DB size after pruning was still 1.154 MB (expected, again because of the vacuum)
• DB size after vacuum was also 785 MB.

Conclusion

It's worth observing that running Vacuum on the original db decreased it's size from 1447 MB to 1154 MB. That means that defragmentation added ~25% disk usage. If we assume the same defragmentation, resulting db would occupy 984 MB (instead of 785 MB), which would leave us under close to the target capacity (with a bit of room).

This defragmentation ratio is not fixed. It depends on the content that is stored (it's different between state and history content) and how long client is running (longer running client tends to have higher defragmentation, but probably there is a limit). The original DB had radius between 2% and 4% for both history and state network, which implies that there was significant defragmentataion (if I remember correctly, I run the code before the max 5% radius PR was merged), which would imply that defragmentation might not be significantly higher even in the worse case.

Things to consider

This would result in all our nodes (or at least the ones that reached their capacity) to trim content at startup. Different nodes are affected differently:

• history nodes - they wouldn't be significantly affected, as disk usage estimate wasn't as bad (in comparison to state)
  • in my local example, I deleted only ~6% of history content
• regular state nodes - Since they were restarted recently with 5% max radius, they didn't reach they capacity yet. This change would just change how much capacity they estimated they use, but they wouldn't prune anything (unless they get full by the time this PR is deployed)
• big storage nodes - These would be impacted the most because their defragmentation is very low (so far they only insert and never delete content). Running this PR would cause them to delete 1/3 of their content, which would:
  • run pruning very long (making them unresponsive for quite some time)
  • leave it with significant overestimated disk usage (I believe good in the long run)