Memory leak on full nodes and validators

[tkporter]

Expected Behavior

Memory usage by tx-nodes and validators remains constant

Current Behavior

There seems to be a memory leak that's being seen on validators and tx-nodes (though not as extreme).

Example validator:

Example tx-node:

Here's a pprof rendering from a tx-node:

I wasn't able to find anything super conclusive, just that levelDB looked suspicious. Planning on running my own testnet with --pprof and seeing if there is anything noticeable

[mattharrop]

My validator node consistently crashes after a few hours running on an instance with 8gb RAM. Logs from the crash attached.
celocrash.log

[tkporter]

Looks like discv5 is the one to blame:

(Note it doesn't seem like the sum of all the entries comes out to 631 MB, so there must be some inaccuracy here ^)

[daithi-coombes]

This issue is very very similar: ethereum/go-ethereum#16859
and should've been fixed in: ethereum/go-ethereum#16880

from @mattharrop logs above, there's plenty of goroutines hanging for 2hrs

 goroutine 1 [chan receive, 160 minutes]:

Also in network logs there's false positives about bad enode urls:

Error in upserting a valenode entry      AnnounceMsg="{Address: 0x132EA61AaE41
A3beE6a2faf45aAE68e2A7f9457F, View: {Round: 0, Sequence: 496530}, IncompleteEnodeURL: enode://c8f8bc50da
1780163761d738a1cbed328df7b82900e9d7ada38fb505fe589af60982b91e9b950a75bf677fabaafdf409fe9d2c0f9ead523946
70ed5f1da7f5eb, Signature: f3d205bd4a570b95671c9afba818d73b27cb62fb21376c7a184354b7f9e4745f3d65d4b8aabee
3621abe115ac099539037949771e4d6497dd72ac6ffe351d83001}" error="old announce message"

(am basing false positive on `error="old announce message")

am thinking dangling requests in discovery are the cause, for now. Will post heapdump as per this instructions: #573 (comment)

[trianglesphere]

Closing because it seems like issue #1197 is the same, except with newer logs.