firefox connections last ten minutes

[bemasc]

Currently, it seems like Firefox (as client) may get bogged down, and eventually disconnect, when in heavy use (i.e. loading several pages simultaneously). More investigation is needed to determine whether this problem is reproducible.

[trevj]

Can you describe your setup?

[bemasc]

Hmm. It's possible that I was not using a new enough build of freedom-for-firefox, and didn't have your fix for the very similar behavior we'd observed previously. I'll test again with the next cut.

[trevj]

Yeah, I''m browsing here with Simple SOCKS on Firefox and don't see any issues...a pretty beefy machine admittedly but, still. I'll continue browsing with it, see how it goes...

[trevj]

Hmm, correction: I am seeing my Firefox proxy die after ~5-10 minutes browsing. Trying to figure out what's happening...looks like I see heartbeat failures, then of course it self-terminates.

[trevj]

I don't know yet what's happening...nothing sticks out to me from the logs. If I had to guess, I'd say there's still some slowdown when there's large numbers of open TCP sockets -- this may happen when actually browsing with the proxy using an actual browser, as a browser typically tries to re-use proxy connections, keeping them open for potentially a long time.

[trevj]

FYI, I only see this with churn.

[bemasc]

EDIT: Moved previous hypothesis to #1830.

Current best thinking from Mozilla is that this is actually a PMTU Discovery bug in usrsctplib. Firefox operates with PMTUD enabled. Detailed tracing shows that something causes packet sizes to start growing, and once a packet reaches 1464 bytes, it can never be received.

This is running through CHURN, which interferes with PMTUD because usrsctplib's implementation is ICMP-based. There is a patch to disable PMTUD under review at https://bugzilla.mozilla.org/show_bug.cgi?id=1194817.

[bemasc]

It looks like this was only showing on localhost-only tests because FF40 and lower were truncating all packets at 1500 bytes. This is fixed in FF41, so we should make sure to test on real networks even if things appear to be working on localhost.

[trevj]

So, you're saying we should incorporate MTU into our tests to prevent false positives on FF 41+?

[bemasc]

Sure, that would be a good way to go.

[trevj]

OK, I've re-opened this issue to track adding MTU to some script we run regularly, e.g. release.py:
#1834

[trevj]

So, to summarise:

• we have improved uProxy's auto-reconnect feature
• there is a patch to disable PMTUD worming its way into Firefox
• this patch should make it into Aurora, i.e. Firefox 42.
• we will push a little to get this into Firefox 41 (seems reasonable to me, this is a pretty bad bug)

Do you agree, @bemasc?

[bemasc]

OK!

[bemasc]

The underlying Firefox issue has been approved to be merged into Beta, so it will land in stable on Sept. 22.

With the heartbeat timeout change and auto-reconnect (#1836) to mitigate the severity of this issue, I think we've done enough.

[trevj]

Amazing work!

💯 * 💯