CredHub interactions take 97% of ATC CPU time

[DanielJonesEB]

Bug Report

Given a Concourse with secrets backed by CredHub, we were seeing 97% of the ATC's CPU time going to TLS handshakes when it was looking up secrets.

This is because credhub-cli (understandably, as a CLI library) was not using HTTP Keepalive, forcing each secret lookup to use a new connection, and this a new TLS handshake. The issue was fixed in this PR: cloudfoundry/credhub-cli#45

This has since been fixed as of commit cloudfoundry/credhub-cli@0887387. Bumping to this version will see a significant performance improvement for those using CredHub.

• Concourse version: 3.14.1
• Deployment type (BOSH/Docker/binary): BOSH
• Infrastructure/IaaS: AWS
• Did this used to work? No.

Before:

After:

Difference:

[jama22]

Pretty sure this is similar, if not the same, as #2300

[DanielJonesEB]

'ello @jama-pivotal! We think this is orthogonal. Proper engineers wot write proper code could tell you more, but it's to do with the way the stream is being read or something, so instantiation isn't relevant (EDIT: unless it establishes a connection on construction?).

[takeyourhatoff]

@jama-pivotal Hi, without the fix for #2300 then cloudfoundry/credhub-cli#45 will have no effect on ATC. They are two separate problems however. I actually had to downgrade concourse to v3.13 to expose this issue. The reason #2300 was so bad was because as part of the client construction credhub-cli reads and parses all of the system CA files.

With neither issue fixed, the flame graph is all parsing system root certificates, with #2300 fixed, it's all in TLS handshakes to credhub, and with both fixed ATC can actually get on and do work 🎉

[DanielJonesEB]

Hey @jama-pivotal,

We (read: @takeyourhatoff) just realised the fix in #2300 doesn't work because the method that does the lazy initialisation had a non-pointer receiver, so was acting on a fresh instance of the lazyCredhub struct each time.

We're just testing a fix for that, so disregard this until we've had a chance to address #2300. We'll verify it with a profiler before raising a PR.

[jama22]

thanks for the context @takeyourhatoff @DanielJonesEB . Gonna watch the PR closely for the fix, looking forward to it 🙏