Build controller lists every build config on every image change

[smarterclayton]

Build image_change_controller.go, on every image stream update, walks every build config. In high density runs, roughly 1/2 the CPU load is in that controller.

    // TODO: this is inefficient
    for _, bc := range c.BuildConfigStore.List() {
        config := bc.(*buildapi.BuildConfig)

And the CPU top:

132.59s of 153.42s total (86.42%)
Dropped 1282 nodes (cum <= 0.77s)
      flat  flat%   sum%        cum   cum%
    26.57s 17.32% 17.32%     67.52s 44.01%  github.com/openshift/origin/pkg/build/controller.(*ImageChangeController).HandleImageRepo
    20.25s 13.20% 30.52%     20.25s 13.20%  runtime.memeqbody
     8.56s  5.58% 36.10%     17.54s 11.43%  strings.genSplit
     5.28s  3.44% 39.54%      5.28s  3.44%  runtime.heapBitsSetType
     5.16s  3.36% 42.90%      5.16s  3.36%  runtime.futex
     5.03s  3.28% 46.18%      5.03s  3.28%  runtime.duffcopy
     4.87s  3.17% 49.35%      4.87s  3.17%  runtime.aeshashbody
     4.61s  3.00% 52.36%      9.38s  6.11%  runtime.mapiternext
     4.57s  2.98% 55.34%      9.79s  6.38%  runtime.scanobject
     4.41s  2.87% 58.21%     15.16s  9.88%  runtime.mallocgc
     3.84s  2.50% 60.72%      5.93s  3.87%  runtime.heapBitsSweepSpan
     2.86s  1.86% 62.58%      3.10s  2.02%  syscall.Syscall
     2.53s  1.65% 64.23%      2.53s  1.65%  runtime.heapBitsForObject
     2.38s  1.55% 65.78%      2.38s  1.55%  runtime.memmove
     2.33s  1.52% 67.30%      3.98s  2.59%  runtime.mapaccess2_faststr
     2.01s  1.31% 68.61%      2.06s  1.34%  runtime.(*mspan).sweep.func1
     1.86s  1.21% 69.82%      2.71s  1.77%  runtime.greyobject
     1.62s  1.06% 70.88%     11.11s  7.24%  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache.(*threadSafeMap).List
     1.62s  1.06% 71.93%     10.85s  7.07%  runtime.makeslice
     1.58s  1.03% 72.96%      1.58s  1.03%  runtime/internal/atomic.Xchg
     1.55s  1.01% 73.97%      1.55s  1.01%  runtime.usleep
     1.54s  1.00% 74.98%      8.16s  5.32%  runtime.(*mspan).sweep
     1.22s   0.8% 75.77%      1.22s   0.8%  runtime.memclr
     0.96s  0.63% 76.40%      8.80s  5.74%  runtime.findrunnable
     0.95s  0.62% 77.02%      2.18s  1.42%  runtime.pcvalue
     0.92s   0.6% 77.62%      0.92s   0.6%  math/big.addMulVVW
     0.92s   0.6% 78.22%      0.92s   0.6%  runtime/internal/atomic.Load
     0.85s  0.55% 78.77%      0.85s  0.55%  runtime.epollwait
     0.81s  0.53% 79.30%     11.50s  7.50%  runtime.newarray
     0.74s  0.48% 79.78%      2.42s  1.58%  runtime.runqgrab
     0.70s  0.46% 80.24%      3.30s  2.15%  runtime.lock
     0.56s  0.37% 80.60%      3.42s  2.23%  runtime.gentraceback
     0.42s  0.27% 80.88%     18.01s 11.74%  github.com/openshift/origin/pkg/image/api.SplitImageStreamTag

We'll need to switch to using an indexed BuildConfigStore that allows us to lookup by their trigger.

[bparees]

"we're number one! we're number one!"

[timothysc]

lol..

/cc @jeremyeder @mffiedler

[0xmichalis]

We'll need to switch to using an indexed BuildConfigStore that allows us to lookup by their trigger.

@bparees the index is already there, you just need to add support for build configs:

origin/pkg/client/cache/index.go

Line 17 in 14a1ed8

// TODO: Add support for build configs

(and switch the controller to use an indexer ofc)

[smarterclayton]

This is p1 for 1.4 - I'd like it in as soon as possible so we can get a jump on 1.4 perf testing.

[timothysc]

/cc @mffiedler fyi.