Optimise plugin disk space in Jenkins

[chmouel]

The plugins directory consume over 383Mb out of 478mb and is copied over and over everywhere which cause a scale issue on the gluster volumes,

Longer discussion here :

https://chat.openshift.io/developers/pl/ic6z8bppkbf8jb78z8bhsoispe

[chmouel]

More discussions and investigation here:

https://chat.openshift.io/developers/pl/sm4bacg41iye3bwn3cau4hwr7r

patch to symlink the hpi/jpi instead of copying fabric8-jenkins/jenkins-openshift-base#49 saves around 187mb

[chmouel]

Symlink directory into the image saves us 358MB in total,

[ppitonak]

"In total" is confusing... Is it 358 MB per user or per cluster?

How will symlink handle various versions of Jenkins?

[chmouel]

"In total" is confusing... Is it 358 MB per user or per cluster?

there is no global plugins 'folder' in our infrastructure so it is per user,

How will symlink handle various versions of Jenkins?

I am not sure i understand your question, did you read the patch?

[hrishin]

Some observations for the sym link patch

Plugin management:

1. installed artifcatory plugin from plugins.txt
   plugin worked ✔️
   plugins size

First boot

sh-4.2$ du -hs /var/lib/jenkins/plugins/
247M    /var/lib/jenkins/plugins/

2. installed prometheous plugin (without restart option) from Jenkins admin conosle
   plugin worked ✔️
   plugins size

First boot (without restarting jenkins)

sh-4.2$ du -hs .
216M	

3. installed MSBuild plugin (with restart option) from jenkins admin console
   Plugin worked ✔️
   plugins size

After the first boot but jenkins process restarted in running pod

217M	/var/lib/jenkins/plugins/

---

after second boot (scaling jenkins pod down and up)

without rm -rf plugins

sh-4.2$ du -hs /var/lib/jenkins/plugins/
26M     /var/lib/jenkins/plugins/

with rm -rf plugins

sh-4.2$ du -hs /var/lib/jenkins/plugins/
247M    /var/lib/jenkins/plugins/

Basic test

1. Run the pipeline builds
   In all cases it worked well ✔️

[chmouel]

Thanks for all of those tests @hrishin, is there any chances you can try another scenario if that's not too hard to test :

boot with a version of a plugin and reboot with an updated version of a plugin and see if it pickups the new version?

[hrishin]

Yes @chmouel we have tested by deploying a new version of sync-plugin, Jenkins was able to pick it up.

I was not able to run pipelines, as jenkins is behaving wiredly due to idler issue. Rest of behavior stays same as per previous observation.

[chmouel]

As for doing the proper way when we merge the image with upstream here is some thought out of this discussion from https://chat.openshift.io/developers/pl/n3z1jyxh93b33cw1zdgayp5oce :

[piyush-garg]

Tenant update has been done. Closing this,

[hrishin]

@pbergene could we confirm if this issue has fixed with recent tenant update? It should drop the gluster storage usage(Not able to make grasp looking at Zabbix). If not feel free to reopen this issue. Thank you

[hrishin]

Can see some reduction in gluster storage usage.

@chmouel @sthaha

[chmouel]

Really nice! 😍

Great work 💪

[pbergene]

Good job indeed! We have a couple of graphs which show it even more dramatically. The device space also contains snapshots and is not a direct measure of space used by PVs, but we can clearly see the updates go in with this one :)

[hrishin]

Thanks @pbergene ! for giving confimation.

[chmouel]

@hrishin We need to plan for merging a better looking (i.e: not hack) patch to jenkins upstream, since they are affected as well and we want to consume their images asap,

/cc @arilivigni

[hrishin]

Yes @chmouel.
Filed an issue fabric8io/openshift-jenkins-s2i-config#232 for it.