[get.jenkins.io/mirrors/mirrorbit - Azure] High costs due to usage of Azure File Storage

[dduportal]

Service(s)

get.jenkins.io, mirrors.jenkins.io, pkg.jenkins.io, Update center

Summary

While checking the sources of costs in Azure for the year 2023, it appeared that the storage account prodjenkinsreleases (in the release group prod-core-releases) costs us around 20-25% of the monthly bills.

The amount is around 1800-2000$ monthly which is insane for a 1 Tb shared file storage.

Most of the cost comes from "LRS Write Operation" meter (from 1000$ up to 1700$ in the past months), followed by "Read Operations" (~180-> 200$) and "Protocol Operation" (~170 -> 190$) [costs per month]. The storage in itself is really cheap: ~30$ monthly + 6$ "hot" (e.g. caching data often read in the filer).

This issue is to track the analysis and study to see if we can decrease this cost one way or another.

---

The usage of this storage is:

• get.jenkins.io (mirrors.jenkins.io):

  • Mirrorbits reads the volume content to check files considered as "references" to compare to mirrors when scanning. It primilarly scan the FS tree hierchically and the inodes metadatas (name, checksum, date, etc.) as regular intervals.
  • Apache (httpd) reads file when used as a fallback from the load balancer (e.g. either when serving non-files resources such as directory listings or when accessed through fallback.get.jenkins.io).
  • Both services are replicatd for high availability: mirrorbits and httpd). We have 2 replicas with a budget of "always 1 running" maintained by Kubernetes for rolling updates and maintenance.
  • Replication + services split between httpd requires concurrent accesses to the same file storage by the 4 processes. We used to have a ReadWriteMany persistent volume to achieve this but we are going into a full ReadOnlyMany mode soon as no write is needed.
    • Azure File Storage is the only CSI volume type supporting one or the other (e.g. mounting the same volume on different nodes).

• trusted.ci.jenkins.io:

  • Every 3 minutes, the jenkins-infra./update_center2 project is run an ends in executing (remotely on the pkg/updates.jenkins.io VM) the script sync-recent-releases.sh which copies recently released plugins to the storage (with blobxfer or eventually, soon, azcopy)

• release.ci.jenkins.io:

  • During Core releases (weekly every Tuesday and LTS + Security), the publication process executes (remotely on the pkg/updates.jenkins.io VM) the script sync.sh which copies recently released binaries (plugins, core packages) and indexes to the storage (with blobxfer or eventually, soon, azcopy)

=> Both trusted.ci and release.ci usages are reponsible for writing data. They can by using commands remotely on pkg.jenkins.io (AWS VM) which is allowed to access the storage through the Azure storage API. This pattern (writing avery 3 minutes) is clearly the culprit for the cost here.
=>

---

Storage consideration (usage/price):

• This storage is bound to always grow: we do not garbage collect data as we want to keep it forever.
  • This policy could be revised as we have archives.jenkins.io.
• Current usage today is:
  • Storage uses around 520 Gb.
  • Current maximum IOPS are 1000/s
  • Current maximum bandwitdh is 60 Mbits/s for ingress, same for egress

Reproduction steps

No response

[dduportal]

Proposal:

Since the original implementation of get.jenkins.io with mirrorbits, many things changed:

• The amount of replicas for get.jenkins.io have been decreased to 2 which means only 2 "visions" of the persistent storage are needed
• As we recently worked on parallelization of remote copy operations on the update center, the overhead time to copy on 2 copies to a block storage are now acceptable
• The flow of copies (azure -> AWS -> azure) changed quite often in 2020/2021 but were not revisited since then

We could try switching to using 2 PVCs of type block storage instead of the current Azure File Storage:

• We would need to implement a set of "write to disk" services to provide entrypoint to write to the disks

  • Luckily, the recent work of @hervelemeur added rsyncd as either included in the mirrorbits-parent chart or as a distinct chart with a ReadWriteOnce policy
  • We would need to replace blobxfer calls by rsync calls:
    • Use SSH key based authentication so it would be easier to run rsync from pkg/update VM (and eventually release.ci pipeline for core release)
    • Parallelize the copy of the 2 rsyncd copies on each script (to avoid reaching the 3 min limit on update center)
    • Restrict IPs at LB level to increase security

• Scheduling constraint: due to the ReadWriteOnce policy, for each get.jenkins.io replica (e.g. a set of 1 mirrorbits pod, 1 httpd pod and 1 rsyncd pod) would be schedule on the same machine

  • Not an HA problem: we can keep 2 distinct machines, each one for 1 "replica" with its own PVC
  • However, we would need to roll back httpd and rsync to using x86_64 nodes instead of arm64 nodes, at least until mirrorbits is amr64 compliant. Not a billing problem if we "pack" properly

• Pricing changes:

  • As per https://azure.microsoft.com/en-us/pricing/details/managed-disks/, a Premium SSD of 1 Tb (512 Gb is the smaller option available below 1 Tb) is $184.32 monthly with 5000 IOPS (burstable) and 200 MB/s. There could be "outbound bandwidth" billed per Gb but only if we read the disk from another availability zone: if we use LRS we are fine.
  • That means we could expected a static billing of ~400$ monthly (usage and error margin included) instead of ~1800 / 2000 today ! Even including the engineering effort it is clearly worthwhile.

Besides switching to a block storage-based persistent volume would:

• Fix Past Release sites are taking long time to load #3525 (slow directory listing by Apache).
• Drastically reduce the scope of Check if we could replace blobxfer by azcopy #3414

[dduportal]

Alternative Proposal:

• As per https://azure.microsoft.com/en-us/pricing/details/storage/files/, switching to a "Premium Storage" would remove the operation costs (marked as "included"):

Premium file shares use a provisioned billing model, where you pay for the amount of storage you would like your file share to have, regardless of how much you use

This solution could be better than my disk proposal (less engineering effort) as the volume billing would be:

• 164$ monthly (0,16*1024 for premium)
• Snapshot costs would drastically increase though: (0,136$ / Gb instead of 0,06$ / Gb Today). As we already copy data in pkg VM and archives (2 other different locations), this is acceptable.

=> It would also allows us to use NFS file share for our volumes which could greatly improve #3525

Gotta check if the conversion from the existing volume is possible and under which constraints

---

(edit)

Looks like a migration is required to a new distinct Storage account: https://learn.microsoft.com/en-us/answers/questions/413129/how-to-migrate-aure-standard-storage-to-premium-st

Still worth the effort

[dduportal]

Update: PR opened to create new storage (premium): jenkins-infra/azure#598

[smerle33]

as per https://azure.microsoft.com/en-us/pricing/details/storage/files/

the ZRS Redundancy is a good choice, the price is a little higher than LRS but still cheap (in Premium version) as we consume only 600Go:

LRS:

ZRS:

And the write are included as premium:

[dduportal]

Update:

• The new (premium) SA and its (700 Gb) file storage has been created with success
  • feat(get.jenkins.io/storage): new premium storage class for get.jenkins.io azure#598
  • fix(get.jenkins.io) correct SA parameter to allow creation azure#607

Next step:

• Start copying the data between the 2 storage accounts (should take the night)
• Test a new PVC pointing to this storage with NFS in publick8s
• Test access from pkg.origin.jenkins.io

[dduportal]

Update (Start copying the data between the 2 storage accounts):

• Data is being copied:
  • Executed in a screen named helpdesk-3917 in pkg.origin.jenkins.io's VM.
  • azcopy existed in /bin/azcopy and was upgraded to latest 10.x version.
  • Generated an SAS (with only file and read/list permissions) on prodjenkinsreleases valid only 5 days.
  • Command is azcopy copy 'https://prodjenkinsreleases.file.core.windows.net/mirrorbits?sv=2022-11-02&<redacted>' 'https://getjenkinsio.file.core.windows.net/mirrorbits?sv=2022-11-02&<redacted>' --preserve-smb-permissions=true --preserve-smb-info=true --recursive

[dduportal]

Update (Test access from pkg.origin.jenkins.io and Keeping storage accounts in sync):

• Starting the "parallel" update of both storages to keep the sync:
  • Added the new credentials in pkg.origin.jenkins.io:/home/mirrorbrain/.azure-storage-env
  • Tested the bloxfer access from pkg.origin.jenkins.io ✅
  • Updated the "mirror-scripts": jenkins-infra/mirror-scripts@13e8842 to copy on both storages (and also a bit of shellcheck and [INFRA-3125] Migrate jenkins-infra repositories from branch "master" to "main" #2671 (comment))
  • Currently watching the /srv/release/last_sync.log

[dduportal]

WiP (Test a new PVC pointing to this storage with NFS in publick8s):

• Helm charts:
  • feat(mirrorbits) use readonly mounts for data volume helm-charts#1037 to ensure a read-only mount of the data storage in mirrorbits pods
  • Then Bump mirrorbits-parent subchart versions helm-charts#1039
  • And [get.jenkins.io/mirrors/mirrorbit - Azure] High costs due to usage of Azure File Storage #3917 which is blocked as we need to validate all of this!
• Kubernetes management:
  • WiP on creating a "manual" installation of mirrorbits to verify everything works. A draft PR in jenkins-infra/kubernetes-management is expected but it will never be merged! using the new chart (with read only). Main points:
    • Use another Redis base (but same server) to avoid any impact on get.jenkins.io
    • Use the new (premium) storage
    • Use NFS(v4) instead of CIFS for the PVC
    • Use read-only mounts
  • ToDo: once the installation (draft PR + manual installation + manual cleanup) is considered ready, then a PR to migrate get-jenkins-io is expected, with announcement

[dduportal]

Update: production migration to be started:

• Work on WIP(do not merge) configure a temporary instance of get-jenkins-io for helpdesk-3917 kubernetes-management#4909 showed that the new Storage accounts works perfectly in publick8s if we keep the SMB/CIFS mode (NFS is another story).
  • Cleaned up my IP from the vault allow-list
  • Cleaned up the test DNS records
  • Cleaned up the test namespace and PV
• PR opened feat(get.jenkins.io) Migrate to the new premium storage account kubernetes-management#4911 to track the code change to apply (will be done manually to ensure secrerts are also applied at the same time without breaking the main production
• Communication sent for operation (ref. open helpdesk-3917 maintenance window status#465)

[dduportal]

Update: Operation on get.jenkins.io finished with success:

• feat(get.jenkins.io) Migrate to the new premium storage account kubernetes-management#4911 applied locally and then merged
  • Secrets updated: https://github.com/jenkins-infra/charts-secrets/commit/7caaea2736c26229f85bc8e42fd5b8bb477a3248
  • Had to delete the whole get-jenkins-io namespace as the PV is immutable
• Logs of get.jenkins.io are 👍
• Health Check: https://get.jenkins.io/windows-stable/2.426.3/jenkins.msi?mirrorlist is 👍
• Last build of jenkins-infra/kubernetes-management ran with success (build 29684) 👍
• Ran ci.jenkins.io plugins update + a local build of jenkins-infra/docker-jenkins-weekly images to ensure plugins download are 👍

Next steps:

• Close operation (and communication)
• Remove access to prodjenkinsreleases from PKG
• Cleanup prod-core-releases resource group
• profit

[dduportal]

Proceeding to deletion of prod-core-releases resource group which only has the storage account prodjenkinsreleases with 2 file shares: mirrorbits (replaced by getjenkinsio's mirrorbits file share) and an unused website file share:

=> This storage account has not used since the past 15 min and no error are seen on update center, trusted or PKG VM.

[smerle33]

usage of the new premium storage class

[dduportal]

=> Looks like the outcome is really good: the daily rate in the range "70 to 110" $ daily is now ~5 $ daily for get.jenkins.io!

Closing the issue as complete, but there should be 1 new issue to automatically track the storage size from jenkins-infra/azure to the Kubernetes PVC size + 1 comment to have updates.jenkins.io changed with the same pattern (premium storage).

[dduportal]

See #3913 (comment) for overall costs

[lemeurherve]

Proceeding to deletion of prod-core-releases resource group which only has the storage account prodjenkinsreleases with 2 file shares: mirrorbits (replaced by getjenkinsio's mirrorbits file share) and an unused website file share:

Likely the cause of #3927