https://updates.jenkins.io/current/latest/jenkins.war returns 404

[alextu]

Service(s)

get.jenkins.io, Update center

Summary

We've recently changed Jenkins download URLs to https://updates.jenkins.io/ because get.jenkins.io returned 404 at the time, see jenkinsci/gradle-plugin#387.
It looks like like now https://updates.jenkins.io/current/latest/jenkins.war returns 404 should we go back to get.jenkins.io based URLs ?

Reproduction steps

curl -I -L https://updates.jenkins.io/current/latest/jenkins.war

HTTP/2 302
date: Tue, 30 Jan 2024 14:51:04 GMT
server: Apache
location: https://updates.jenkins.io/download/war/2.443/jenkins.war
content-type: text/html; charset=iso-8859-1

HTTP/2 302
date: Tue, 30 Jan 2024 14:51:05 GMT
server: Apache
location: https://get.jenkins.io/war/2.443/jenkins.war
content-type: text/html; charset=iso-8859-1

HTTP/2 404
date: Tue, 30 Jan 2024 14:51:05 GMT
content-type: text/plain; charset=utf-8
content-length: 10
x-content-type-options: nosniff
strict-transport-security: max-age=2592000; includeSubDomains; preload

[MarkEWaite]

Thanks for reporting the issue @alextu .

The Jenkins update center, https://updates.jenkins.io/ includes a page https://updates.jenkins.io/latest/ that lists the preferred download links for the latest releases. For jenkins.war, that download link is https://updates.jenkins.io/latest/jenkins.war .

The list of mirrors for a specific download link can be viewed by appending ?mirrorlist to the preferred download link. Currently that will redirect to get.jenkins.io like this:

• https://updates.jenkins.io/latest/zulip.hpi?mirrorlist redirects to https://get.jenkins.io/plugins/zulip/2.1.1/zulip.hpi?mirrorlist which is the most recent release of the zulip plugin

Unfortunately, something is wrong with the preferred link to the latest release of Jenkins core

• https://updates.jenkins.io/latest/jenkins.war?mirrorlist redirects to https://get.jenkins.io/war/2.443/jenkins.war?mirrorlist but that currently reports HTTP 404

We'll need to investigate further in order to understand what went wrong with the 2.443 release. That release was built a few hours ago.

[lemeurherve]

Looking into it, there is an error mounting the volumes in the release job pod:

Mounting command: mount                                                                                                                                                                                                                            
Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,uid=1000,gid=1000,mfsymlinks,nobrl,actimeo=30,nosharesock,<masked> //prodjenkinsreleases.file.core.windows.net/mirrorbits /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com
/f5fe83d0cef11484xxxxxxbf58c3f958fa8bd9ca1a24ad9d1bd9524e1432c/globalmount                                                                                                                                                                      
Output: mount error: could not resolve address for prodjenkinsreleases.file.core.windows.net: Unknown error                                                                                                                                        
                                                                                                                                                                                                                                                   
Please refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.                                                                                                                                                   
  Warning  FailedMount  113s (x7 over 2m56s)  kubelet  MountVolume.MountDevice failed for volume "website-core-packages" : rpc error: code = Internal desc = volume(website-core-packages) mount //prodjenkinsreleases.file.core.windows.net/websit
e on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/a39f6c9618d704c0e13853726ddddf02c40e7ff3f44cc97c3d235ab6899d6f/globalmount failed with mount failed: exit status 1                                                            
Mounting command: mount                                                                                                                                                                                                                            
Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,uid=1000,gid=1000,mfsymlinks,nobrl,actimeo=30,nosharesock,<masked> //prodjenkinsreleases.file.core.windows.net/website /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/a3
9f6c9618d704c0eddddd60f02c40e7ff3f44cc97c3d235ab6899d6f/globalmount                                                                                                                                                                         
Output: mount error: could not resolve address for prodjenkinsreleases.file.core.windows.net: Unknown error                                                                                                                                        
                                                                                                                                                                                                                                                   
Please refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.                                                                                                                                                   
  Warning  FailedMount  54s                  kubelet  Unable to attach or mount volumes: unmounted volumes=[website-core-packages binary-core-packages], unattached volumes=[website-core-packages workspace-volume kube-api-access-gxr8l binary-co
e-packages[]: timed out waiting for the condition                                                                                                                                                                                                  
  Warning  FailedMount  49s (x5 over 2m57s)  kubelet  MountVolume.MountDevice failed for volume "binary-core-packages" : rpc error: code = Internal desc = volume(binary-core-packages) mount //prodjenkinsreleases.file.core.windows.net/mirrorbit
s on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/f5fe83d0cef11484xxxxxxbf58c3f958fa8bd9ca1a24ad9d1bd9524e1432c/globalmount failed with mount failed: exit status 1                                                            
Mounting command: mount                                                                                                                                                                                                                            
Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,uid=1000,gid=1000,mfsymlinks,nobrl,nosharesock,actimeo=30,<masked> //prodjenkinsreleases.file.core.windows.net/mirrorbits /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com
/f5fe83d0cef11484xxxxxxbf58c3f958fa8bd9ca1a24ad9d1bd9524e1432c/globalmount                                                                                                                                                                      
Output: mount error: could not resolve address for prodjenkinsreleases.file.core.windows.net: Unknown error                                                                                                                                        
                                                                                                                                                                                                                                                   
Please refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.                                                                                                                                                   
  Warning  FailedMount  48s (x2 over 2m56s)  kubelet  MountVolume.MountDevice failed for volume "website-core-packages" : rpc error: code = Internal desc = volume(website-core-packages) mount //prodjenkinsreleases.file.core.windows.net/website
 on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/f5fe83d0cef11484xxxxxxbf58c3f958fa8bd9ca1a24ad9d1bd9524e1432c/globalmount failed with mount failed: exit status 1                                                             
Mounting command: mount                                                                                                                                                                                                                            
Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,uid=1000,gid=1000,mfsymlinks,nobrl,nosharesock,actimeo=30,<masked> //prodjenkinsreleases.file.core.windows.net/website /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/a3
f5fe83d0cef11484xxxxxxbf58c3f958fa8bd9ca1a24ad9d1bd9524e1432c/globalmount                                                                                                                                                                         
Output: mount error: could not resolve address for prodjenkinsreleases.file.core.windows.net: Unknown error                                                                                                                                        
                                                                                                                                                                                                                                                   
Please refer to http://aka.ms/filemounterror for possible causes and solutions for mount errors.

[lemeurherve]

Incident opened: jenkins-infra/status#468

[lemeurherve]

The storage account which seem to have been previously used in the Release Core process has been deleted since last Friday: #3917 (comment)

Trying to recreate the missing website File Share in the get.jenkins.io storage account where the mirrorbits File Share is now, and to set it up to be accessible and used by release.ci.jenkins.io agents.

[lemeurherve]

Issue finally resolved, I'll post a complete report tomorrow.

[alextu]

Thanks! I'll let you close this issue then

[lemeurherve]

The storage account which seem to have been previously used in the Release Core process has been deleted since last Friday: #3917 (comment)

Trying to recreate the missing website File Share in the get.jenkins.io storage account where the mirrorbits File Share is now, and to set it up to be accessible and used by release.ci.jenkins.io agents.

Summary of steps we've took to fix the issue:

• Created the website File Share in the getjenkinsio storage account
  • fix: add website file share to getjenkinsio storage account azure#611
• Manually synchronized it with its expected content
• Added privatek8s_release_tier subnet (used by release.ci.jenkins.io agents) to the allowed vnets of this storage account
  • fix(get-jenkins-io): allow release.ci.jenkins.io to interact with its storage account azure#613
• Added the "Storage Account Contributor" role assignment for this storage account to the privatek8s cluster (where release.ci.jenkins.io and its agents are running).
  • fix(get-jenkins-io): allow release.ci.jenkins.io to interact with its storage account azure#613
• Fixed resource group and apiKey of the correct storage account to use, getjenkinsio instead of prodjenkinsrelease
  • https://github.com/jenkins-infra/charts-secrets/commit/14cab6f166e9d8670ba2d671b1db7e131ba20a58 & https://github.com/jenkins-infra/charts-secrets/commit/60ce64a58c9e4cd489d4e475426a4ce4ecb5c0ef (private repo)
• Set default values for the resource group and file share of the storage account
  • https://github.com/jenkins-infra/charts-secrets/commit/66bd71bb1c0c044ce3255753ab4fc134e8642dc1 (private repo)

With that in place, the Release Core Packaging job resumed as it managed to spawn a pod with the correct access to the mirrorbits and website File Shares.

To check the changes reapplied today as code, I've tested a replay on release.ci.jenkins.io with success:

pipeline {
  agent {
    kubernetes {
      yamlFile 'PodTemplates.d/package-linux.yaml'
    }
  }

  stages {
    stage('Test') {
      steps {
        echo 'OK'
      }
    }
  }
}

Note: this issue underlined that there was a remnant issue with the PV in privatek8s cluster, the env-release chart we're using has prodreleasecore as default value for the storage account's resource group but AFAIK this value was never set in our releases: https://github.com/jenkins-infra/helm-charts/blob/004051b7d1fb52cb8d90d5c2f147984764b9b4a5/charts/env-jenkins-release/values.yaml#L8