4.6 UPI Installation fails to bootstrap when using proxy

[thurcombe]

Describe the bug

4.6 UPI Installation fails to bootstrap when using a corporate proxy.

Nov 25 10:54:35 okd-master-1 machine-config-daemon[7043]: I1125 10:54:35.362110    7043 update.go:948] Applying extensions : ["update" "--install" "glusterfs" "--install" "glusterfs-fuse" "--install" "open-vm-tools>
Nov 25 11:00:36 okd-master-1 machine-config-daemon[7043]: I1125 11:00:36.594617    7043 update.go:361] Rolling back applied changes to OS due to error: failed to execute rpm-ostree ["update" "--install" "glusterfs">
Nov 25 11:00:36 okd-master-1 machine-config-daemon[7043]: I1125 11:00:36.595785    7043 rpm-ostree.go:261] Running captured: rpm-ostree cleanup -p
Nov 25 11:00:40 okd-master-1 machine-config-daemon[7043]: I1125 11:00:40.340536    7043 update.go:1034] Updating files
Nov 25 11:00:40 okd-master-1 machine-config-daemon[7043]: I1125 11:00:40.342149    7043 update.go:1071] Deleting stale data
Nov 25 11:00:40 okd-master-1 machine-config-daemon[7043]: error: failed to execute rpm-ostree ["update" "--install" "glusterfs" "--install" "glusterfs-fuse" "--install" "open-vm-tools" "--install" "NetworkManager-o>
Nov 25 11:00:40 okd-master-1 systemd[1]: machine-config-daemon-firstboot.service: Main process exited, code=exited, status=1/FAILURE
Nov 25 11:00:40 okd-master-1 systemd[1]: machine-config-daemon-firstboot.service: Failed with result 'exit-code'.
Nov 25 11:00:40 okd-master-1 systemd[1]: Failed to start Machine Config Daemon Firstboot.
Nov 25 11:00:40 okd-master-1 systemd[1]: machine-config-daemon-firstboot.service: Consumed 43.161s CPU time.

Version
4.6.0-0.okd-2020-11-24-050344, also tested 4.6.0-0.okd-2020-11-26-085309

How reproducible
100%

Workaround

$ mkdir -p /etc/systemd/system/rpm-ostreed.service.d
$ cat > /etc/systemd/system/rpm-ostreed.service.d/http-proxy.conf << EOF
[Service]
Environment="http_proxy=http://proxy.server:3128"
Environment="https_proxy=http://proxy.server:3128"
EOF

$ systemctl daemon-reload
$ systemctl restart rpm-ostreed
$ systemctl restart machine-config-daemon-firstboot.service

[timbrd]

I confirm it, I had to add the proxy env vars to the service as well.

@thurcombe: Were you able to roll out a cluster successfully? I am trying to deploy an okd cluster (4.5 and 4.6 releases) on vsphere over an http proxy for days, but it stops at 72%. The logs of the bootstrap and master nodes were not helpful yet, I am still trying to figure out why the installation stalls.

[vrutkovs]

You don't need external connection, iiuc openshift/machine-config-operator#2161 fixes it

[thurcombe]

@vrutkovs thanks, confirmed resolved after deploying 4.6.0-0.okd-2020-11-26-141747

@timbrd Yes we are deploying 4.5/4.6 clusters in UPI. Out of interest, what is the status of your oauth pods in the openshift-authentication namespace?

[timbrd]

@timbrd Yes we are deploying 4.5/4.6 clusters in UPI. Out of interest, what is the status of your oauth pods in the openshift-authentication namespace?

My cluster installation breaks at the step where the sdn pods should start, so there is no oauth pod yet.