Create cluster using RKE2 and OpenStack node driver

[manuelstein]

Environmental Info:
RKE2 Version: v2.9.2
OpenStack: Queens

Describe the bug:

I am trying to create an RKE2/K3s cluster on OpenStack (Cluster: Create OpenStack) and filling in the OpenStack machine template parameters, I get a pool of nodes with InternalIPs and, by setting the floatingIPPool to the network name that provides floating IPs, the VMs also get floating IPs. However, those floating IPs do not appear as external IPs in the OpenstackMachines.rke-machine.cattle.io/v1 status section. And as a result, they don't show up as external IPs in the dashboard and the cluster nodes doesn't have external IPs configured either.

Steps To Reproduce:

• Installed RKE2: v2.9.2
• OpenStack: Queens

Use the UI to create a cluster, select RKE2, select OpenStack, enter OpenStack API information to create a node pool.

Expected behavior:
Nodes show up in the UI with both their Internal IP and their External IP (aka floating IP on OpenStack).

Actual behavior:
Nodes only show up with a correct Internal IP and no External IP.

Additional context / logs:
I don't know where to look to debug this further. The OpenstackMachines are in the namespace rke-machine.cattle.io/v1 but I couldn't locate that anywhere on GitHub. Is this using https://github.com/kubernetes-sigs/cluster-api-provider-openstack under the hood?

[brandond]

I'm a little confused - when you say "Use the UI", what UI are you referring to?. Are you using rancher? If you're using Rancher then this is the wrong repo for this question.

[manuelstein]

Ok, so I'm using the Rancher dashboard, yes. But there is no error in the dashboard, the error is with setting up RKE2/K3S on OpenStack. AFAIU the OpenStack node driver provides a node pool and the VMs are correctly setup (with a floating IP). However, RKE2 does not use these floating IPs as external IPs in the K8s node setup. Now, it may be that this is an issue of the node driver or it may be that this is an issue of RKE2 and as explained, I don't know where to look for the issue, because I wasn't able to find any documentation or code for the node pool resources that Rancher created on my management cluster with type openstackmachines.rke-machine.cattle.io/v1. If you happen to know about how this works under the hood, I'd appreciate any hint that helps me to find out where this all goes wrong.

[brandond]

Did you want to to disable the RKE2 stub cloud provider and deploy the OpenStack cloud provider to the cluster instead? RKE2 has no awareness of anything openstack related including the floating IPs. You could try explicitly configuring the node-external-ip CLI flag or config file entry on each node to the correct value but that's probably not what you want to do.

[manuelstein]

Thanks, that's getting me closer to where that external IP might come from.

I do prefer an easy workflow, yes, wouldn't mind GitOps btw. In our Rancher 2.9.2, I navigated to Create a cluster, switched to RKE2/K3s, selected OpenStack and filled in all the fields of the node pool (authUrl, username, userDomain, tenantDomain, flavor, imageName, etc.). The cluster was created and nodes started, the agents reported back to RKE. The provisioning job for each node running the image rancher/machine:v0.15.0-rancher116 completed successfully and the cluster is there but has problems.

I got rid of one problem source though: Provider RKE2 and Kubernetes version v1.30.7 +rke2r1 do not like to run on my Ubuntu 22.04 image due to kube-proxy not being able to use --xor-mark with iptables - kube-proxy keeps failing with:

E0115 13:46:23.397689       1 proxier.go:1511] "Failed to execute iptables-restore" err=<
	exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
	ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"
	Error occurred at line: 17
	Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
 >

Now that I've switched to Ubuntu 24.04, I still have no external IP addresses, but your question pointing to CCM is a great hint. I might have expected the provisioning job to launch the control plane installation (user-data) with the correct values for internal and external IP. After all, it does attach the floating IP to the OpenStack VM. It could provide the values for both the --node-ip and --node-external-ip flags.

I couldn't find any option in the Cluster resource (provisioning.cattle.io/v1) that would allow me to trigger the --disable-cloud-controller flag. Is it somehow possible to provide an RKE server configuration to set disable-cloud-controller? Would the whole cluster registration still work if I installed my own external OpenStack CCM?

[brandond]

Kubernetes version v1.30.7 +rke2r1 do not like to run on my Ubuntu 22.04 image due to kube-proxy not being able to use --xor-mark with iptables - kube-proxy keeps failing

This is a bug in your kernel. Upgrade or downgrade the kernel version. Ref: #7438 (comment)