Port forwarding fails using VM type VZ and aarch64

[jasperf]

Description

Using Ubuntu with VM type VZ and the following setup on macOS Sequoia 15.01 and

limactl -v
limactl version 1.0.1

limactl create --arch=aarch64 --vm-type=vz --mount-type=virtiofs template://ubuntu
? Creating an instance "ubuntu" Open an editor to review or modify the current configuration
INFO[0071] Attempting to download the image              arch=aarch64 digest="sha256:d71df0bcca6c3d2e7530517d3885f1d007fd9210d40ce2054db36af2a2176c38" location="https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-arm64.img"
Downloading the image (ubuntu-24.10-server-cloudimg-arm64.img)
592.71 MiB / 592.71 MiB [-----------------------------------] 100.00% 7.39 MiB/s
INFO[0152] Downloaded the image from "https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-arm64.img"
INFO[0152] Converting "/Users/user/.lima/ubuntu/basedisk" (qcow2) to a raw disk "/Users/user/.lima/ubuntu/diffdisk"
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.28 GiB/s
INFO[0155] Expanding to 100GiB
INFO[0155] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:fe085381a09aa240ae5d1e0bbef1beccfb7c1d6dbb98bdc55bd416581d46ebc8" location="https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
INFO[0155] Using cache "/Users/user/Library/Caches/lima/download/by-url-sha256/1699e54a52757df863155fca76f8a77b50f05d993edca23421798af6635156f0/data"
INFO[0155] Run `limactl start ubuntu` to start the instance.

with addition

ssh:
  localPort: 2022
portForwards:
  - guestPort: 80
    hostPort: 8080
  - guestPort: 443
    hostPort: 8443
  - guestPort: 6379
    hostPort: 6380

leading to

# Review and modify the following configuration for Lima instance "ubuntu".
# - To cancel starting Lima, just save this file as an empty file.

minimumLimaVersion: "1.0.0"
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:ee070d95a2ba5a1500264e75b3e14aa85518220c24d25f1535407c55f0e33e4d"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:d71df0bcca6c3d2e7530517d3885f1d007fd9210d40ce2054db36af2a2176c38"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-riscv64.img"
  arch: "riscv64"
  digest: "sha256:9cfcec0f9635b34e3c6d5294b33444b56c13a7ac5ac01c39e911c754b8018395"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release-20241023/ubuntu-24.10-server-cloudimg-armhf.img"
  arch: "armv7l"
  digest: "sha256:d51b578420a76e4d9eb1ed333acf2c1f1c61564e5325a4e888d693b721fbc7bd"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/24.10/release/ubuntu-24.10-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release/ubuntu-24.10-server-cloudimg-arm64.img"
  arch: "aarch64"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release/ubuntu-24.10-server-cloudimg-riscv64.img"
  arch: "riscv64"
- location: "https://cloud-images.ubuntu.com/releases/24.10/release/ubuntu-24.10-server-cloudimg-armhf.img"
  arch: "armv7l"
mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true

# 9p is broken in Linux v6.9, v6.10, and v6.11 (used by Ubuntu 24.10).
# The issue was fixed in Linux v6.12-rc5 (https://github.com/torvalds/linux/commit/be2ca38).
mountTypesUnsupported: ["9p"]
mountType: virtiofs
arch: aarch64
vmType: vz
ssh:
  localPort: 2022
portForwards:
  - guestPort: 80
    hostPort: 8080
  - guestPort: 443
    hostPort: 8443
  - guestPort: 6379
    hostPort: 6380
~
~
~
~
~
~
~

has all ports / port forwarding added ignored:

limactl start ubuntu
INFO[0000] Using the existing instance "ubuntu"
INFO[0000] Starting the instance "ubuntu" with VM driver "vz"
INFO[0000] [hostagent] hostagent socket created at /Users/user/.lima/ubuntu/ha.sock
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/user/.lima/ubuntu/serial*.log")
INFO[0001] SSH Local Port: 2022
INFO[0000] [hostagent] [VZ] - vm state change: running
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0010] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0011] [hostagent] The essential requirement 1 of 2 is satisfied
INFO[0011] [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh"
INFO[0011] [hostagent] The essential requirement 2 of 2 is satisfied
INFO[0011] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available"
INFO[0011] [hostagent] Guest agent is running
INFO[0011] [hostagent] The optional requirement 1 of 2 is satisfied
INFO[0011] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed"
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.54:53
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0011] [hostagent] Not forwarding TCP 0.0.0.0:22
INFO[0011] [hostagent] Not forwarding TCP [::]:22
INFO[0032] [hostagent] The optional requirement 2 of 2 is satisfied
INFO[0032] [hostagent] Waiting for the guest agent to be running
INFO[0032] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0035] [hostagent] Forwarding TCP from 127.0.0.1:37179 to 127.0.0.1:37179
INFO[0044] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0045] READY. Run `limactl shell ubuntu` to open the shell.

Inside the virtual machine I checked open ports and I see

user@lima-ubuntu:/Users/user/code/stedding$ sudo ss -tuln
Netid          State           Recv-Q          Send-Q                       Local Address:Port                    Peer Address:Port
udp            UNCONN          0               0                               127.0.0.54:53                           0.0.0.0:*
udp            UNCONN          0               0                            127.0.0.53%lo:53                           0.0.0.0:*
udp            UNCONN          0               0                        192.168.5.15%eth0:68                           0.0.0.0:*
tcp            LISTEN          0               4096                            127.0.0.54:53                           0.0.0.0:*
tcp            LISTEN          0               4096                             127.0.0.1:37179                        0.0.0.0:*
tcp            LISTEN          0               4096                         127.0.0.53%lo:53                           0.0.0.0:*
tcp            LISTEN          0               4096                               0.0.0.0:22                           0.0.0.0:*
tcp            LISTEN          0               4096                                  [::]:22                              [::]:*

So I do not see ports 22, 80, 443, 6379 open on the gues. It seems the added rules for port forwarding are being ignored. Shell access fails:

ssh user@lima-ubuntu -p 2022
The authenticity of host '[localhost]:2022 ([127.0.0.1]:2022)' can't be established.
ED25519 key fingerprint is SHA256:R8uyxqXkpwkScy25mKR6JGHcKDjpnYrUcov30Lh7mg4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2022' (ED25519) to the list of known hosts.
user@localhost: Permission denied (publickey).
ssh [email protected] -p 2022
The authenticity of host '[127.0.0.1]:2022 ([127.0.0.1]:2022)' can't be established.
ED25519 key fingerprint is SHA256:R8uyxqXkpwkScy25mKR6JGHcKDjpnYrUcov30Lh7mg4.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:63: [localhost]:2022
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:2022' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey).

But public key was added

user@lima-ubuntu:/Users/user/code/stedding$ cat ~/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICx1uTiqms5PWWz1CBbxeicda4iAi7yCy5npS5hHGt1U [email protected]

port checks fail

nc -zv 127.0.0.1 8443
nc: connectx to 127.0.0.1 port 8443 (tcp) failed: Connection refused
nc -zv 127.0.0.1 8080
nc: connectx to 127.0.0.1 port 8080 (tcp) failed: Connection refused

Also tried this with Ubuntu 24.04 but same issue using same virtualization.

abiosoft/colima#1181 might be related

[jasperf]

Shell works after I update the public key on the Lima Virtual Machine. Here redacted command:

echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ--------USqOEz8h1wXVeiIZLa3S0vnxbc0sto [email protected]" > ~/.ssh/authorized_keys

But I do not understand why Lima did not add the correct ED25519 key like it did the last time. Also, other added ports are stil being blocked

nc -zv 127.0.0.1 8080
nc: connectx to 127.0.0.1 port 8080 (tcp) failed: Connection refused
redis-cli -h 127.0.0.1 -p 6380
Could not connect to Redis at 127.0.0.1:6380: Connection refused
not connected>
nc -zv 127.0.0.1 8443
nc: connectx to 127.0.0.1 port 8443 (tcp) failed: Connection refused

[jasperf]

Seems

ssh:
  localPort: 2022
portForwards:
  - guestPort: 80
    hostPort: 8080
  - guestPort: 443
    hostPort: 8443
  - guestPort: 6379
    hostPort: 6380

starts working as soon as setup of Lima VM works with local ports 80, 443 and so on. Why wrong public key was added is not clear yet.

[jasperf]

closing for now