Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS: cannot add network services: adding watcher failed #3145

Closed
dronenb opened this issue Jan 23, 2025 · 25 comments
Closed

macOS: cannot add network services: adding watcher failed #3145

dronenb opened this issue Jan 23, 2025 · 25 comments

Comments

@dronenb
Copy link

dronenb commented Jan 23, 2025

Description

Hi, I am brand new to using lima, and for some reason I cannot seem to start any VM's. When I try limactl start --log-level=trace, I get the following:

DEBU[0000] ResolveVMType: resolved VMType "vz" (existing instance, with "/Users/dronenb/.lima/default/vz-identifier")
INFO[0000] Using the existing instance "default"
DEBU[0000] ResolveVMType: resolved VMType "vz" (existing instance, with "/Users/dronenb/.lima/default/vz-identifier")
DEBU[0000] Make sure "user-v2" network is stopped
DEBU[0000] Make sure usernet network is stopped
DEBU[0000] Make sure "shared" network is stopped
DEBU[0000] Make sure "bridged" network is stopped
DEBU[0000] Make sure "host" network is stopped
INFO[0000] Starting the instance "default" with VM driver "vz"
INFO[0000] Attempting to download the image              arch=aarch64 digest="sha256:34c1b8c2fe220f67faa351a240d199987db52c574d1a100548386998641434c4" location="https://cloud-images.ubuntu.com/releases/24.10/release-20250110/ubuntu-24.10-server-cloudimg-arm64.img"
DEBU[0000] downloading "https://cloud-images.ubuntu.com/releases/24.10/release-20250110/ubuntu-24.10-server-cloudimg-arm64.img" into "/Users/dronenb/Library/Caches/lima/download/by-url-sha256/bf6c5d347f148f32c31d5e0a0975a41cc540ef4e9f04a5713e6450413db81fd1/data"
Downloading the image (ubuntu-24.10-server-cloudimg-arm64.img)
579.04 MiB / 579.04 MiB [-----------------------------------] 100.00% 9.75 MiB/s
DEBU[0060] res.ValidatedDigest=true
INFO[0060] Downloaded the image from "https://cloud-images.ubuntu.com/releases/24.10/release-20250110/ubuntu-24.10-server-cloudimg-arm64.img"
INFO[0060] Converting "/Users/dronenb/.lima/default/basedisk" (qcow2) to a raw disk "/Users/dronenb/.lima/default/diffdisk"
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.63 GiB/s
INFO[0062] Expanding to 100GiB
INFO[0062] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:2a97f78e14cb3e024068d936c8fb801365981e5b48577e278907079de47e4d2c" location="https://github.com/containerd/nerdctl/releases/download/v2.0.3/nerdctl-full-2.0.3-linux-arm64.tar.gz"
DEBU[0062] downloading "https://github.com/containerd/nerdctl/releases/download/v2.0.3/nerdctl-full-2.0.3-linux-arm64.tar.gz" into "/Users/dronenb/Library/Caches/lima/download/by-url-sha256/c6ebad532b2f0562fcaf9a509f21375c20a44ed3c148c4ad1847bd06107f375f/data"
Downloading the nerdctl archive (nerdctl-full-2.0.3-linux-arm64.tar.gz)
202.48 MiB / 202.48 MiB [----------------------------------] 100.00% 13.12 MiB/s
DEBU[0078] res.ValidatedDigest=true
INFO[0078] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v2.0.3/nerdctl-full-2.0.3-linux-arm64.tar.gz"
DEBU[0077] [hostagent] ResolveVMType: resolved VMType "vz" (existing instance, with "/Users/dronenb/.lima/default/vz-identifier")
INFO[0078] [hostagent] debug mode detected, adding more guest agent candidates: /opt/_output/share/lima/lima-guestagent.Linux-aarch64
DEBU[0078] [hostagent] Creating iso file /Users/dronenb/.lima/default/cidata.iso
DEBU[0078] [hostagent] Using /var/folders/z1/wk4hmrp1471ff58bl2js2sgw0000gn/T/diskfs_iso3598083488 as workspace
DEBU[0078] [hostagent] Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.
DEBU[0078] [hostagent] OpenSSH version 9.8.1 detected
DEBU[0078] [hostagent] AES accelerator seems available, prioritizing [email protected] and [email protected]
INFO[0078] [hostagent] hostagent socket created at /Users/dronenb/.lima/default/ha.sock
DEBU[0078] [hostagent] Start udp DNS listening on: 127.0.0.1:57969
INFO[0078] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/dronenb/.lima/default/serial*.log")
DEBU[0078] [hostagent] Start tcp DNS listening on: 127.0.0.1:49488
DEBU[0078] [hostagent] Using search domains: [attlocal.net]
INFO[0078] [hostagent] reloading dns nameservers to [192.168.1.254:53]
DEBU[0079] received an event                             event="{2025-01-23 17:21:15.655732 -0500 EST {false false true [] 0}}"
FATA[0079] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/dronenb/.lima/default/ha.stderr.log")

The error log shows the following:

$ cat /Users/dronenb/.lima/default/ha.stderr.log
{"level":"debug","msg":"ResolveVMType: resolved VMType \"vz\" (existing instance, with \"/Users/dronenb/.lima/default/vz-identifier\")","time":"2025-01-23T17:21:14-05:00"}
{"level":"info","msg":"debug mode detected, adding more guest agent candidates: /opt/_output/share/lima/lima-guestagent.Linux-aarch64","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Creating iso file /Users/dronenb/.lima/default/cidata.iso","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Using /var/folders/z1/wk4hmrp1471ff58bl2js2sgw0000gn/T/diskfs_iso3598083488 as workspace","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2025-01-23T17:21:15-05:00"}
{"level":"info","msg":"hostagent socket created at /Users/dronenb/.lima/default/ha.sock","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:57969","time":"2025-01-23T17:21:15-05:00"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/dronenb/.lima/default/serial*.log\")","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:49488","time":"2025-01-23T17:21:15-05:00"}
{"level":"debug","msg":"Using search domains: [attlocal.net]","time":"2025-01-23T17:21:15-05:00"}
{"level":"info","msg":"reloading dns nameservers to [192.168.1.254:53]","time":"2025-01-23T17:21:15-05:00"}
{"level":"fatal","msg":"cannot add network services: adding watcher failed: \"/private/var/run/mdnsd_aud\": operation not supported on socket","time":"2025-01-23T17:21:15-05:00"}

That last line seems to be the problem:

{
  "level": "fatal",
  "msg": "cannot add network services: adding watcher failed: \"/private/var/run/mdnsd_aud\": operation not supported on socket",
  "time": "2025-01-23T17:21:15-05:00"
}

I can't seem to find any other issues related to this, nor anything in GitHub discussions. I am not sure how to proceed with troubleshooting. I have tried different templates, deleted the ${HOME}/.lima and ${HOME}/Library/Caches/lima directories to force re-download of the base image, all yield the same results.

Lima Version

$ limactl --version
limactl version 1.0.4

macOS Version

$ sw_vers
ProductName:		macOS
ProductVersion:		15.2
BuildVersion:		24C101
@dronenb
Copy link
Author

dronenb commented Jan 24, 2025

This appears to only happen with vz VM's, as I was able to spin up a qemu backed VM just fine.

@jandubois
Copy link
Member

The file /var/run/mdnsd_aud is not very common. It seems related to the UAD Thunderbolt DSP processors, and seems incompatible with the way Lima manages networks.

Is the file a symlink? If yes, does it point to an existing socket file?

I assume if you delete or rename the file, Lima will work normally.

You could also try to reboot your machine; it will clear out old stuff from /var/run automatically.

I think this issue is related to containers/gvisor-tap-vsock#462

@dronenb
Copy link
Author

dronenb commented Jan 24, 2025

$ ls -al /private/var/run/mdnsd_aud
lrwxr-xr-x  1 root  daemon  22 Jan 24 16:10 /private/var/run/mdnsd_aud -> /var/run/mDNSResponder

I presume you found this thread - looking through that gave me a hint as to where the problem arose.

ggrep -irl mdnsd_aud /Library/Application\ Support/Audinate
/Library/Application Support/Audinate/ConMon.bundle/Contents/MacOS/launch_conmon.sh
cat /Library/Application\ Support/Audinate/ConMon.bundle/Contents/MacOS/launch_conmon.sh

Yields:

#!/bin/sh

# Redirect legacy responder to current

MDNSD_PIPE=/var/run/mdnsd_aud
rm -f "${MDNSD_PIPE}"
ln -s /var/run/mDNSResponder "${MDNSD_PIPE}"

exec "/Library/Application Support/Audinate/ConMon.bundle/Contents/MacOS/conmon_cmm" "$@"

This looks like something Audinate puts in place to make either Dante Controller or Dante Virtual Soundcard work correctly. As you correctly suggested, deleting that file causes lima to work just fine. Presumably there aren't a ton of individuals using both Dante Virtual Soundcard and/or Dante Controller and also trying to leverage lima, but I am sure I am not the only one, so hopefully this thread helps somebody in the future.

All of that said, it seems like having that symlink there shouldn't cause lima to not work, correct?

@jandubois
Copy link
Member

it seems like having that symlink there shouldn't cause lima to not work, correct?

Yes, but it is an issue with gvisor-tap-vsock, not Lima itself, so needs to be addressed there. So you may want to follow up in the issue I linked above.

@dronenb
Copy link
Author

dronenb commented Jan 24, 2025

It is very ironic that I experienced this in addition to the issue w/ gvproxy that affects podman installed via brew...

Thanks for the help! Closing.

@dronenb dronenb closed this as completed Jan 24, 2025
@rfay
Copy link
Contributor

rfay commented Jan 25, 2025

I'm seeing this on DDEV test runners (pre-existing instance) since the v1.0.4 release

{"level":"fatal","msg":"cannot add network services: adding watcher failed: \"/private/var/run/docker.sock\": operation not supported on socket","time":"2025-01-24T17:45:49-07:00"}

/var/run/docker.sock is a symlink (to Orbstack). This never prevented lima from starting before. And it has its own socket...

@jandubois
Copy link
Member

@rfay It is a new issue introduced in gvisor-tap-vsock 0.8.2 via containers/gvisor-tap-vsock#450.

@flphvlck
Copy link

flphvlck commented Jan 25, 2025

Hi, same issue with Lima 1.0.4:

{"level":"fatal","msg":"cannot add network services: adding watcher failed: \"/private/var/run/docker.sock\": operation not supported on socket","time":"2025-01-25T12:04:22+01:00"}

I use Docker Desktop and it creates the socket.
What will be next? The bug is somewhere outside the Lima and we have to wait when they fix it?
I tested 1.0.3 and it wasn't affected.

@rfay
Copy link
Contributor

rfay commented Jan 25, 2025

@jandubois I think this might be a logic error in lima, because it's trying to use a socket that it doesn't own. It's trying to use /var/run/docker.sock, but it's got a different docker context that it should be using, so is it accidentally using /var/run/docker.sock?

I'm trying to add a workaround for DDEV tests, but I don't think lima should fail to run just because an unrelated docker provider is running using an unrelated docker.sock.

@rfay
Copy link
Contributor

rfay commented Jan 25, 2025

Just a note that this breaks Colima in the same way, as one would expect. I don't imagine there are too many folks out there running multiple docker providers, but at least a couple of us have already showed up.

@jandubois
Copy link
Member

I think this might be a logic error in lima, because it's trying to use a socket that it doesn't own. It's trying to use /var/run/docker.sock, but it's got a different docker context that it should be using, so is it accidentally using /var/run/docker.sock?

@rfay The error is in gvisor-tap-vsock, not in Lima; I've linked to the issue that created the problem above. This issue itself is containers/gvisor-tap-vsock#462 and there is a PR in progress to address it.

@Savemech
Copy link

Hey folks! Adding few my bits here!
My error was:

{"level":"info","msg":"reloading dns nameservers to [100.100.100.100:53]","time":"2025-01-28T14:49:15+02:00"}
{"level":"fatal","msg":"cannot add network services: adding watcher failed: \"/private/var/run/docker.sock\": operation not supported on socket","time":"2025-01-28T14:49:15+02:00"}

Fix that worked for me:

brew uninstall colima

ls -alh /private/var/run/docker.sock
   rwxr-xr-x   1   root   daemon     34 B     Mon Dec 16 11:51:28 2024    docker.sock  ⇒ /Users/svx/.docker/run/docker.sock
rm -f /private/var/run/docker.sock
rm: cannot remove '/private/var/run/docker.sock': Permission denied
sudo rm -f /private/var/run/docker.sock
ls -alh /Users/svx/.docker/run/docker.sock
   rwxr-xr-x   1   svx   staff      0 B     Mon Jul 31 12:19:48 2023    docker.sock
rm -f /Users/svx/.docker/run/docker.sock

then
brew install colima
also note that i use docker as frontend

brew list -1 | grep docker
docker
docker-completion
docker-compose
docker
docker context inspect

[
    {
        "Name": "colima",
        "Metadata": {
            "Description": "colima"
        },
        "Endpoints": {
            "docker": {
                "Host": "unix:///Users/svx/.colima/default/docker.sock",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "/Users/svx/.docker/contexts/meta/hash",
            "TLSPath": "/Users/svx/.docker/contexts/tls/hash"
        }
    }
]

@mcomello
Copy link

sudo rm -f /private/var/run/docker.sock

This is the one command that fixed the entire issue for me.
The symlink pointed to a docker.sock that simply didnt exist anymore (not sure why).
Removing the symlink and simply starting colima again, things started working.

@jandubois
Copy link
Member

FYI: all the recent refactoring of DNS code in gvisor-tap-vsock has been reverted in containers/gvisor-tap-vsock#467.

So all the file watching code should be gone in the next release.

@jandubois
Copy link
Member

@rfay gvisor-tap-vsock has just been updated to 0.8.3, which reverts all the DNS related changes since 0.7.5.

It would be great if you could verify that your issues are resolved with the latest Lima master!

@rfay
Copy link
Contributor

rfay commented Feb 4, 2025

I'm happy to check! A casual review of docs didn't turn up any build instructions, and simple make got

rfay@MacBookAir:~/workspace/lima$ go version
go version go1.23.5 darwin/arm64

rfay@MacBookAir:~/workspace/lima$ make
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOVERSION'
clang: error: no input files
CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 CC=ccache clang go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/bin/limactl ./cmd/limactl
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'build'
make: *** [_output/bin/limactl] Error 1

rfay@MacBookAir:~/workspace/lima$ make binaries
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOVERSION'
clang: error: no input files
CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 CC=ccache clang go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/bin/limactl ./cmd/limactl
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'build'
make: *** [_output/bin/limactl] Error 1

I guess it requires CGO_ENABLED=1 and clang refuses to do that on my mac m1 ?

rfay@MacBookAir:~/workspace/lima$ clang --version
Apple clang version 16.0.0 (clang-1600.0.26.6)
Target: arm64-apple-darwin24.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
rfay@MacBookAir:~/workspace/lima$ ccache --version
ccache version 4.10.2
Features: file-storage http-storage redis+unix-storage redis-storage

Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2024 Joel Rosdahl and other contributors

I did get an apparent successful build with

CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb" -o _output/bin/limactl ./cmd/limactl

(that did get ld: warning: ignoring duplicate libraries: '-lobjc')

_output/bin/limactl --version
limactl version 1.0.4-35-g9f1390bb

After a bit more fiddling I got a failure to limactl start that I don't know how to solve:

{"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. The process doesn’t have the “com.apple.security.virtualization” entitlement.\" UserInfo={\n    NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n    NSLocalizedFailureReason = \"The process doesn\\U2019t have the \\U201ccom.apple.security.virtualization\\U201d entitlement.\";\n}","time":"2025-02-03T19:45:22-07:00"}

@rfay
Copy link
Contributor

rfay commented Feb 4, 2025

Oh, forgot about brew install --HEAD:

rfay@rfay-tag1-m1:~$ limactl --version
limactl version 1.0.4-35-g9f1390bb

I confirm that 1.0.4-35-g9f1390bb works and is able to limactl start and confirm that 1.0.4 fails with {"level":"fatal","msg":"cannot add network services: adding watcher failed: \"/private/var/run/docker.sock\": operation not supported on socket","time":"2025-02-03T20:40:29-07:00"}

So all is good, thanks! (Build instructions would be a great addition to the README. Maybe they're there and I just didn't find them...)

@jandubois
Copy link
Member

So all is good, thanks!

Thank you for confirming.

(Build instructions would be a great addition to the README. Maybe they're there and I just didn't find them...)

They are slightly hidden: There is a link to the docs for "more information", and the Installation page contains a tab to show how you build from source.

However, it is pretty straight forward: make && make install.

@jandubois
Copy link
Member

it is pretty straight forward: make && make install

Oops, I see that you tried that, and it didn't work. Which is weird, because I've been running it thousands of times (well, I normally run make native to build a subset, but still). No idea what is different in your setup.

@jandubois
Copy link
Member

@rfay If you want to debug this, then I would run make clean && make --dry-run and then run the commands one at a time to see what fails.

Here is the list of commands I got for `make --dry-run`:
❯ make --dry-run
rm -rf _output/bin/limactl.exe
CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 CC=cc go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/bin/limactl ./cmd/limactl
codesign -f -v --entitlements vz.entitlements -s - _output/bin/limactl
mkdir -p _output/bin
cp -a cmd/lima _output/bin/lima
cp -a cmd/nerdctl.lima _output/bin/nerdctl.lima
cp -a cmd/apptainer.lima _output/bin/apptainer.lima
cp -a cmd/docker.lima _output/bin/docker.lima
cp -a cmd/podman.lima _output/bin/podman.lima
cp -a cmd/kubectl.lima _output/bin/kubectl.lima
mkdir -p _output/share/lima
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-aarch64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-aarch64
CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-armv7l ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-armv7l
CGO_ENABLED=0 GOOS=linux GOARCH=riscv64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-riscv64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-riscv64
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-x86_64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-x86_64
mkdir -p _output/share/lima/templates
cp -aL templates/README.md _output/share/lima/templates/README.md
cp -aL templates/almalinux-8.yaml _output/share/lima/templates/almalinux-8.yaml
cp -aL templates/almalinux-9.yaml _output/share/lima/templates/almalinux-9.yaml
cp -aL templates/almalinux.yaml _output/share/lima/templates/almalinux.yaml
cp -aL templates/alpine-iso.yaml _output/share/lima/templates/alpine-iso.yaml
cp -aL templates/alpine.yaml _output/share/lima/templates/alpine.yaml
cp -aL templates/apptainer-rootful.yaml _output/share/lima/templates/apptainer-rootful.yaml
cp -aL templates/apptainer.yaml _output/share/lima/templates/apptainer.yaml
cp -aL templates/archlinux.yaml _output/share/lima/templates/archlinux.yaml
cp -aL templates/buildkit.yaml _output/share/lima/templates/buildkit.yaml
cp -aL templates/centos-stream-10.yaml _output/share/lima/templates/centos-stream-10.yaml
cp -aL templates/centos-stream-9.yaml _output/share/lima/templates/centos-stream-9.yaml
cp -aL templates/centos-stream.yaml _output/share/lima/templates/centos-stream.yaml
cp -aL templates/debian-11.yaml _output/share/lima/templates/debian-11.yaml
cp -aL templates/debian-12.yaml _output/share/lima/templates/debian-12.yaml
cp -aL templates/debian.yaml _output/share/lima/templates/debian.yaml
cp -aL templates/default.yaml _output/share/lima/templates/default.yaml
cp -aL templates/docker-rootful.yaml _output/share/lima/templates/docker-rootful.yaml
cp -aL templates/docker.yaml _output/share/lima/templates/docker.yaml
cp -aL templates/faasd.yaml _output/share/lima/templates/faasd.yaml
cp -aL templates/fedora.yaml _output/share/lima/templates/fedora.yaml
cp -aL templates/k3s.yaml _output/share/lima/templates/k3s.yaml
cp -aL templates/k8s.yaml _output/share/lima/templates/k8s.yaml
cp -aL templates/opensuse-leap.yaml _output/share/lima/templates/opensuse-leap.yaml
cp -aL templates/opensuse.yaml _output/share/lima/templates/opensuse.yaml
cp -aL templates/oraclelinux-8.yaml _output/share/lima/templates/oraclelinux-8.yaml
cp -aL templates/oraclelinux-9.yaml _output/share/lima/templates/oraclelinux-9.yaml
cp -aL templates/oraclelinux.yaml _output/share/lima/templates/oraclelinux.yaml
cp -aL templates/podman-rootful.yaml _output/share/lima/templates/podman-rootful.yaml
cp -aL templates/podman.yaml _output/share/lima/templates/podman.yaml
cp -aL templates/rocky-8.yaml _output/share/lima/templates/rocky-8.yaml
cp -aL templates/rocky-9.yaml _output/share/lima/templates/rocky-9.yaml
cp -aL templates/rocky.yaml _output/share/lima/templates/rocky.yaml
cp -aL templates/ubuntu-20.04.yaml _output/share/lima/templates/ubuntu-20.04.yaml
cp -aL templates/ubuntu-22.04.yaml _output/share/lima/templates/ubuntu-22.04.yaml
cp -aL templates/ubuntu-24.04.yaml _output/share/lima/templates/ubuntu-24.04.yaml
cp -aL templates/ubuntu-24.10.yaml _output/share/lima/templates/ubuntu-24.10.yaml
cp -aL templates/ubuntu-lts.yaml _output/share/lima/templates/ubuntu-lts.yaml
cp -aL templates/ubuntu.yaml _output/share/lima/templates/ubuntu.yaml
mkdir -p _output/share/lima/templates/experimental
cp -aL templates/experimental/alsa.yaml _output/share/lima/templates/experimental/alsa.yaml
cp -aL templates/experimental/gentoo.yaml _output/share/lima/templates/experimental/gentoo.yaml
cp -aL templates/experimental/opensuse-tumbleweed.yaml _output/share/lima/templates/experimental/opensuse-tumbleweed.yaml
cp -aL templates/experimental/rke2.yaml _output/share/lima/templates/experimental/rke2.yaml
cp -aL templates/experimental/u7s.yaml _output/share/lima/templates/experimental/u7s.yaml
cp -aL templates/experimental/vnc.yaml _output/share/lima/templates/experimental/vnc.yaml
cp -aL templates/experimental/wsl2.yaml _output/share/lima/templates/experimental/wsl2.yaml
mkdir -p _output/share/doc/lima
cp -aL MAINTAINERS.md _output/share/doc/lima/MAINTAINERS.md
cp -aL README.md _output/share/doc/lima/README.md
cp -aL ROADMAP.md _output/share/doc/lima/ROADMAP.md
cp -aL LICENSE _output/share/doc/lima/LICENSE
echo "Moved to https://github.com/lima-vm/.github/blob/main/SECURITY.md" > _output/share/doc/lima/SECURITY.md
echo v1.0.4-35-g9f1390bb > _output/share/doc/lima/VERSION
rm -rf _output/share/doc/lima/templates
ln -sf ../../lima/templates _output/share/doc/lima/templates
mkdir -p _output/share/man/man1
_output/bin/limactl generate-doc _output/share/man/man1 \
		--output _output --prefix /usr/local
</details>
</summary>

@rfay
Copy link
Contributor

rfay commented Feb 4, 2025

Here's the result:

`make clean and make --dry-run`
rfay@MacBookAir:~/workspace/lima$ make clean
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOVERSION'
clang: error: no input files
rm -rf _output vendor
rfay@MacBookAir:~/workspace/lima$ make --dry-run
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOVERSION'
clang: error: no input files
rm -rf _output/bin/limactl.exe
CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 CC=ccache clang go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/bin/limactl ./cmd/limactl
codesign -f -v --entitlements vz.entitlements -s - _output/bin/limactl
mkdir -p _output/bin
cp -a cmd/lima _output/bin/lima
cp -a cmd/nerdctl.lima _output/bin/nerdctl.lima
cp -a cmd/apptainer.lima _output/bin/apptainer.lima
cp -a cmd/docker.lima _output/bin/docker.lima
cp -a cmd/podman.lima _output/bin/podman.lima
cp -a cmd/kubectl.lima _output/bin/kubectl.lima
mkdir -p _output/share/lima
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-aarch64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-aarch64
CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-armv7l ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-armv7l
CGO_ENABLED=0 GOOS=linux GOARCH=riscv64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-riscv64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-riscv64
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/share/lima/lima-guestagent.Linux-x86_64 ./cmd/lima-guestagent
chmod 644 _output/share/lima/lima-guestagent.Linux-x86_64
mkdir -p _output/share/lima/templates
cp -aL templates/README.md _output/share/lima/templates/README.md
cp -aL templates/almalinux-8.yaml _output/share/lima/templates/almalinux-8.yaml
cp -aL templates/almalinux-9.yaml _output/share/lima/templates/almalinux-9.yaml
cp -aL templates/almalinux.yaml _output/share/lima/templates/almalinux.yaml
cp -aL templates/alpine-iso.yaml _output/share/lima/templates/alpine-iso.yaml
cp -aL templates/alpine.yaml _output/share/lima/templates/alpine.yaml
cp -aL templates/apptainer-rootful.yaml _output/share/lima/templates/apptainer-rootful.yaml
cp -aL templates/apptainer.yaml _output/share/lima/templates/apptainer.yaml
cp -aL templates/archlinux.yaml _output/share/lima/templates/archlinux.yaml
cp -aL templates/buildkit.yaml _output/share/lima/templates/buildkit.yaml
cp -aL templates/centos-stream-10.yaml _output/share/lima/templates/centos-stream-10.yaml
cp -aL templates/centos-stream-9.yaml _output/share/lima/templates/centos-stream-9.yaml
cp -aL templates/centos-stream.yaml _output/share/lima/templates/centos-stream.yaml
cp -aL templates/debian-11.yaml _output/share/lima/templates/debian-11.yaml
cp -aL templates/debian-12.yaml _output/share/lima/templates/debian-12.yaml
cp -aL templates/debian.yaml _output/share/lima/templates/debian.yaml
cp -aL templates/default.yaml _output/share/lima/templates/default.yaml
cp -aL templates/docker-rootful.yaml _output/share/lima/templates/docker-rootful.yaml
cp -aL templates/docker.yaml _output/share/lima/templates/docker.yaml
cp -aL templates/faasd.yaml _output/share/lima/templates/faasd.yaml
cp -aL templates/fedora.yaml _output/share/lima/templates/fedora.yaml
cp -aL templates/k3s.yaml _output/share/lima/templates/k3s.yaml
cp -aL templates/k8s.yaml _output/share/lima/templates/k8s.yaml
cp -aL templates/opensuse-leap.yaml _output/share/lima/templates/opensuse-leap.yaml
cp -aL templates/opensuse.yaml _output/share/lima/templates/opensuse.yaml
cp -aL templates/oraclelinux-8.yaml _output/share/lima/templates/oraclelinux-8.yaml
cp -aL templates/oraclelinux-9.yaml _output/share/lima/templates/oraclelinux-9.yaml
cp -aL templates/oraclelinux.yaml _output/share/lima/templates/oraclelinux.yaml
cp -aL templates/podman-rootful.yaml _output/share/lima/templates/podman-rootful.yaml
cp -aL templates/podman.yaml _output/share/lima/templates/podman.yaml
cp -aL templates/rocky-8.yaml _output/share/lima/templates/rocky-8.yaml
cp -aL templates/rocky-9.yaml _output/share/lima/templates/rocky-9.yaml
cp -aL templates/rocky.yaml _output/share/lima/templates/rocky.yaml
cp -aL templates/ubuntu-20.04.yaml _output/share/lima/templates/ubuntu-20.04.yaml
cp -aL templates/ubuntu-22.04.yaml _output/share/lima/templates/ubuntu-22.04.yaml
cp -aL templates/ubuntu-24.04.yaml _output/share/lima/templates/ubuntu-24.04.yaml
cp -aL templates/ubuntu-24.10.yaml _output/share/lima/templates/ubuntu-24.10.yaml
cp -aL templates/ubuntu-lts.yaml _output/share/lima/templates/ubuntu-lts.yaml
cp -aL templates/ubuntu.yaml _output/share/lima/templates/ubuntu.yaml
mkdir -p _output/share/lima/templates/experimental
cp -aL templates/experimental/alsa.yaml _output/share/lima/templates/experimental/alsa.yaml
cp -aL templates/experimental/gentoo.yaml _output/share/lima/templates/experimental/gentoo.yaml
cp -aL templates/experimental/opensuse-tumbleweed.yaml _output/share/lima/templates/experimental/opensuse-tumbleweed.yaml
cp -aL templates/experimental/rke2.yaml _output/share/lima/templates/experimental/rke2.yaml
cp -aL templates/experimental/u7s.yaml _output/share/lima/templates/experimental/u7s.yaml
cp -aL templates/experimental/vnc.yaml _output/share/lima/templates/experimental/vnc.yaml
cp -aL templates/experimental/wsl2.yaml _output/share/lima/templates/experimental/wsl2.yaml
mkdir -p _output/share/doc/lima
cp -aL MAINTAINERS.md _output/share/doc/lima/MAINTAINERS.md
cp -aL README.md _output/share/doc/lima/README.md
cp -aL ROADMAP.md _output/share/doc/lima/ROADMAP.md
cp -aL LICENSE _output/share/doc/lima/LICENSE
echo "Moved to https://github.com/lima-vm/.github/blob/main/SECURITY.md" > _output/share/doc/lima/SECURITY.md
echo v1.0.4-35-g9f1390bb > _output/share/doc/lima/VERSION
rm -rf _output/share/doc/lima/templates
ln -sf ../../lima/templates _output/share/doc/lima/templates
mkdir -p _output/share/man/man1
_output/bin/limactl generate-doc _output/share/man/man1 \
		--output _output --prefix /usr/local

And...

rfay@MacBookAir:~/workspace/lima$ go version
go version go1.23.5 darwin/arm64
rfay@MacBookAir:~/workspace/lima$ clang --version
Apple clang version 16.0.0 (clang-1600.0.26.6)
Target: arm64-apple-darwin24.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
rfay@MacBookAir:~/workspace/lima$ ccache --version
ccache version 4.10.2
Features: file-storage http-storage redis+unix-storage redis-storage

Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2024 Joel Rosdahl and other contributors

See <https://ccache.dev/credits.html> for a complete list of contributors.

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
rfay@MacBookAir:~/workspace/lima$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

@jandubois
Copy link
Member

@rfay There must be something wrong with your make command:

rfay@MacBookAir:~/workspace/lima$ make clean
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOVERSION'
clang: error: no input files
rm -rf _output vendor

The clean target is extremely simple and does not invoke clang:

.PHONY: clean
clean:
        rm -rf _output vendor

@jandubois
Copy link
Member

jandubois commented Feb 4, 2025

The Makefile doesn't mention clang, so why is it included in the generated commands:

CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 CC=ccache clang go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v1.0.4-35-g9f1390bb"  -o _output/bin/limactl ./cmd/limactl
                                                 ^^^^^

The only thing I can come up with is that you have a GO environment variable. I tried to reproduce the error, and got a similar but still different result:

GO="clang go" make clean
clang: error: no such file or directory: 'go'
clang: error: no such file or directory: 'env'
clang: error: no such file or directory: 'GOHOSTARCH'
clang: error: no input files
...

Anyways, I'll stop now; I'm sure the root cause is in your build setup and not the Lima repo.

@rfay
Copy link
Contributor

rfay commented Feb 4, 2025

Several set -x added in the stuff in the Makefile showed it to be the use of $(CC)

export CC='ccache clang'
export CXX='ccache clang++'
export CCACHE_CPP2=yes

Since CC was two things, the output of the awk stuff was

+ echo _output/bin/limactl:
+ CGO_ENABLED=1
+ GOOS=darwin
+ GOARCH=arm64
+ CC=ccache
+ clang go env

I don't know where those came from.

There may be something wrong with the use of $(CC) in the awk stuff in the Makefile but I don't think it's worth studying.

@jandubois
Copy link
Member

There may be something wrong with the use of $(CC) in the awk stuff in the Makefile but I don't think it's worth studying.

I agree that it is probably not worth it, but maybe you could still test if simply adding quotes would fix it?

- ENVS__output/bin/limactl$(exe) = CGO_ENABLED=1 GOOS=$(GOOS) GOARCH=$(GOARCH) CC=$(CC)
+ ENVS__output/bin/limactl$(exe) = CGO_ENABLED=1 GOOS=$(GOOS) GOARCH=$(GOARCH) CC="$(CC)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants