"libguestfs requires the kernel executable to be readable" - Image no longer scans correctly

[cco3]

I had a qcow2 image that scanned on previous versions (not sure if it was before or after 30.0.0), but on 30.1.0, I get the following error using the same image:

libguestfs requires the kernel executable to be readable. This is the case by default on most Linux distributions except on Ubuntu. Please follow the ExtractCode installation instructions in the README.rst at: https://github.com/nexB/extractcode/blob/main/README.rst '

Any idea what might be going on here or what information I can provide without the image itself?

[tdruez]

It seems that the issue was introduced when we added the --no-install-recommends to reduce the size of the docker images in c55168a#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R33

One or more of the “recommended” package is actually required for proper VM extraction.

For now, I've remove the --no-install-recommends and released a 30.1.1 version at https://github.com/nexB/scancode.io/releases/tag/v30.1.1

@cco3 let me know if the qcow2 extraction works again on your side with that release.

[pombredanne]

Note that the error message is there because the kernel of the host machine needs to be made readable (which should be a problem ONLY on Ubuntu). @cco3 are you running SCIO on Ubuntu?

See https://github.com/nexB/extractcode/blob/main/README.rst#adding-support-for-vm-images-extraction for details

@tdruez if the error was triggered for another reason, this is a bug to also fix in extractcode

[cco3]

I'm on Debian, will try it out.

[tdruez]

@pombredanne There's a bug in extractcode that returns the "read-only" issue while the real issue is the following:

Failed to run guestfish to extract VM image: /usr/bin/guestfish --ro --format=qcow2 --add /var/scancodeio/workspace/projects/***/input/***qcow2 r
un : list-filesystems output: *stdin*:0: libguestfs: error: /usr/bin/supermin exited with error status 1. 
To see full error messages you may need to enable debugging. 
Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. 
For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs 
You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list.

[tdruez]

With the debug mode:

ENV LIBGUESTFS_DEBUG 1
ENV LIBGUESTFS_TRACE 1

Failed to run guestfish to extract VM image: /usr/bin/guestfish --ro --format=qcow2 --add /var/scancodeio/workspace/projects/***/input/***.qcow2 run : list-filesystems
output: libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x56427d98e680, program = guestfish
libguestfs: trace: add_drive "/var/scancodeio/workspace/projects/***/input/***.qcow2" "readonly:true" "format:qcow2"
libguestfs: creating COW overlay to protect original drive content
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: disk_create "/tmp/libguestfsQrskot/overlay1.qcow2" "qcow2" -1 "backingfile:/var/scancodeio/workspace/projects/***/input/d***.qcow2" "backingformat:qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/scancodeio/workspace/projects/***/input/***qcow2,backing_fmt=qcow2
libguestfs: command: run: \ /tmp/libguestfsQrskot/overlay1.qcow2
Formatting '/tmp/libguestfsQrskot/overlay1.qcow2', fmt=qcow2 size=2147483648 backing_file=/var/scancodeio/workspace/projects/***/input/***.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
libguestfs: trace: disk_create = 0
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 40, release: 2, extra: , >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=guestfish
libguestfs: launch: version=1.40.2
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsQrskot
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib/x86_64-linux-gnu/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.20
supermin: package handler: debian/dpkg
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib/x86_64-linux-gnu/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 199 packages, including dependencies
supermin: build: 11145 files
supermin: build: 6704 files, after matching excludefiles
supermin: build: 6707 files, after adding hostfiles
supermin: build: 6707 files, after removing unreadable files
supermin: build: 6710 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: looking for kernels in /boot ...
supermin: failed to find a suitable kernel (host_cpu=x86_64).

I looked for kernels in /boot and modules in /lib/modules.

If this is a Xen guest, and you only have Xen domU kernels
installed, try installing a fullvirt kernel (only for
supermin use, you shouldn't boot the Xen guest with it).
*stdin*:0: libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above
libguestfs: trace: launch = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x56427d98e680 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsQrskot
>

[tdruez]

Kernels are missing:

supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: looking for kernels in /boot ...
supermin: failed to find a suitable kernel (host_cpu=x86_64).

[tdruez]

Add apt-get install linux-image-amd64 makes the kernels available and fix the vm images extraction:

$ ls -la /boot/
System.map-4.19.0-18-amd64
config-4.19.0-18-amd64
initrd.img-4.19.0-18-amd64
vmlinuz-4.19.0-18-amd64

The linux-image-amd64 adds about 300MB to the docker image size, which is much better than installing all the recommended packages which adds 700MB.

[pombredanne]

@tdruez excellent! I wish it would have been smaller once installed... but at least this is minimal in terms of number of packages.

[tdruez]

• Add a new GitHub action that build the docker-compose images and run the test suite. This ensure that the app is properly working and tested when running with Docker.

• Add --no-install-recommends in the Dockerfile apt-get install and add the linux-image-amd64 package. This packages makes available the kernels required by extractcode and libguestfs for proper VM images extraction.