Falco with modern bpf does not appear to work on 6.6 LTS kernels >6.6.56

[marwinski]

Describe the bug

We noticed that Falco does not start on our Garden Linux release 1592.3 (kernel 6.6.62) (https://gardenlinux.io) while it worked well with 1592.2 (kernel 6.6.56). We tried the kernels in between and noticed that Falco breaks with 6.6.57. We have two different error messages from two different Falco versions:

Falco 0.37.1 (from a commercial vendor)

Wed Nov 27 10:15:51 2024: [libs]: libpman: failed to load BPF object (errno: 22 | message: Invalid argument)

Falco 0.39.1 (community image falco-distroless)

Fri Nov 29 16:01:19 2024:    Fileless execution via memfd_create
Fri Nov 29 16:01:19 2024: (25) enabled rules in total
Fri Nov 29 16:01:19 2024: Loaded event sources: syscall
Fri Nov 29 16:01:19 2024: Enabled event sources: syscall
Fri Nov 29 16:01:19 2024: Opening event source 'syscall'
Fri Nov 29 16:01:19 2024: Opening 'syscall' source with modern BPF probe.
Fri Nov 29 16:01:19 2024: One ring buffer every '2' CPUs.
Fri Nov 29 16:01:20 2024: An error occurred in an event source, forcing termination...
Fri Nov 29 16:01:20 2024: Stopping capture for event source 'syscall'
Fri Nov 29 16:01:20 2024: closing inspectors
Error: Initialization issues during scap_init
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:

There are not kernel messages and nothing in the journal.

I would like to be careful here, but it appears that the following kernel change in 6.6.57 might be relevant here:

commit 5d5e3b4cbe8ee16b7bf96fd73a421c92a9da3ca1
Author: Xu Kuohai <[email protected]>
Date:   Fri Jul 19 19:00:53 2024 +0800

    bpf: Prevent tail call between progs attached to different hooks
    
    [ Upstream commit 28ead3eaabc16ecc907cfb71876da028080f6356 ]
    
    bpf progs can be attached to kernel functions, and the attached functions
    can take different parameters or return different return values. If
    prog attached to one kernel function tail calls prog attached to another
    kernel function, the ctx access or return value verification could be
    bypassed.
    
    For example, if prog1 is attached to func1 which takes only 1 parameter
    and prog2 is attached to func2 which takes two parameters. Since verifier
    assumes the bpf ctx passed to prog2 is constructed based on func2's
    prototype, verifier allows prog2 to access the second parameter from
    the bpf ctx passed to it. The problem is that verifier does not prevent
    prog1 from passing its bpf ctx to prog2 via tail call. In this case,
    the bpf ctx passed to prog2 is constructed from func1 instead of func2,
    that is, the assumption for ctx access verification is bypassed.
    
    Another example, if BPF LSM prog1 is attached to hook file_alloc_security,
    and BPF LSM prog2 is attached to hook bpf_lsm_audit_rule_known. Verifier
    knows the return value rules for these two hooks, e.g. it is legal for
    bpf_lsm_audit_rule_known to return positive number 1, and it is illegal
    for file_alloc_security to return positive number. So verifier allows
    prog2 to return positive number 1, but does not allow prog1 to return
    positive number. The problem is that verifier does not prevent prog1
    from calling prog2 via tail call. In this case, prog2's return value 1
    will be used as the return value for prog1's hook file_alloc_security.
    That is, the return value rule is bypassed.
    
    This patch adds restriction for tail call to prevent such bypasses.

How to reproduce it

As the Garden Linux kernels don't contain patches in the bpf module I would expect this problem to be reproducible on Kubernetes nodes running a Linux 6.6.x LTS Kernel, x >= 57.

Expected behaviour

Falco should run.

Screenshots

Environment

We ran this in Kubernetes clusters managed by our Gardener environment (https://gardener.cloud) running Garden Linux as operating system. As Falco generally runs fine on all kinds of Kubernetes- and OS versions we believe that this is not relevant.

• Falco version: 0.37.1, 0.39.1

• System info: Linux ip-10-180-26-208.eu-west-1.compute.internal 6.6.62-cloud-amd64 Digwatch compiler #1 SMP PREEMPT_DYNAMIC Debian 6.6.62-0gl1~bp1592 (2024-11-19) x86_64 GNU/Linux

• Cloud provider or hardware configuration:
• OS: Garden Linux

• Kernel: 6.6.x, x >= 57

• Installation method:

Additional context

I haven't tried Falco 0.39.2 but can do so, however I did not see anything in the release notes that indicates a fix for such an issue. I can provide further assistance if needed.

[Andreagit97]

Hi @marwinski! Thank you for reporting! Falco 0.39.2 should fix exactly the issue you mentioned bpf: Prevent tail call between progs attached to different hooks

I haven't tried Falco 0.39.2 but can do so, however I did not see anything in the release notes that indicates a fix for such an issue. I can provide further assistance if needed.

Unfortunately, you are right, the fix was in another repo and so you can find the changelog there https://github.com/falcosecurity/libs/releases/tag/0.18.2

[Andreagit97]

just tried on 6.6.62 and 6.6.63 and it works well

[marwinski]

Thanks.

I have just tested it and it works fine with 6.6.62 on my side. I should have checked the other repos...

[Andreagit97]

don't worry! I will close this, thank you for the quick feedback!