Segmentation fault

[michalschott]

Bug Report

General Information

• Environment description (GKE, VM-Kubeadm, vagrant-dev-env, minikube, microk8s, ...)
• Kernel version (run uname -a)
• Orchestration system version in use (e.g. kubectl version, ...)
• Link to relevant artifacts (policies, deployments scripts, ...)
• Target containers/pods

EKS 1.22
Bottlerocket OS 1.9.0 (aws-k8s-1.22)
kernel 5.10.130
containerd://1.6.6+bottlerocket

Noticed these segfaults errors in kubearmor daemonset pods:

2022-08-10 15:29:57.382655      INFO    Build Time: 2022-08-05 11:15:05.638889565 +0000 UTC
2022-08-10 15:29:57.382780      INFO    Arguments [cluster:default coverageTest:false criSocket: defaultCapabilitiesPosture:block defaultFilePosture:block defaultNetworkPosture:block enableKubeArmorHostPolicy:false enableKubeArmorPolicy:true enableKubeArmorVm:false gRPC:32767 host:ip-10-88-60-49 hostDefaultCapabilitiesPosture:block hostDefaultFilePosture:block hostDefaultNetworkPosture:block hostVisibility:default k8s:true logPath:/tmp/kubearmor.log seLinuxProfileDir:/tmp/kubearmor.selinux visibility:process,file,network,capabilities]
2022-08-10 15:29:57.382938      INFO    Configuration [{Cluster:default Host:ip-10-88-60-49 GRPC:32767 LogPath:/tmp/kubearmor.log SELinuxProfileDir:/tmp/kubearmor.selinux CRISocket:unix:///run/dockershim.sock Visibility:process,file,network,capabilities HostVisibility:default Policy:true HostPolicy:true KVMAgent:false K8sEnv:true DefaultFilePosture:block DefaultNetworkPosture:block DefaultCapabilitiesPosture:block HostDefaultFilePosture:block HostDefaultNetworkPosture:block HostDefaultCapabilitiesPosture:block CoverageTest:false}]
2022-08-10 15:29:57.382970      INFO    Final Configuration [{Cluster:default Host:ip-10-88-60-49 GRPC:32767 LogPath:/tmp/kubearmor.log SELinuxProfileDir:/tmp/kubearmor.selinux CRISocket:unix:///run/dockershim.sock Visibility:process,file,network,capabilities HostVisibility:none Policy:true HostPolicy:true KVMAgent:false K8sEnv:true DefaultFilePosture:block DefaultNetworkPosture:block DefaultCapabilitiesPosture:block HostDefaultFilePosture:block HostDefaultNetworkPosture:block HostDefaultCapabilitiesPosture:block CoverageTest:false}]
2022-08-10 15:29:57.383407      INFO    Initialized Kubernetes client
2022-08-10 15:29:57.383511      INFO    Started to monitor node events
2022-08-10 15:29:57.383563      INFO    GlobalCfg.Host=ip-10-88-60-49, KUBEARMOR_NODENAME=ip-10-88-60-49.eu-west-1.compute.internal
2022-08-10 15:29:58.384133      INFO    Node Name: ip-10-88-60-49
2022-08-10 15:29:58.384183      INFO    Node IP: 10.88.60.49
2022-08-10 15:29:58.384214      INFO    Node Annotations: map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-097c9fe24d3a1c55f"} io.cilium.network.ipv4-cilium-host:10.88.60.243 io.cilium.network.ipv4-health-ip:10.88.60.167 io.cilium.network.ipv4-pod-cidr:10.49.0.0/16 kubearmor-policy:audited kubearmor-visibility:none node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true]
2022-08-10 15:29:58.384229      INFO    OS Image: Bottlerocket OS 1.9.0 (aws-k8s-1.22)
2022-08-10 15:29:58.384241      INFO    Kernel Version: 5.10.130
2022-08-10 15:29:58.384295      INFO    Kubelet Version: v1.22.10-eks-7dc61e8
2022-08-10 15:29:58.384364      INFO    Container Runtime: containerd://1.6.6+bottlerocket
2022-08-10 15:29:58.384875      INFO    Initialized KubeArmor Logger
2022-08-10 15:29:58.387239      INFO    checking if kernel headers path (/media/root/usr/src/linux-headers-5.10.130) exists
2022-08-10 15:29:58.387389      INFO    Initializing an eBPF program
/KubeArmor/entrypoint.sh: line 34: 28599 Segmentation fault      (core dumped) /KubeArmor/kubearmor ${ARMOR_OPTIONS[@]}
Error code: 139

[nyrahul]

Thank you @michalschott for reporting this. Any chance you have the cluster deployment yaml that we can use to mimic this cluster creation? Thanks

[nyrahul]

Hey @michalschott , we figured out the reason for this segmentation fault. We had faced this issue before with GKE COS k8s clusters as well and the issue is related to BCC dumping core. The kubearmor eventually starts up ok after few attempts i.e. BCC recovers eventually.

In the latest kubearmor code, we have removed BCC dependency altogether (#677) and we do not see this problem anymore. However, the current stable version points to v0.5 that does not have this fix. Kubearmor community is working towards getting the v0.6 release made by the end of August.

[michalschott]

Hey @nyrahul indeed it eventually recovers, thanks for an update. Looking forward for 0.6 release!

[Ankurk99]

The issue is fixed with BCC dependency removal in v0.6 release.