Integration between Falco and FalcoSidekick #2565

chaitu577 · 2023-05-22T11:25:32Z

Hi Team,

Can anyone point me to the documentation or guide me on how to integrate Falco (setup using helm on Kubernetes - https://falco.org/docs/getting-started/try-falco/try-falco-on-kubernetes/) and Falco Sidekick / UI(setup reference: https://github.com/falcosecurity/falcosidekick, https://github.com/falcosecurity/falcosidekick-ui) coz I don't see any events on the Web UI, thanks.

Regards,
Che

Issif · 2023-05-22T11:44:56Z

Hi,
Do you use helm for your deployment? If you do, by just setting --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true with the Falco chart, everything will be configured for you

chaitu577 · 2023-05-22T13:54:16Z

Hi Issif,

I did the deployment using Helm, but when I tried setting the same, the WebUI is not accessible due to storage issues (I mean we don't have any persistent storage in k8's env. which Redis is expecting)

Issif · 2023-05-22T14:08:18Z

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

chaitu577 · 2023-05-23T10:41:23Z

Thanks Issif, deployed the same by setting the above flag, but i cant access the webUI via 2802 port.

kubectl port-forward svc/falco-1684835778-falcosidekick-ui 2802:2802
Forwarding from 127.0.0.1:2802 -> 2802

-----------------------------------kubectl get pods-----------------------------------------------------------------------
NAME READY STATUS RESTARTS AGE
alpine-deployment-65c65b9476-b5b2g 1/1 Running 0 3d21h
falco-1684835778-9pmjd 0/2 Init:CrashLoopBackOff 10 (2m2s ago) 41m
falco-1684835778-falcosidekick-5fbdc8b559-nzbwv 1/1 Running 0 41m
falco-1684835778-falcosidekick-5fbdc8b559-tq2zc 1/1 Running 0 41m
falco-1684835778-falcosidekick-ui-799b77d96-282rv 0/1 CrashLoopBackOff 16 (4m54s ago) 41m
falco-1684835778-falcosidekick-ui-799b77d96-qnkhr 0/1 Running 17 (5m14s ago) 41m
falco-1684835778-falcosidekick-ui-redis-0 1/1 Running 0 41m
falco-1684835778-j4xnm 0/2 Init:CrashLoopBackOff 10 (92s ago) 41m
falco-1684835778-n2pqd 0/2 Init:CrashLoopBackOff 10 (116s ago) 41m
falco-1684835778-sd2qj 0/2 Init:CrashLoopBackOff 10 (2m12s ago) 41m
falco-1684835778-wpdkj 0/2 Init:CrashLoopBackOff 10 (2m18s ago) 41m
falco-1684835778-wxbxr 0/2 Init:CrashLoopBackOff 10 (118s ago) 41m
falco-56fmc 2/2 Running 0 3d22h
falco-6xhjv 2/2 Running 0 3d22h
falco-cg9dg 2/2 Running 0 3d22h
falco-klpjd 2/2 Running 0 3d22h
falco-q4cr7 2/2 Running 0 3d22h
falco-xd497 2/2 Running 0 3d22h

Issif · 2023-05-23T11:02:06Z

what errors and logs do you have?

chaitu577 · 2023-05-23T11:32:20Z

this is wat I see

chaitu577 · 2023-05-23T11:34:24Z

kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
falco-1684835778-falcosidekick ClusterIP 10.1.1.6 2801/TCP 93m
falco-1684835778-falcosidekick-ui ClusterIP 10.1.1.1 2802/TCP 93m
falco-1684835778-falcosidekick-ui-redis ClusterIP 10.1.9.2 6379/TCP 93m

Issif · 2023-05-23T14:40:23Z

At falcosidekick level it's ok, and for falcosidekick-ui? Is the redis correctly running? When you try to access through the port-forward, what http error do you get?

chaitu577 · 2023-05-24T10:58:11Z

Yep redis is up & running, PFB snip.

docker exec -it 7d705638e61b redis-cli

127.0.0.1:6379> ping
PONG
127.0.0.1:6379>

this is the error I see now, thanks.

Issif · 2023-05-24T11:31:03Z

Your service name is strange, on my side, I start the port-forward with just:

kubectl port-forward svc/falco-falcosidekick-ui 2802:2802 -n falco

Can you list your services to be sure please?

chaitu577 · 2023-05-24T13:41:27Z

Here we go:

Issif · 2023-05-24T14:01:08Z

Everything is ok at this level, I don't understand why the port-forward fails. It doesn't seem related to falcosidekick-ui but more on your K8s config. Any CNI that could create an issue? Are you able to create a port-forward to another svc? Can you try directly to the pod?

chaitu577 · 2023-05-29T09:50:21Z

I've deployed a new Redis server on a different machine, can you please check the below & correct it, as executing the below command pick the IP of Redis Cluster IP(from kubectl get svc) not the one passed below, thanks.

helm install falco -n falco --set tty=true falcosecurity/falco
--set falcosidekick.enabled=true
--set redis.address=rndcrs101
--set FALCOSIDEKICK_UI_REDIS_URL=10.X.X.X:6379
--set falcosidekick.webui.enabled=true

Issif · 2023-05-29T10:21:03Z

Here's the correct syntax:

helm install falco -n falco --set tty=true falcosecurity/falco
  --set falcosidekick.enabled=true
  --set falcosidekick.webui.enabled=true
  --set falcosidekick.webui.externalRedis.enabled=true
  --set falcosidekick.webui.externalRedis.url=10.X.X.X
  --set falcosidekick.webui.externalRedis.port=6379

I agree falcosidekick.webui.externalRedis.url is not well named.

chaitu577 · 2023-05-29T11:31:07Z

Thanks, Issif, I see the below error where I can't find the deployment-ui in the falcosidekick GitHub repo, but it exists in the charts repo here(https://github.com/falcosecurity/charts/tree/master/falcosidekick/templates). do I need to clone it and do something?

Error: INSTALLATION FAILED: execution error at (falco/charts/falcosidekick/templates/deployment-ui.yaml:3:5): Both webui.redis and webui.externalRedis modules are enabled. Please disable one of them.

Issif · 2023-05-29T14:34:09Z

Add --set falcosidekick.webui.redis.enabled=false.

All values are there: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml

chaitu577 · 2023-05-29T17:16:50Z

it worked but the UI pod's arent coming up..!

kubectl get pods
NAME READY STATUS RESTARTS AGE
alpine-deployment-65c65b9476-g6vft 1/1 Running 0 3d21h
falco-b5sqt 2/2 Running 0 114s
falco-falcosidekick-5dd98bcdc4-4vw24 1/1 Running 0 114s
falco-falcosidekick-5dd98bcdc4-84qtg 1/1 Running 0 114s
falco-falcosidekick-ui-595b78c7c7-4q6f5 0/1 CrashLoopBackOff 4 (7s ago) 114s
falco-falcosidekick-ui-595b78c7c7-w7t7f 0/1 CrashLoopBackOff 4 (16s ago) 114s
falco-fcmgm 2/2 Running 0 114s
falco-h5fb5 2/2 Running 0 114s
falco-m27jk 2/2 Running 0 114s
falco-pbf54 2/2 Running 0 114s
falco-s2dt2 2/2 Running 0 114s

kubectl logs falco-falcosidekick-ui-595b78c7c7-w7t7f
2023/05/29 15:32:23 [WARN] : Index does not exist
2023/05/29 15:32:23 [WARN] : Create Index
2023/05/29 15:32:23 ERR unknown command FT.CREATE, with args beginning with: eventIndex, SCHEMA, output, TEXT, rule, TEXT, priority, TEXT, hostname, TEXT, source, TEXT, tags, TEXT,

Issif · 2023-05-31T10:54:43Z

Are you sure to run a redis instance with redisearch module? It's a requirement.

chaitu577 · 2023-06-02T07:43:21Z

Yes, installed redisearch and all pods are up and running, also i did port-forward the UI pod to 2802 and i see the UI console as well, very much thanks Issif.

now the issue is I'm unable to see the events triggered on Web UI console. Any ideas?

I see the config is pointing to the newly installed Redis instance as well.

UI

Issif · 2023-06-02T09:37:30Z

Can you check in logs if :

falco triggered an alert
falcosidekick received the alert
falcosidekick forward the alert to the UI

chaitu577 · 2023-06-07T08:38:15Z

Sure, PFB logs.

Triggered Alert
kubectl exec -it alpine-deployment-65c65b9476-g6vft -- sh -c "cat /etc/passwd"
root:x:0:0:root:/root:/bin/ash

Falco Logs
docky]# for pod in $(kubectl get pods -o=name); do kubectl logs "$pod" --all-containers | grep "passwd"; done
{"hostname":"falco-pbf54","output":"07:32:08.067095951: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=falco k8s.pod=alpine-deployment-65c65b9476-g6vft container=4dca6289b5c9 shell=sh parent=runc cmdline=sh -c cat /etc/passwd pid=1876022 terminal=34816 container_id=4dca6289b5c9 image=docker.io/library/alpine)","priority":"Notice","rule":"Terminal shell in container","source":"syscall","tags":["T1059","container","mitre_execution","shell"],"time":"2023-06-02T07:32:08.067095951Z", "output_fields": {"container.id":"4dca6289b5c9","container.image.repository":"docker.io/library/alpine","evt.time":1685691128067095951,"k8s.ns.name":"falco","k8s.pod.name":"alpine-deployment-65c65b9476-g6vft","proc.cmdline":"sh -c cat /etc/passwd","proc.name":"sh","proc.pid":1876022,"proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

Falco Sidekick Logs

kubectl logs falco-falcosidekick-5dd98bcdc4-84qtg -c falcosidekick
2023/05/29 15:29:25 [INFO] : Falco Sidekick version: 2.27.0
2023/05/29 15:29:25 [INFO] : Enabled Outputs : [WebUI]
2023/05/29 15:29:25 [INFO] : Falco Sidekick is up and listening on :2801

Falco Sidekick UI Logs
kubectl logs falco-falcosidekick-ui-595b78c7c7-kbbc5 -c falcosidekick-ui
2023/06/01 20:22:17 [INFO] : Falcosidekick UI is listening on 0.0.0.0:2802
2023/06/01 20:22:17 [INFO] : log level is info
2023/06/02 07:31:40 [INFO] : user 'admin' authenticated
2023/06/05 08:26:04 [INFO] : user 'admin' authenticated
2023/06/07 07:22:05 [INFO] : user 'admin' authenticated
[root@ docky]# kubectl logs falco-falcosidekick-ui-595b78c7c7-rl9s8
2023/06/01 20:23:40 [INFO] : Falcosidekick UI is listening on 0.0.0.0:2802
2023/06/01 20:23:40 [INFO] : log level is info

Issif · 2023-06-07T10:35:05Z

Are you running falcosidekick-ui on arm64? We discovered a strange bug falcosecurity/falcosidekick-ui#95

For now, I don't have access to an arm64 machine to reproduce.

chaitu577 · 2023-06-07T12:14:35Z

oops, its x86 64-bit architecture ("AMD64" or "Intel 64") but not arm64.

uname -mvnrs
Linux abc.xx.com 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64

Issif · 2023-06-07T12:28:46Z

Can you paste me your whole values.yaml please? Just edit the sensitive data. Right now, I don't see any issue.

chaitu577 · 2023-06-07T17:20:04Z

Here's the correct syntax:

helm install falco -n falco --set tty=true falcosecurity/falco
  --set falcosidekick.enabled=true
  --set falcosidekick.webui.enabled=true
  --set falcosidekick.webui.externalRedis.enabled=true
  --set falcosidekick.webui.externalRedis.url=10.X.X.X
  --set falcosidekick.webui.externalRedis.port=6379

I agree falcosidekick.webui.externalRedis.url is not well named.

I've used helm to install, its fetching directly from falcosecurity/falco.

Issif · 2023-06-08T09:49:46Z

I don't see any issue with this config. Can you try to create a port-forward to a falcosidekick pod and trigger a test event:

kubectl port-forward svc/falco-falcosidekick 2801:2801 -n falco
curl -XPOST "http://localhost:2801/test"

If you see something in the falcosidekick logs and in the webui, it means the issue is between falco and falcosidekick.

Do you have any specific networkpolicies? or cilium? or calico? that could interfere?

chaitu577 · 2023-06-08T12:00:18Z

I see no errors/handling connection when using 127.0.0.1 and it throws an connection refused error while using localhost/hostname:2801, I'm using calico but i dont see anything that's blocking, any specific checks/info that u need? please let me know, thanks.

curl -XPOST "http://XXXX:2801/test"
curl: (7) Failed connect to XXXX:2801; Connection refused

kubectl port-forward svc/falco-falcosidekick 2801:2801 -n falco
Forwarding from 127.0.0.1:2801 -> 2801
Handling connection for 2801

kubectl logs falco-falcosidekick-5dd98bcdc4-4vw24
2023/05/29 15:29:24 [INFO] : Falco Sidekick version: 2.27.0
2023/05/29 15:29:24 [INFO] : Enabled Outputs : [WebUI]
2023/05/29 15:29:24 [INFO] : Falco Sidekick is up and listening on :2801
2023/06/08 11:47:38 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout
2023/06/08 11:47:38 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout
2023/06/08 11:51:21 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout
2023/06/08 11:51:21 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout
2023/06/08 11:53:33 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout
2023/06/08 11:53:33 [ERROR] : WebUI - Post "http://falco-falcosidekick-ui:2802": dial tcp 10.X:2802: i/o timeout

Issif · 2023-06-08T12:21:31Z

I see the issue now. For a reason I don't understand, your svc for falcosidekick-ui is falco-1684835778-falcosidekick-ui but the helm chart has used falco-falcosidekick-ui.

The template is this one:

  {{- if .Values.webui.enabled -}}
  {{ $weburl := printf "http://%s-ui:2802" (include "falcosidekick.fullname" .) }}
  WEBUI_URL: "{{ $weburl | b64enc }}"
  {{- end }}

With falcosidekick.fullname defines as:

{{- define "falcosidekick.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

It should work, but it doesn't, I need to search where the bug is.

chaitu577 · 2023-06-08T13:58:14Z

Yes, previously it was like that due to prior installations may be, but currently my SVC are below, please check & assist.

kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
falco-falcosidekick ClusterIP 10.XXX 2801/TCP 9d
falco-falcosidekick-ui ClusterIP 10.1XXXX 2802/TCP 9d

Issif · 2023-06-08T15:45:25Z

As everything is stateless, can you delete your deployments and all resources and redeploy everything, by checking the names?

dneto82 · 2023-12-20T20:43:43Z

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

This save me when was evaluating at stateless scenario such minikube / eks. I hope we should quote this in doc as note in "evaluation / trying mode".

Issif · 2023-12-21T11:35:26Z

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

This save me when was evaluating at stateless scenario such minikube / eks. I hope we should quote this in doc as note in "evaluation / trying mode".

Feel free to submit a PR to add it 😉

poiana · 2024-03-20T15:49:55Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

poiana · 2024-04-19T15:51:52Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

poiana · 2024-05-19T15:52:57Z

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

poiana · 2024-05-19T15:53:01Z

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Github-user191 · 2024-08-29T18:50:48Z

I see, you can disable the PVC but all events will be kept in memory: --set falcosidekick.webui.redis.storageEnabled=false

Thank you! Was stuck for an hour and setting this flag in my helm install worked and my redis could start

chaitu577 added the kind/documentation label May 22, 2023

Andreagit97 added this to the TBD milestone Aug 31, 2023

poiana added the lifecycle/stale label Mar 20, 2024

poiana added lifecycle/rotten and removed lifecycle/stale labels Apr 19, 2024

poiana closed this as completed May 19, 2024

Github-user191 mentioned this issue Aug 29, 2024

Helm Install Falco/Falcosidekick Redis Pods not starting FIX kodekloudhub/devsecops#9

Open

Integration between Falco and FalcoSidekick #2565

Integration between Falco and FalcoSidekick #2565

Comments

chaitu577 commented May 22, 2023

Issif commented May 22, 2023 • edited Loading

chaitu577 commented May 22, 2023

Issif commented May 22, 2023

chaitu577 commented May 23, 2023

Issif commented May 23, 2023

chaitu577 commented May 23, 2023

chaitu577 commented May 23, 2023

Issif commented May 23, 2023 • edited Loading

chaitu577 commented May 24, 2023

docker exec -it 7d705638e61b redis-cli

Issif commented May 24, 2023

chaitu577 commented May 24, 2023

Issif commented May 24, 2023

chaitu577 commented May 29, 2023

Issif commented May 29, 2023

chaitu577 commented May 29, 2023

Issif commented May 29, 2023

chaitu577 commented May 29, 2023

Issif commented May 31, 2023

chaitu577 commented Jun 2, 2023

Issif commented Jun 2, 2023

chaitu577 commented Jun 7, 2023

Issif commented Jun 7, 2023

chaitu577 commented Jun 7, 2023

Issif commented Jun 7, 2023

chaitu577 commented Jun 7, 2023

Issif commented Jun 8, 2023 • edited Loading

chaitu577 commented Jun 8, 2023

Issif commented Jun 8, 2023

chaitu577 commented Jun 8, 2023

Issif commented Jun 8, 2023

dneto82 commented Dec 20, 2023

Issif commented Dec 21, 2023

poiana commented Mar 20, 2024

poiana commented Apr 19, 2024

poiana commented May 19, 2024

poiana commented May 19, 2024

Github-user191 commented Aug 29, 2024

Issif commented May 22, 2023 •

edited

Loading

Issif commented May 23, 2023 •

edited

Loading

Issif commented Jun 8, 2023 •

edited

Loading