tetragon: harden actions a bit #3279

olsajiri · 2025-01-07T09:17:17Z

Adding new filter logic that triggers when process is not found in
execve_map (likely due to exhausted capacity).

In this case we currently exit the bpf program and that might leave
some crucial actions not executed (like sigkill).

We add new event_find_curr_probe call to gather process info in filter
context and use it to execute filters. The resulting kprobe event will
have minimal process data with 'unknown' flags field.

mtardy · 2025-01-08T09:40:16Z

pkg/sensors/base/base.go

+	threads := readFileDefault("/proc/sys/kernel/threads-max", 32768)
+	ExecveMap.SetMaxEntries(int(threads))


Why do we need this? It seems it will make the size of the execve_map even larger as threads-max is often well above 32K. This will take significant space while running threads-max threads is pretty rare nop?

I tried this as mitigation for https://github.com/isovalent/security/issues/88 .. I think we need some combination of this change (with some reasonable size for execve_map) and other ways mentioned in the issue

ah I see, maybe this is one of the cases where NO_PREALLOC could help so that we can dynamically size to a very large map. But I could see how this can lead to memory issues in the future. That's not an easy problem :/

netlify · 2025-01-13T14:29:46Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`18aac9b`
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/67bd7cbd499cd400081ef1ac
😎 Deploy Preview	https://deploy-preview-3279--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

kkourt · 2025-01-17T10:42:02Z

pkg/option/flags.go

@@ -416,4 +419,6 @@ func AddFlags(flags *pflag.FlagSet) {
 	flags.Int(KeyEventCacheRetryDelay, defaults.DefaultEventCacheRetryDelay, "Delay in seconds between event cache retries")

 	flags.Bool(KeyCompatibilitySyscall64SizeType, false, "syscall64 type will produce output of type size (compatibility flag, will be removed in v1.4)")
+
+	flags.String(KeyExecveMapEntries, "", "Set entries for execve_map table (default 32768)")


Can we have the default from the actual default value here? If we ever change it, the change should be reflected in help.

olsajiri · 2025-02-19T11:34:02Z

note the checkpatch fails on container_of macro which is false alarm

mtardy

Thank you, there are a few things I didn't understand though.

mtardy · 2025-02-24T10:47:20Z

bpf/process/generic_calls.h

-	e->common.flags |= MSG_COMMON_FLAG_RETURN;
+	e->common.flags = MSG_COMMON_FLAG_RETURN;


not sure to follow why we want this change?

setting flags early in the processing so in following changes the filter tail call can change it (which is executed before current flags setup code) (from commit changelog)

hum for that specific line we need to initialize the flags for return kprobe, I guess it could have probably went to separate patch

mtardy · 2025-02-24T10:49:36Z

bpf/process/bpf_process_event.h

+#ifdef __LARGE_BPF_PROG
+FUNC_INLINE struct execve_map_value *
+event_find_curr_probe(struct msg_generic_kprobe *msg)
+{
+	struct task_struct *task = (struct task_struct *)get_current_task();
+	struct execve_map_value *curr;
+
+	curr = &msg->curr;
+	curr->key.pid = BPF_CORE_READ(task, tgid);
+	curr->key.ktime = ktime_get_ns();
+	curr->nspid = get_task_pid_vnr_by_task(task);
+
+	get_current_subj_caps(&curr->caps, task);
+	get_namespaces(&curr->ns, task);
+	set_in_init_tree(curr, NULL);
+
+	read_exe((struct task_struct *)get_current_task(), &msg->exe);
+	return curr;
+}
+#else
+FUNC_INLINE struct execve_map_value *
+event_find_curr_probe(struct msg_generic_kprobe *msg)
+{
+	return NULL;
+}
+#endif


what's the benefit of doing this instead of writing the function unconditionally and making the calling conditional to LARGE PROGS? Wouldn't that be more explicit that non large prog do not support that?

well I think it's logically isolated piece of code which should be in function and is different for __LARGE_BPF_PROG define, so it makes sense to me to do the above

also I think it's better/readable/more manageble than having ifdefs inside the code

mtardy · 2025-02-24T10:53:02Z

bpf/process/generic_calls.h

+	if (!enter) {
+		enter = event_find_curr_probe(msg);
+		if (!enter)
+			return PFILTER_CURR_NOT_FOUND;


is it okay that this function always return PFILTER_CURR_NOT_FOUND on !__LARGE_BPF_PROG kernels? I guess this related to my other comment, there's something I don't understand I think

hum, not sure I understand, even current code returns PFILTER_CURR_NOT_FOUND

mtardy · 2025-02-24T10:54:16Z

pkg/sensors/tracing/kprobe_sigkill_test.go

+		t.Skip("sigkill requires at least 5.3.0 version")
+	}
+
+	// makeSpecFile creates a new spec file bsed on the template, and the provided arguments


nit: we don't really care but since I saw it, little typo here bsed to based

mtardy · 2025-02-24T10:56:12Z

pkg/sensors/tracing/kprobe_sigkill_test.go

+
+	option.Config.ExecveMapEntries = 1
+	testSigkill(t, makeSpecFile, checker)
+	option.Config.ExecveMapEntries = 0


why the set to 0 here?

IIUC option.Config is global, so we need to reset option.Config.ExecveMapEntries so other tests will get default value

Adding new 'unknown' value for flags field in the message Process. It will be used in following changes for process without details. Signed-off-by: Jiri Olsa <[email protected]>

Moving d_path into bpf_d_path.h header so it can be easily used in following changes from other places. Signed-off-by: Jiri Olsa <[email protected]>

Moving read_exe into process.h header so it can be easily used in following changes from other places. Signed-off-by: Jiri Olsa <[email protected]>

Setting flags early in the processing so in following changes the filter tail call can change it (which is executed before current flags setup code). Signed-off-by: Jiri Olsa <[email protected]>

Adding new filter logic that triggers when process is not found in execve_map (likely due to exhausted capacity). In this case we currently exit the bpf program and that might leave some crucial actions not executed (like sigkill). We add new event_find_curr_probe call to gather process info in filter context and use it to execute filters. The resulting kprobe event will have minimal process data with 'unknown' flags field. Signed-off-by: Jiri Olsa <[email protected]>

Adding test for unknown process kprobe kill. Signed-off-by: Jiri Olsa <[email protected]>

Adding test for unknown process tracepoint kill. Signed-off-by: Jiri Olsa <[email protected]>

kkourt

LGTM, thanks

olsajiri added the release-note/minor This PR introduces a minor user-visible change label Jan 7, 2025

olsajiri force-pushed the pr/olsajiri/harden branch 3 times, most recently from 7b664dd to ab3513b Compare January 7, 2025 13:31

mtardy reviewed Jan 8, 2025

View reviewed changes

olsajiri force-pushed the pr/olsajiri/harden branch 2 times, most recently from c64e7e4 to fcfa0a9 Compare January 13, 2025 14:28

kkourt reviewed Jan 17, 2025

View reviewed changes

olsajiri force-pushed the pr/olsajiri/harden branch 7 times, most recently from db51eba to 44fdaa4 Compare January 21, 2025 11:17

olsajiri force-pushed the pr/olsajiri/harden branch from 44fdaa4 to 6922611 Compare February 3, 2025 15:50

olsajiri changed the title ~~tetragon: Setup execve_map max entries~~ tetragon: harden actions a bit Feb 3, 2025

olsajiri force-pushed the pr/olsajiri/harden branch 5 times, most recently from b8b9d4e to bafb514 Compare February 16, 2025 14:15

olsajiri force-pushed the pr/olsajiri/harden branch 3 times, most recently from 6132585 to f6e5835 Compare February 18, 2025 22:05

olsajiri marked this pull request as ready for review February 19, 2025 09:13

olsajiri requested a review from a team as a code owner February 19, 2025 09:13

olsajiri requested a review from jrfastab February 19, 2025 09:13

olsajiri requested review from kkourt and mtardy February 24, 2025 10:50

mtardy approved these changes Feb 24, 2025

View reviewed changes

olsajiri added 7 commits February 25, 2025 08:09

tetragon: Add unknown process flag

e26f1a1

Adding new 'unknown' value for flags field in the message Process. It will be used in following changes for process without details. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Move d_path into bpf_d_path.h header

c93ff1a

Moving d_path into bpf_d_path.h header so it can be easily used in following changes from other places. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Move read_exe into process.h header

ff464d3

Moving read_exe into process.h header so it can be easily used in following changes from other places. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Set flags early in processing

7c335b4

Setting flags early in the processing so in following changes the filter tail call can change it (which is executed before current flags setup code). Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for unknown process kprobe kill

411a0e9

Adding test for unknown process kprobe kill. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for unknown process tracepoint kill

18aac9b

Adding test for unknown process tracepoint kill. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/harden branch from f6e5835 to 18aac9b Compare February 25, 2025 08:18

kkourt approved these changes Feb 25, 2025

View reviewed changes

olsajiri merged commit 48c5066 into main Feb 25, 2025
51 of 52 checks passed

olsajiri deleted the pr/olsajiri/harden branch February 25, 2025 09:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tetragon: harden actions a bit #3279

tetragon: harden actions a bit #3279

olsajiri commented Jan 7, 2025 •

edited

Loading

mtardy Jan 8, 2025

olsajiri Jan 8, 2025

mtardy Jan 10, 2025

netlify bot commented Jan 13, 2025 •

edited

Loading

kkourt Jan 17, 2025

olsajiri commented Feb 19, 2025

mtardy left a comment

mtardy Feb 24, 2025

olsajiri Feb 25, 2025

olsajiri Feb 25, 2025

mtardy Feb 24, 2025

olsajiri Feb 25, 2025

mtardy Feb 24, 2025

olsajiri Feb 25, 2025

mtardy Feb 24, 2025

mtardy Feb 24, 2025

olsajiri Feb 25, 2025

kkourt left a comment

		threads := readFileDefault("/proc/sys/kernel/threads-max", 32768)
		ExecveMap.SetMaxEntries(int(threads))

		e->common.flags \|= MSG_COMMON_FLAG_RETURN;
		e->common.flags = MSG_COMMON_FLAG_RETURN;

tetragon: harden actions a bit #3279

tetragon: harden actions a bit #3279

Conversation

olsajiri commented Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netlify bot commented Jan 13, 2025 • edited Loading

✅ Deploy Preview for tetragon ready!

Choose a reason for hiding this comment

olsajiri commented Feb 19, 2025

mtardy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkourt left a comment

Choose a reason for hiding this comment

olsajiri commented Jan 7, 2025 •

edited

Loading

netlify bot commented Jan 13, 2025 •

edited

Loading