Monitor changes in ioctl flags in file inodes / auditd backend

[thesw4rm]

I'm going to talk Linux specific here because that's the only kernel I am familiar with. However, I think the following ideas can be ported to other Unix systems.

In Linux, file inodes have special characteristics (referred to as inode flags), of which some relate to input output control. They are listed here in the Linux source https://github.com/torvalds/linux/blob/ad1d69735878a6bf797705b5d2a20316d35e1113/include/uapi/linux/fs.h#L334.

I believe FUSE filesystems can watch for changes to these flags via the ioctl handler. Was wondering if similar could be possible with a file watcher instead? As an example, what I'm trying to implement right now is a watcher for the CLOSE_NOWRITE event, which when triggered, opens the inode on the specific file and checks for a change in immutability. The program initially caches the immutable bit option (whether it is set or not) for each file that it watches.

Problem is that I keep triggering my own CLOSE_NOWRITE handler by opening and closing the inode, and for some reason inotify doesn't record the PID of the process that triggered the event, so I keep triggering my own handler. LOL.

Any way to fix this in this Rust package? Perhaps a custom C binding for a watcher that exposes PID of origin.

Also, inotify has extended attribute support (IN_ATTRIB). Can we add that to this Rust wrapper?

[passcod]

That would entail writing a completely new backend that doesn't use inotify at all, right? It's... certainly possible, but non-trivial at this point. (It will be somewhat easier in vNext.)

[passcod]

Notify also already exposes inotify's IN_ATTRIB events (as Op::CHMOD).

[thesw4rm]

I'll respond to both in one comment.

1. Not necessarily. I haven't looked far enough into notify's architecture to give a definite answer here, but I was thinking that with the "next-gen"ish file watchers coming out in new Linux kernels (like fanotify), it might be good to change at least the Linux design to be more abstract. So whatever stuff is available for inotify right now in this project can be ported to fanotify (as an example). Then with fanotify, you can also get PID of the process that triggered the event (although I think fanotify is only there in 3.x kernels and later). So essentially then all we need to do is have a CLOSE_NOWRITE handler, and make the handler check for immutability changes, while ignoring triggers where the PID that triggered the handler matches one of our dynamically created PID ignore list (in case of multiple threads and stuff).

Or of course the dirty bit option which I don't like, because then you get all these permission problems and actually have to mess with the file, which defeats the point of a watcher somewhat.

2. Correct, but chmod only does general file permissions. I'm talking about setfattr and getfattr related things, where you set user and system defined extended attributes at the end of a file. These work differently on different filesystems though. IN_ATTRIB includes these changes in the inotify system, but this Rust package does not identify xattr changes as different from chmod, as far as I have seen.

[passcod]

Correct, but chmod only does general file permissions.

No, I mean, this library translates IN_ATTRIB to Op::CHMOD. (There was a call to rename that event Op::METADATA to indicate that it's wider than just permissions, but such a change would break the API.) Inotify doesn't give us any more detail within that event type, and therefore we don't differentiate. We can't report information we don't get.

So whatever stuff is available for inotify right now in this project can be ported to fanotify

But it can't! Don't get me wrong: I am excited about fanotify for the limitations it removes on a subset of what inotify does, but currently fanotify has severe limitations (such as requiring root) and conceptual differences with inotify in its usage. (If anything, adding support for fanotify would make the current v4 linux design less abstract.) Not everything can be ported to it, and thus it would have to be some kind of hybrid system, only some part of which would use fanotify only when available.

This is the kind of thing that v5 will likely enable: the possibility of plugging in custom backends a lot more easily, to use them for subsets of workloads, and to build up and above existing ones without rewriting everything. In the meantime (v5 is tentatively slated for the end of the year), you'll need to do what you want yourself.

[thesw4rm]

this library translates IN_ATTRIB to Op::CHMOD

Yes, agreed. I think I did not correctly communicate what I was trying to say. So would it be possible to change it to Op:METADATA in v5? Wouldn't it be possible right now for Op:CHMOD to be triggered but no permission changes are actually in the file, rather it is an extended attribute that was set?

fanotify has severe limitations

Definitely. I was only using it as an example. So I guess I can show my suggestion as an adaptation of the current backend in v5 (that works for all OS).

What I was trying to convey was would it be possible to implement not necessarily a new backend (although maybe that is the best approach since it is already the direction for v5), but rather an adaptation on the current one. So for (better?) example, you could have an optional dependency on auditd for Linux, and compare inotify triggers with auditd logs. (auditd is available as a kernel module since 4.18 so in the future, although it already has non-root support, we can use it as any user without worrying about permissions).

Similar tools are there for Windows builtin, FreeBSD in later kernels, and OpenBSM for Mac.

[passcod]

Yes, agreed.

Ah, alright.

So would it be possible to change it to Op:METADATA in v5?

There's no need. V5 has a different event classification design so it already does this. Inotify's IN_ATTRIB in v5 is translated to Modify(Metadata(Any)). A more precise metadata event could be Modify(Metadata(Permissions)) or Modify(Metadata(Other("immut")), as examples. That way a consumer that only cares about whether it's a modification can discard all but the top level info, one who wants to differentiate between data and meta changes can as well, and one who wishes to get full information is also well-served.

Wouldn't it be possible right now for Op:CHMOD to be triggered

Indeed.

[passcod]

...could have an optional dependency on auditd for Linux, and compare inotify triggers with auditd logs...

Ahh, totally.

At a logistics level, the way this would go (and could be started right now, if one accepts the inherent instability of v5 being pre-alpha now) is that a new backend would be created as an external project. One of the core features of v5 will be to allow users (library users, but perhaps even end-users later) to plug in their own backends (and other components) into notify. So an auditd or bsm or fanotify backend would start as an external thing, possibly by a third-party, then if wanted and if high-quality enough, could be brought in as a feature-gated built-in, and finally once stable and general-utility enough, could be enabled by default.

So it's entirely possible, and that's how it would happen.

[thesw4rm]

A more precise metadata event ...

OK, perfect, that's what I was talking about. To confirm, are there plans to implement this approach with Modify(Metadata(...))?

At a logistics level ...

Nice! I personally really like the way v5 is designed then, definitely makes contributing and keeping the project up to date with latest technologies much easier. Could I claim the auditd backend for this project in that case? I think it will be really useful for Linux 4.18+ kernels especially, and would like to see how it can be implemented in Rust.

Although I still have one quesiton. With implementing a new backend, does the scope of this project cover writing the C bindings with FFI for the related APIs? I noticed that backend-kqueue and backend-inotify use external crates that provide these bindings in the next branch.

[passcod]

Those events were one of the first things I made for v5: https://github.com/passcod/notify/blob/next/backend/src/event.rs so it's already here.

Note that this interface is not stable, like the rest of v5. Notably I already have plans to remove the time field and add instead an attributes (or some other name) field containing an AnyMap, enabling backends to return arbitrary extended information, and enabling Notify to use (or add to) information in there without breaking the API. (That's done now.)

To clarify, backends (except those in core) would start completely external. That is, not even within this project! There's nothing to claim, just go for it.

Notify itself doesn't mandate or provide anything beyond the backend interface. kqueue and inotify (and the fsevent and windows backends in branches) use existing bindings because they already were there. However writing new C bindings is perfectly in scope for a backend, if that's what it needs.

[thesw4rm]

This looks really good then. To make sure I understand, is the project at a stage where others (like me :)) can also contribute to the new architecture design, or are you mainly working on the prototype right now to enable us to contribute?

Also, to address the second point. That's perfect. I'll start working on a crate for auditd then. I have found really sparse docs on the kernel module for now though, would it be alright to integrate something into the project that runs system commands and writes to a rule file directly, and implement the C bindings once the kernel module has matured a bit?

Lastly, like a noob, I didn't realise that auditd doesn't work without root privileges (unless configured otherwise). Is it cool to have a backend with that restriction for now (or ask users to manually configure auditd to allow the user the program will be run with?).

[passcod]

The core is... not quite there yet. I've figured out what I want to do, and there's enough implemented that one can hook up backends, give it a path, then see the backend resolution happen and events come in (useful for testing it actually works outside the test harness!), but the current "next step" is to figure out how “event processors” work and that's all mostly in my head for now.

So you can try things out, but not really contribute yet. Hopefully soon, though!

---

For those two last points: yes, that's fine. The auditd backend would likely live outside the core project for the foreseeable future or at least until the kernel module (and the backend with it) stabilises enough, but that's alright.