[Security Manager Replacement] Strengthen OS core security via systemd configuration

[kumargu]

Please describe the end goal of this project

The OS core's security in absence of security manager can be strengthened by using a stronger systemd unit configuration. We can imagine this as sandboxing via systemd protecting itself from vulnerability in core or untrusted code (plugins). However, this is not a complete replacement of security manager, a part of it like controlling access to egress network, controlling access to specific file locations can be sought as a replacement. Some of the newly introduced configs will add more security than what is being offered by security manager.

Supporting References

#1687

Issues

#16634

Related component

Other

[kumargu]

A list of systemd unit configs which could be useful to restrict access to sys resources, lock down network access, restrict system call etc. This is a fairly exhaustive list but not a complete list -- such as access to socket ip addresses are not exhaustive. Similarly, some of them could be also removed depending on the issues seen during actual integration.

configs to be amended to exiting config

# Prevent modifications to the control group filesystem
ProtectControlGroups=true

# Prevent loading or reading kernel modules
ProtectKernelModules=true

# Prevent altering kernel tunables (sysctl parameters)
ProtectKernelTunables=true

# Restrict access to the filesystem:
# 'strict' makes /usr and /boot read-only, and hides other directories
ProtectSystem=strict

# Set device access policy to 'closed', allowing access only to specific devices
DevicePolicy=closed

# Make /proc invisible to the service, enhancing isolation
ProtectProc=invisible

# Make /usr, /boot, and /etc read-only (less restrictive than 'strict')
ProtectSystem=full

# Prevent changes to control groups (redundant with earlier setting, can be removed)
ProtectControlGroups=yes

# Prevent changing the execution domain
LockPersonality=yes


# System call filtering
# System call filterings which restricts which system calls a process can make
# @ means allowed 
# ~ means not allowed
SystemCallFilter=@system-service
SystemCallFilter=~@reboot
SystemCallFilter=~@swap

SystemCallErrorNumber=EPERM

# Capability restrictions
# Remove the ability to block system suspends
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND

# Remove the ability to establish leases on files
CapabilityBoundingSet=~CAP_LEASE

# Remove the ability to use system resource accounting
CapabilityBoundingSet=~CAP_SYS_PACCT

# Remove the ability to configure TTY devices
CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG

# Remov below capabilities:
# - CAP_SYS_ADMIN: Various system administration operations
# - CAP_SYS_PTRACE: Ability to trace processes
# - CAP_NET_ADMIN: Various network-related operations
CapabilityBoundingSet=~CAP_SYS_ADMIN ~CAP_SYS_PTRACE ~CAP_NET_ADMIN


# Address family restrictions
RestrictAddressFamilies=~AF_INET ~AF_INET6 ~AF_NETLINK ~AF_PACKET ~AF_UNIX

 
# Filesystem Access
 
ReadWritePaths=/var/log/opensearch /var/lib/opensearch 

RestrictNamespaces=true

NoNewPrivileges=true

# Memory and execution protection
MemoryDenyWriteExecute=true           # Prevent creating writable executable memory mappings
SystemCallArchitectures=native        # Allow only native system calls
KeyringMode=private                   # Service does not share key material with other services
LockPersonality=true                  # Prevent changing ABI personality
RestrictSUIDSGID=true                 # Prevent creating SUID/SGID files
RestrictRealtime=true                 # Prevent acquiring realtime scheduling
ProtectHostname=true                  # Prevent changes to system hostname
ProtectKernelLogs=true                # Prevent reading/writing kernel logs
ProtectClock=true                     # Prevent tampering with the system clock

# Socket restrictions
SocketBindAllow=tcp:9200
SocketBindAllow=tcp:9300                          
SocketBindDeny=any                    # Deny all other socket bindings

[kumargu]

// todo

explore below configs

# Optional directives
IPAddressDeny=                       # Deny access to specific IP addresses
PrivateNetwork=true                  # Use a private network namespace

[rmuir]

nice to see this work here, especially sandboxing things such as filesystem with systemd fills a big gap that security manager was doing, and IMO does it in a much better way.

I'd be curious to see change in reported exposure level from systemd-analyze security opensearch.service with your improvements.

[kumargu]

thanks @rmuir. I will post the results from systemd-analyze security opensearch.service.

[kumargu]

cc @andrross

[andrross]

Thanks @kumargu, I think this approach is super promising.

It does raise some questions around testing and maintaining a properly secured systemd config. We'll need to evaluate our integration testing and release pipeline to ensure we have proper coverage of this, as the evolving code base may sometimes (rarely I hope) require tweaks to this config.

[rmuir]

You don't need to have all the SystemCallFilter= entries. Most of what you have listed is already excluded via @system-service.

i'd nuke all the CapabilityBoundingSet entries, just replace with CapabilityBoundingSet=. I'd set NoNewPrivileges=true along the same lines of not allowing escalation.

Same goes with your RestrictNamespaces= entries, they are not needed. it is enough to just set RestrictNamespaces=true.

[kumargu]

Ack @rmuir, I'll make changes as suggested by you, all your comments makes sense to me. I am going to try out the actual integration tomorrow and post results.

---

(edit) 3/12/24

Updated the initially proposed configs based on rmuir@ comments. Haven't yet changed the CapabilityBoundingSet since I need to read a little bit of it, rest all suggestions are incorporated.

[kumargu]

@andrross -- thanks for bringing up the testing part of it. I will think more about it. At the moment, I could think of having a test.service which gets spawned in test suite and verifies that it doesn't have access to restricted resources, e.g; no read/ write access to var/log/elasticsearch.

I don't think we will be able to get a full coverage, but we can cover for the most critical ones. And yes, it would be rare we'd be changing the configs.

[kumargu]

Adding the output of systemd-analyze security opensearch.service Before vs After

Before

(represents the systemd config currently shipped with 2.18)

→ Overall exposure level for opensearch.service: 8.8 EXPOSED 🙁

---

After

(systemd configs edited as per comment to add restrictive policies)

→ Overall exposure level for opensearch.service: 4.6 OK 🙂

systemd status

[ec2-user@ip-172-31-0-63 multi-user.target.wants]$ sudo systemctl status opensearch.service
● opensearch.service - OpenSearch
     Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-12-02 07:51:55 UTC; 6min ago
       Docs: https://opensearch.org/
   Main PID: 17776 (java)
      Tasks: 179 (limit: 601012)
     Memory: 1.5G
        CPU: 45.962s
     CGroup: /system.slice/opensearch.service
             └─17776 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow >

Full report

✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ PrivateDevices=                                             Service potentially has access to hardware devices                                                       0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                                                  0.2
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                                                     0.1
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer                                             0.2
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                                                       0.1
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                                                     0.2
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                                                         0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                                                               0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                                                               0.1
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                                                   0.1
✗ CapabilityBoundingSet=~CAP_MKNOD                            Service may create device nodes                                                                          0.1
✗ RestrictSUIDSGID=                                           Service may create SUID/SGID files                                                                       0.2
✗ ProtectHostname=                                            Service may change system host/domainname                                                                0.1
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unrestricted                                  0.2
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                                                       0.3
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                                                      0.2
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                                                     0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                                                       0.1
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                                                      0.3
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                                                             0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                                                  0.1
✗ ProtectSystem=                                              Service has very limited write access to the OS file hierarchy                                           0.1
✓ ProtectProc=                                                Service has restricted access to process tree (/proc hidepid=)
✗ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has raw I/O access                                                                               0.2
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                                                                 0.3
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters                                                 0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups
✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges
✓ PrivateTmp=                                                 Service has no access to other software's temporary files
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                                                             0.2
✗ ProcSubset=                                                 Service has full access to non-process /proc files (/proc subset=)                                       0.1
✗ ProtectHome=                                                Service has full access to home directories                                                              0.2
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                                                               0.1
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                                                       0.1
✗ PrivateNetwork=                                             Service has access to the host's network                                                                 0.5
✗ PrivateUsers=                                               Service has access to other users                                                                        0.2
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                                                     0.1
✓ DeviceAllow=                                                Service has a minimal device ACL
✓ KeyringMode=                                                Service doesn't share key material with other services
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✗ IPAddressDeny=                                              Service does not define an IP address allow list                                                         0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state
✓ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service cannot use acct()
✓ ProtectControlGroups=                                       Service cannot modify the control group file system
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service cannot issue vhangup()
✓ PrivateMounts=                                              Service cannot install system mounts
✓ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service cannot establish wake locks
✓ RestrictNamespaces=~user                                    Service cannot create user namespaces
✓ RestrictNamespaces=~pid                                     Service cannot create process namespaces
✓ RestrictNamespaces=~net                                     Service cannot create network namespaces
✓ RestrictNamespaces=~uts                                     Service cannot create hostname namespaces
✓ RestrictNamespaces=~mnt                                     Service cannot create file system namespaces
✓ CapabilityBoundingSet=~CAP_LEASE                            Service cannot create file leases
✓ RestrictNamespaces=~cgroup                                  Service cannot create cgroup namespaces
✓ RestrictNamespaces=~ipc                                     Service cannot create IPC namespaces
✓ LockPersonality=                                            Service cannot change ABI personality
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)
✓ RestrictAddressFamilies=~AF_(INET|INET6)                    Service cannot allocate Internet sockets
✗ UMask=                                                      Files created by service are world-readable by default                                                   0.1

→ Overall exposure level for opensearch.service: 4.6 OK 🙂

[andrross]

thanks for bringing up the testing part of it. I will think more about it. At the moment, I could think of having a test.service which gets spawned in test suite and verifies that it doesn't have access to restricted resources

@kumargu That's one aspect of the testing. That other side is ensuring that the OpenSearch software doesn't need access to something that the systemd config restricts. I'm not sure where in our testing pipeline we ever launch the OpenSearch process with systemd. An extreme example: you could block port 9200 in the systemd config and I think ./gradlew check would still pass because the process is never launched with systemd as a part of that test suite. We need to make sure that as the software evolves going forward that what the software needs access to stays in sync with what the systemd config allows.

[rmuir]

yes, you need to test systemd service works (using vm or container).

But lack of testing for the systemd service is nothing new with this issue, it is a pre-existing condition as there is already a systemd service being shipped, just with an EXPOSED configuration.

[rmuir]

and the change will come anyway, probably via distros if you dont do it yourself (e.g. https://fedoraproject.org/wiki/Changes/SystemdSecurityHardening). Distros will probably just let end-users be the testers, so you might as well get ahead of it

[reta]

we actually used to have full fledged disto tests (see please [1]), driven by VMs and vagrant, but I am not sure it is in a good shape now, it has not been run for a looong time

[1] #538

[rmuir]

yeah, if you just want to test systemd service unit, you can use container for that (you have to hold it special: happy to help as i do this at dayjob). If all you are shipping is a service file, that's enough. Perversely, you have to give systemd container special privileges otherwise it can't actually implement the very sandboxing you wish to test, so you have to supply some special docker flags. See https://systemd.io/CONTAINER_INTERFACE/#what-you-shouldnt-do for some background.

if you are also shipping packages (such as RPM, DEB, etc), you can still use same approach, to also check that package and its dependencies install/remove/upgrade correctly, too, all within container. So it gets you pretty far, from upstream project perspective I think its the way to go.

VM is a bit extreme: of course you need it if you want to also test full integration with shenanigans of distro kernel, since container cannot do that: but it makes life so unfriendly to developers and CI systems that you end out with tests not being run at all or not being maintained like #538

[rmuir]

Here's example docker-compose.yml to get you started. In practice you might use testcontainer for java integration, so you'd pass the equivalent flags to docker, but this goes healthy and plumbs logs to stdout and illustrates how to do it with amazonlinux 2023, installing the RPM:

---
services:
  # self-contained systemd example: run 'docker-compose up' to see it
  amazonlinux:
    build:
      dockerfile_inline: |
        FROM amazonlinux:2023
        # install systemd
        RUN dnf -y install systemd && dnf clean all
        # in practice, you'd COPY in the RPM you want to test right here
        RUN dnf -y install https://artifacts.opensearch.org/releases/bundle/opensearch/2.18.0/opensearch-2.18.0-linux-x64.rpm && dnf clean all
        # no colors
        ENV SYSTEMD_COLORS=0
        # no escapes
        ENV SYSTEMD_URLIFY=0
        # explicitly specify docker virtualization
        ENV container=docker
        # for debugging systemd issues in container, you want this, but it is very loud!
        # ENV SYSTEMD_LOG_LEVEL=debug
        # plumb journald logs to stdout
        COPY <<EOF /etc/systemd/journald.conf
        [Journal]
        ForwardToConsole=yes
        EOF
        # start systemd as PID 1
        CMD ["/sbin/init"]
        # enable opensearch service
        RUN systemctl enable opensearch
        # shutdown systemd properly
        STOPSIGNAL SIGRTMIN+3
        # disable security plugin, as i don't configure SSL (but could be done with openssl or whatever right here)
        RUN echo "plugins.security.disabled: true" >> /etc/opensearch/opensearch.yml
    # provide /dev/console for journal logs to go to stdout
    tty: true
    # capabilities to allow systemd to sandbox
    cap_add:
      # https://systemd.io/CONTAINER_INTERFACE/#what-you-shouldnt-do bullet 1
      - SYS_ADMIN
      # https://systemd.io/CONTAINER_INTERFACE/#what-you-shouldnt-do bullet 2
      - MKNOD
    # evil, but best you can do on docker? podman is better here.
    cgroup: host
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup
    # tmpfs mounts for systemd
    tmpfs:
      - /run
      - /run/lock
    # health check for opensearch
    healthcheck:
      test: ["CMD", "curl", "-f", "http://127.0.0.1:9200/_cluster/health"]

[rmuir]

And here's what the output looks like on docker-compose up:

amazonlinux-1  | systemd 252.23-2.amzn2023 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ -ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
amazonlinux-1  | Detected virtualization docker.
amazonlinux-1  | Detected architecture x86-64.
amazonlinux-1  | 
amazonlinux-1  | Welcome to Amazon Linux 2023.6.20241121!
amazonlinux-1  | 
amazonlinux-1  | Failed to open libbpf, cgroup BPF features disabled: Operation not supported
amazonlinux-1  | Queued start job for default target graphical.target.
amazonlinux-1  | [  OK  ] Created slice system-getty.slice - Slice /system/getty.
amazonlinux-1  | [  OK  ] Created slice system-modprobe.slice - Slice /system/modprobe.
amazonlinux-1  | [  OK  ] Created slice user.slice - User and Session Slice.
amazonlinux-1  | [  OK  ] Started systemd-ask-password-consol…quests to Console Directory Watch.
amazonlinux-1  | [  OK  ] Started systemd-ask-password-wall.p… Requests to Wall Directory Watch.
amazonlinux-1  | [  OK  ] Reached target paths.target - Path Units.
amazonlinux-1  | [  OK  ] Reached target remote-fs.target - Remote File Systems.
amazonlinux-1  | [  OK  ] Reached target slices.target - Slice Units.
amazonlinux-1  | [  OK  ] Reached target swap.target - Swaps.
amazonlinux-1  | [  OK  ] Listening on systemd-initctl.socket… initctl Compatibility Named Pipe.
amazonlinux-1  | [  OK  ] Listening on systemd-journald-dev-l…ocket - Journal Socket (/dev/log).
amazonlinux-1  | [  OK  ] Listening on systemd-journald.socket - Journal Socket.
amazonlinux-1  | [  OK  ] Listening on systemd-userdbd.socket - User Database Manager Socket.
amazonlinux-1  |          Mounting dev-hugepages.mount - Huge Pages File System...
amazonlinux-1  | tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
amazonlinux-1  |          Mounting tmp.mount - Temporary Directory /tmp...
amazonlinux-1  |          Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
amazonlinux-1  |          Starting systemd-journald.service - Journal Service...
amazonlinux-1  |          Starting systemd-network-generator.… units from Kernel command line...
amazonlinux-1  |          Starting systemd-sysusers.service - Create System Users...
amazonlinux-1  | [  OK  ] Mounted dev-hugepages.mount - Huge Pages File System.
amazonlinux-1  | [  OK  ] Mounted tmp.mount - Temporary Directory /tmp.
amazonlinux-1  | [  OK  ] Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
amazonlinux-1  | [  OK  ] Reached target local-fs.target - Local File Systems.
amazonlinux-1  |          Starting ldconfig.service - Rebuild Dynamic Linker Cache...
amazonlinux-1  | [  OK  ] Finished systemd-network-generator.…rk units from Kernel command line.
amazonlinux-1  | [  OK  ] Reached target network-pre.target - Preparation for Network.
amazonlinux-1  | [  OK  ] Finished ldconfig.service - Rebuild Dynamic Linker Cache.
amazonlinux-1  | [  OK  ] Finished systemd-sysusers.service - Create System Users.
amazonlinux-1  | [  OK  ] Started systemd-journald.service - Journal Service.
amazonlinux-1  |          Starting systemd-journal-flush.serv…h Journal to Persistent Storage...
amazonlinux-1  | [ 3917.820316] systemd[1]: Starting systemd-journal-flush.service - Flush Journal to Persistent Storage...
amazonlinux-1  | [  OK  ] Finished systemd-journal-flush.serv…ush Journal to Persistent Storage.
amazonlinux-1  | [ 3917.878195] systemd[1]: Finished systemd-journal-flush.service - Flush Journal to Persistent Storage.
amazonlinux-1  |          Starting systemd-tmpfiles-setup.ser… Volatile Files and Directories...
amazonlinux-1  | [ 3917.903470] systemd[1]: Starting systemd-tmpfiles-setup.service - Create Volatile Files and Directories...
amazonlinux-1  | [ 3917.915369] systemd-tmpfiles[25]: /usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
amazonlinux-1  | [  OK  ] Finished systemd-tmpfiles-setup.ser…te Volatile Files and Directories.
amazonlinux-1  | [ 3917.922326] systemd[1]: Finished systemd-tmpfiles-setup.service - Create Volatile Files and Directories.
amazonlinux-1  |          Starting systemd-journal-catalog-up…rvice - Rebuild Journal Catalog...
amazonlinux-1  | [ 3917.924580] systemd[1]: Starting systemd-journal-catalog-update.service - Rebuild Journal Catalog...
amazonlinux-1  |          Starting systemd-resolved.service - Network Name Resolution...
amazonlinux-1  | [ 3917.928541] systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
amazonlinux-1  |          Starting systemd-update-utmp.servic…rd System Boot/Shutdown in UTMP...
amazonlinux-1  | [ 3917.930464] systemd[1]: Starting systemd-update-utmp.service - Record System Boot/Shutdown in UTMP...
amazonlinux-1  | [ 3917.950979] systemd[1]: Finished systemd-update-utmp.service - Record System Boot/Shutdown in UTMP.
amazonlinux-1  | [  OK  ] Finished systemd-update-utmp.servic…cord System Boot/Shutdown in UTMP.
amazonlinux-1  | [  OK  ] Finished systemd-journal-catalog-up…service - Rebuild Journal Catalog.
amazonlinux-1  | [ 3917.962861] systemd[1]: Finished systemd-journal-catalog-update.service - Rebuild Journal Catalog.
amazonlinux-1  |          Starting systemd-update-done.service - Update is Completed...
amazonlinux-1  | [ 3917.984708] systemd[1]: Starting systemd-update-done.service - Update is Completed...
amazonlinux-1  | [  OK  ] Finished systemd-update-done.service - Update is Completed.
amazonlinux-1  | [ 3917.992563] systemd[1]: Finished systemd-update-done.service - Update is Completed.
amazonlinux-1  | [ 3918.013839] systemd-resolved[28]: Positive Trust Anchors:
amazonlinux-1  | [ 3918.014029] systemd-resolved[28]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
amazonlinux-1  | [ 3918.014122] systemd-resolved[28]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
amazonlinux-1  | [ 3918.057989] systemd-resolved[28]: Using system hostname '1ccf8eb8209e'.
amazonlinux-1  | [  OK  ] Started systemd-resolved.service - Network Name Resolution.
amazonlinux-1  | [  OK  ] Reached target network.target - Network.
amazonlinux-1  | [  OK  ] Reached target nss-lookup.target - Host and Network Name Lookups.
amazonlinux-1  | [  OK  ] Reached target sysinit.target - System Initialization.
amazonlinux-1  | [ 3918.059036] systemd[1]: Started systemd-resolved.service - Network Name Resolution.
amazonlinux-1  | [  OK  ] Started systemd-tmpfiles-clean.time… Cleanup of Temporary Directories.
amazonlinux-1  | [ 3918.059138] systemd[1]: Reached target network.target - Network.
amazonlinux-1  | [  OK  ] Reached target timers.target - Timer Units.
amazonlinux-1  | [ 3918.059219] systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
amazonlinux-1  | [ 3918.059287] systemd[1]: Reached target sysinit.target - System Initialization.
amazonlinux-1  | [ 3918.059352] systemd[1]: fstrim.timer - Discard unused blocks once a week was skipped because of an unmet condition check (ConditionVirtualization=!container).
amazonlinux-1  | [ 3918.059396] systemd[1]: Started systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories.
amazonlinux-1  | [  OK  ] Listening on dbus.socket - D-Bus System Message Bus Socket.
amazonlinux-1  | [ 3918.059435] systemd[1]: Reached target timers.target - Timer Units.
amazonlinux-1  | [  OK  ] Reached target sockets.target - Socket Units.
amazonlinux-1  | [ 3918.059480] systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket.
amazonlinux-1  | [ 3918.059542] systemd[1]: Reached target sockets.target - Socket Units.
amazonlinux-1  | [  OK  ] Reached target basic.target - Basic System.
amazonlinux-1  | [ 3918.059629] systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of an unmet condition check (ConditionPathExists=/var/lib/rpm/.rebuilddb).
amazonlinux-1  | [ 3918.059717] systemd[1]: Reached target basic.target - Basic System.
amazonlinux-1  |          Starting opensearch.service - OpenSearch...
amazonlinux-1  | [ 3918.090128] systemd[1]: Starting opensearch.service - OpenSearch...
amazonlinux-1  |          Starting systemd-logind.service - User Login Management...
amazonlinux-1  | [ 3918.093908] systemd[1]: Starting systemd-logind.service - User Login Management...
amazonlinux-1  |          Starting systemd-user-sessions.service - Permit User Sessions...
amazonlinux-1  | [ 3918.095444] systemd[1]: Starting systemd-user-sessions.service - Permit User Sessions...
amazonlinux-1  |          Starting dbus-broker.service - D-Bus System Message Bus...
amazonlinux-1  | [ 3918.105242] systemd[1]: Starting dbus-broker.service - D-Bus System Message Bus...
amazonlinux-1  | [  OK  ] Finished systemd-user-sessions.service - Permit User Sessions.
amazonlinux-1  | [ 3918.109489] systemd[1]: Finished systemd-user-sessions.service - Permit User Sessions.
amazonlinux-1  | [ 3918.114346] systemd[1]: Started console-getty.service - Console Getty.
amazonlinux-1  | [  OK  ] Started console-getty.service - Console Getty.
amazonlinux-1  | [ 3918.114778] systemd[1]: [email protected] - Getty on tty1 was skipped because of an unmet condition check (ConditionPathExists=/dev/tty0).
amazonlinux-1  | [  OK  ] Reached target getty.target - Login Prompts.
amazonlinux-1  | [ 3918.118393] systemd[1]: Reached target getty.target - Login Prompts.
amazonlinux-1  | [  OK  ] Started dbus-broker.service - D-Bus System Message Bus.
amazonlinux-1  | [ 3918.133416] systemd[1]: Started dbus-broker.service - D-Bus System Message Bus.
amazonlinux-1  | [ 3918.146793] dbus-broker-lau[34]: Ready
amazonlinux-1  | [ 3918.162058] systemd-logind[32]: New seat seat0.
amazonlinux-1  | [ 3918.163172] systemd[1]: Started systemd-logind.service - User Login Management.
amazonlinux-1  | [  OK  ] Started systemd-logind.service - User Login Management.
amazonlinux-1  | [ 3920.033244] systemd-entrypoint[31]: WARNING: Using incubator modules: jdk.incubator.vector
amazonlinux-1  | [ 3920.114166] systemd-entrypoint[31]: WARNING: A terminally deprecated method in java.lang.System has been called
amazonlinux-1  | [ 3920.114471] systemd-entrypoint[31]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
amazonlinux-1  | [ 3920.114635] systemd-entrypoint[31]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
amazonlinux-1  | [ 3920.114775] systemd-entrypoint[31]: WARNING: System::setSecurityManager will be removed in a future release
amazonlinux-1  | [ 3921.025461] systemd-entrypoint[31]: Dec 03, 2024 4:13:30 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
amazonlinux-1  | [ 3921.025731] systemd-entrypoint[31]: WARNING: COMPAT locale provider will be removed in a future release
amazonlinux-1  | [ 3921.698362] systemd-entrypoint[31]: WARNING: A terminally deprecated method in java.lang.System has been called
amazonlinux-1  | [ 3921.698619] systemd-entrypoint[31]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
amazonlinux-1  | [ 3921.698840] systemd-entrypoint[31]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
amazonlinux-1  | [ 3921.699116] systemd-entrypoint[31]: WARNING: System::setSecurityManager will be removed in a future release
amazonlinux-1  | 
amazonlinux-1  | Amazon Linux 2023.6.20241121
amazonlinux-1  | Kernel 6.12.1-arch1-1 on an x86_64 (-)
amazonlinux-1  | 
amazonlinux-1  | 1ccf8eb8209e login: [ 3939.110104] systemd[1]: Started opensearch.service - OpenSearch.
amazonlinux-1  | [ 3939.110324] systemd[1]: Reached target multi-user.target - Multi-User System.
amazonlinux-1  | [ 3939.111288] systemd[1]: Reached target graphical.target - Graphical Interface.
amazonlinux-1  | [ 3939.150804] systemd[1]: Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...
amazonlinux-1  | [ 3939.175186] systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
amazonlinux-1  | [ 3939.176988] systemd[1]: Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.
amazonlinux-1  | [ 3939.179789] systemd[1]: Startup finished in 21.555s.

[kumargu]

Thanks for the pointers. I am going to try it the full integration today.

[rmuir]

Yeah, it helps to test it. you can already see small problems with the systemd unit. For example it logs with tag systemd-entrypoint which is too vague. It is just the default because it is the name of the process.

I would set SyslogIdentifier=opensearch

[kumargu]

I would set SyslogIdentifier=opensearch

agreed. will address it.

[kumargu]

sorry for the delay, I will come back to this in a week.

[dblock]

[Catch All Triage - 1, 2, 3]

[kumargu]

Being worked on as part of #17107

[peterzhuamazon]

After offline discussion with @RajatGupta02 for a while.

We think this approach only affect RPM and DEB packages, and will not touch tar, docker, and windows zip, means users of those distributions will not get the update.

Since RPM/DEB will be used on many different linux distributions, some of the changes here might have a wider effect than initially anticipated.

Especially in opensearch-build we use another customized sysconfig + systemd file here for the bundle: https://github.com/opensearch-project/opensearch-build/tree/main/scripts/pkg/service_templates/opensearch

I am wondering if the Java Agent approach in #16731 is a more unified change to the core which would update all the distributions instead of using a systemd file for the lockups.

Thanks.

[peterzhuamazon]

After discussing with @andrross offline it seems that this systemd changes is also coupled with java agent.

With this change, if ever synced to the build repo, would:

1. Take affect on rpm/deb user
2. tar, docker, zip user will not get 1
3. docker is already a containerized mechanism

Will still wait for the discussion on this, let me know the updates @RajatGupta02.

Thanks.

[kumargu]

@peterzhuamazon we have already had discussion and conclusion that we will need systemd for deprecating security manager. It's a stronger alternative to JAVA agent and one can not replace other. Please refer: #17181. "We are not deprecating security manager without ````systemd```" -- also this is not new, its an enhancement.

We have also discussed that this will not not work for windows, docker (for docker it's anyways not required). There are alternatives like systemd in windows and whoever so needs its can open a contribution to introduce it for windows. However, I don't understand why it won't work for tar, zip.

Especially in opensearch-build we use another customized sysconfig + systemd file here for the bundle: https://github.com/opensearch-project/opensearch-build/tree/main/scripts/pkg/service_templates/opensearch

Is this even the right mechanism, how could we dump something which is part of core and use a custom systemd config?

[peterzhuamazon]

Hi @kumargu ,

I do not think a user of tar/zip would setup systemd themselves just to run the cluster.
It is not like rpm or deb by default will have systemd in place.
Some linux system also does not have systemd but rather older sysvinit which will also not take this change.

The mechanism to have a different copy in opensearch-build repo has a long historical reason dated back in ODFE and for the time being we will not remove it as it is used for bundle only, not for min.
Let me know when this change is in so we can populate the same on build repo for 3.0.0.

Thanks,
Peter

[kumargu]

I do not think a user of tar/zip would setup systemd themselves just to run the cluster.

Agreed, but that should not stop us to make enhancements. My mental model is to suggest in our docs that running opensearch with systemd is most secure way to run Opensearch.

Let me know when this change is in so we can populate the same on build repo for 3.0.0.

We are targeting 3.0 alpha

[peterzhuamazon]

Hi @kumargu @RajatGupta02,

I think with this change is bound to go in, it is also better to update this doc as well regarding users who want to use customized systemd for their tar distribution:
https://opensearch.org/docs/latest/install-and-configure/install-opensearch/tar/#run-opensearch-as-a-service-with-systemd

Thanks.

[peterzhuamazon]

We are targeting 3.0 alpha

Got it, let us sync up @RajatGupta02 once all your PRs are merged in core, and I will see if I can pull the file directly from core.

Thanks.

[andrross]

Hi @kumargu @RajatGupta02,

I think with this change is bound to go in, it is also better to update this doc as well regarding users who want to use customized systemd for their tar distribution: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/tar/#run-opensearch-as-a-service-with-systemd

Thanks.

Here's the file in question: https://github.com/opensearch-project/documentation-website/blob/0cb3f7c962297b944bc3db4e23b0e908ebe91402/_install-and-configure/install-opensearch/tar.md?plain=1#L535

We should update that to link to the systemd under source control. We need to get to a point where there is a single systemd configuration file that is actively tested with each release, and then is used in all relevant distributions and documentation references.

[peterzhuamazon]

After some offline syncup with @RajatGupta02 @kumargu , here is the steps for 3.0.0:

1. @RajatGupta02 update Add systemd configurations to strengthen OS core security #17107 (review) with changes from build repo copy
2. @RajatGupta02 update Add systemd integ tests to run with docker #17308 by removing docker exec to be easily onboard to opensearch-build integTest
3. @peterzhuamazon add entries to 3.0.0 test manifest to onboard systemd test as part of the integTest, as well as update the integTest scripts to include the ./gradlew qa:systemd-test:integTest --tests org.opensearch.systemdinteg.SystemdIT --info command.
4. @peterzhuamazon update assemble script to use the core systemd files on building process for deb/rpm
5. @RajatGupta02 @peterzhuamazon test the above process end to end to ensure build -> assemble -> test is complete
6. @peterzhuamazon remove copies of systemd files from build repo.
7. @RajatGupta02 update documentations on using systemd with tar: https://github.com/opensearch-project/documentation-website/blob/main/_install-and-configure/install-opensearch/tar.md#run-opensearch-as-a-service-with-systemd

cc: @getsaurabh02

Thanks.