Linux systemd

General

• systemd the service manager.
• journald, networkd, sd-bus, systemd.slice (I think all mandatory).
• systemd-resolved.service, systemd-boot, systemd-homed.service (all optional).
• consoled, systemd-logind.service, systemd-hostnamed.service, systemd-machined.service, systemd-tmpfiles, systemd-stub, systemd-measure, systemd-cryptenroll, systemd-pcrphase, systemd-creds, many more (not sure if all optional).

pidof systemd			# are you using systemd ?

man systemd
man systemd.index
man udev

systemctl -t help		# see types of units
systemctl list-unit-files	# see all units

pstree					# see running processes
systemctl list-units	# show in-memory units, with descriptions
systemctl status		# show all defined units

systemctl status UNITNAME --full --lines 1000
systemctl cat UNITNAME	# see source for a unit
systemctl edit UNITNAME	# create an override/addition for a unit

# check syntax of a unit:
systemd-analyze verify /etc/systemd/system/UNIT.service

# doesn't actually list all targets:
systemctl list-units --type target
# see for example:
systemctl cat rescue.target

systemctl status PID  # find out which unit a process belongs to

systemd-cgtop         # see resource use of services

Checking what your cron job really does can be kind of a mess, but all systemd timer events are carefully logged in systemd journal.

Systemd timers are systemd services with all their capabilities for controlling their resource management, IO, CPU, scheduling, etc.

There can be dependencies on other services.

Services can be started and triggered by different events like user, boot, hardware state changes or for example 5 mins after some hardware plugged in.

Easily enable/disable the whole thing with "systemctl enable/disable" and kill all the job's children with "systemctl start/stop".

Timer events can be scheduled based on finish times of executions some delays can be set between executions.

Communication with other programs is also notable; sometimes it's needed for some other programs to know timers and the state of their tasks.

systemd-run --scope -p IPAddressDeny=any APPNAME

systemd-run --scope -p MemoryMax=500M APPNAME

Want logs since boot?
sudo journalctl -b
The first boot is -b 0, the previous boot is -b -1.

Want end of log for last boot ?
sudo journalctl -b -1 -e

Boot history ?
sudo journalctl --list-boots


Want logs for smartd?
sudo journalctl -u smartd

Want warnings and above?
sudo journalctl -p warning

Want logs since yesterday?
sudo journalctl -S yesterday

Want logs on April 22 between 10 and 11?
sudo journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00"

Want logs with microsecond timestamps?
sudo journalctl -o short-precise

Or in JSON for parsing?
sudo journalctl -o json

Want tail -f?
sudo journalctl -f

Want only kernel logs?
sudo journalctl -k or journalctl --dmesg

Want to filter the output right away?
sudo journalctl --grep=some_regex

You booted a live environment, but want to look at the installed OS's journal:
sudo journalctl --root=/path/to/mounted_root or journalctl --directory=/path/to/journal

Look at logs from a container you booted using systemd-nspawn (and friends)?
sudo journalctl --machine=container_id

What time intervals do the boot indices (for -b) correspond to?
sudo journalctl --list-boots

Add a message to the journal?
echo Hello There | systemd-cat

Apparently GNOME runs every application in its own systemd slice (which really does explain why GNOME has a hard requirement for systemd as well as why the "Force Quit" button actually seems to work reliably and clean up all the associated clutter in one fell swoop), so you can fetch this stuff from your user journal with "systemctl --user status" to find the slices (type /APPNAME and hit the enter key to search for it) and "journalctl --user -u SLICENAME.slice" to look through their output that way.

Good

• Does things that init scripts can't do: react to events, do operations in parallel (but init scripts can do this with startpar ?), let user start/stop daemons, set limits to the resource usage of a service, have one service depend on result of another (but init scripts can do this with insserv ?).
  
  
• Pushes control away from the init-project and towards the distro maintainers. It's easier to enable/disable/add/delete services than to rewrite the init scripts.
  
  
• User is less likely to bork the system by making a mistake while adding a new service than while editing init scripts ? More modular.
  
  
• Has "targets", which are finer-grained and more flexible than init's "run-levels".
  
  
• Various parts solve large-scale problems for enterprises. Desktop users may not understand this.
  
  

It solves a whole lot of problems to me.

• It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd.
  
  
• It removes a bunch of horrible hacks in init scripts such as calls to sleep.
  
  
• It removes the need for pid files.
  
  
• It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts.
  
  
• It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to.
  
  
• It captures all the log output of each process.
  
  
• It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again.
  
  
• As a developer it means I don't need to write a slightly different script for every distro.
  
  
• It makes logging a whole lot pleasant.
  
  
• It allows having user services without having to hack around with cron.
  
  
• It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there.
  
  
• It means services run identically when started by hand, without the confusion of a service inheriting my local environment.
  
  
• It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.
  
  

The perspective from which I care about the advantages is that of a programmer writing daemons: running under a daemon manager (such as, but not limited to, systemd) lets me omit a bunch of code around forking off, daemonisation, automatic restarts, switching uid/gid, logging and rotation. The important thing about these pieces of functionality is that they provide no competitive advantage.

As someone who wants to write a daemon, you need them, but there's no room to delight anyone by doing them well (unlike the actual functionality for which your daemon is being written) and there's plenty of room to mess them up and have people yell at you.

> I'm so glad we've moved past trying to do all of this with shell scripts.

This can't be stated strongly enough. I recently had to fix a vendor-provided (not from the particular software in question) init script that spawned a daemon that inherited file descriptors it shouldn't have. There are simply too many ways to just pile on spaghetti that can have strange effects. There were other maintenance tasks that required rewrites of most of these scripts just because critical software changes: going from udevd to eudev.

I cannot stand init scripts.

...

I recently took a look at some software a customer wanted installed on all their Linux servers. It had an undocumented upgrade function that fetched and installed software without any checksums and without verifying HTTPS certificates, the upgrade function was not disabled even when explicitly using the somewhat-documented flag to disallow upgrades. It dropped lines in /etc/rc.d/rc.local, it spawned itself during postinst scripts outside of the init system, and trying to upgrade the package resulted in the software deleting itself. The postinst script also looks for a line in /etc/rc.d/rc.local that suggests that they previously modified /etc/ssh/sshd_config on install ...

The big difference [between systemd and others] is that systemd (as well as runit, s6, etc) stay attached to the process, whereas the BSD systems (and OpenRC, traditional Linux init scripts) expect the program to "daemonize".

Aside from whatever problems systemd may or may not have, I feel this model is vastly superior in pretty much every single way. It simplifies almost everything, especially for application authors, but also for the init implementation and system as a whole.

Systemd From an Arch init maintainer perspective:

I was the primary maintainer for Arch's init scripts for a while and I can share a couple of thoughts.

Arch's init-scripts were incredibly stupid. In their first phase, there was a static set of steps that would be performed on every boot. There was almost no way to adjust the behaviour here. In their second phase, the configured daemons were started in order, which only meant that init scripts were called one after another.

In the early 2000s, that seemed like a good idea and has worked for a while. But with more-complex setups, the shortcomings of that system become apparent.

• With hardware becoming more dynamic and asynchronous initialization of drivers in the kernel, it was impossible to say when a certain piece of hardware would be available. For a long time, this was solved by first triggering uevents, then waiting for udev to "settle". This often took a very long time and still gave no guarantee that all required hardware was available. Working around this in shell code would be very complex, slow and error-prone: You'd have to retry all kinds of operations in a loop until they succeed. Solution: An system that can perform actions based on events - this is one of the major features of systemd.
  
  
• Init-scripts had no dependency-handling for daemons. In times where only a few services depended on D-Bus and nothing else, that was easy to handle. Nowadays, we have daemons with far more-complex dependencies, which would make configuration in the old init-scripts-style way hard for every user. Handling dependencies is a complex topic and you don't want to deal with it in shell code. Systemd has it built-in (and with socket-activation, a much better mechanism to deal with dependencies).
  
  
• Complex tasks in shell scripts require launching external helper program A LOT. This makes things very slow. Systemd handles most of those tasks with built-in fast C code, or via the right libraries. It won't call many external programs to perform its tasks.
  
  
• The whole startup process was serialized. Also very slow. Systemd can parallelize it and does so quite well.
  
  
• No indication of whether a certain daemon was already started. Each init script had to implement some sort of PID file handling or similar. Most init scripts didn't. Systemd has a 100% reliable solution for this based on Linux cgroups.
  
  
• Race conditions between daemons started via udev rules, D-Bus activation and manual configuration. It could happen that a daemon was started multiple times (maybe even simultaneously), which lead to unexpected results (this was a real problem with bluez). Systemd provides a single instance where all daemons are handled. Udev or D-Bus don't start daemons anymore, they tell systemd that they need a specific daemon and systemd takes care of it.
  
  
• Lack of configurability. It was impossible to change the behaviour of init-scripts in a way that would survive system updates. Systemd provides good mechanisms with machine-specific overrides, drop-ins and unit masking.
  
  
• Burden of maintenance: In addition to the aforementioned design problems, init-scripts also had a large number of bugs. Fixing those bugs was always complicated and took time, which we often did not have. Delegating this task to a larger community (in this case, the systemd community) made things much easier for us.
  
  

I realize that many of these problems could be solved with some work, and some were already solved by other SysV-based init systems. There was no system that solved all of these problems and did so in a reliable manner, as systemd does.

So, for me personally, when systemd came along, it solved all the problems I ever had with system initialization. What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically. You can say what you want about Poettering, but he actually realized what the problems with system initialization were and provided a working solution.

I could go on for hours, but this should be a good summary.

Resolved does two things not available with libc resolvers. First, it has by interface domain routing, so you can have e.g. VPN hosts only in certain domain resolved through its nameserver. And second is that it remembers servers which are down so it doesn't retry them all over again. Plus it has caching, plus dnssec validation.

Having two DNS entries put into /etc/resolv.conf only works if you want to have 100% of the DNS configuration done on the DNS server and you trust your DNS server explicitly.

If you want to be able to validate DNS responses, use VPN connections with private IP addresses with DNS names, connect to public access points, or essentially use anything more complicated then a single statically configured Ethernet port for life ... then putting IP addresses in /etc/resolv.conf isn't going to cut it.

You need to have a local resolver service to provide those features in a sane manner.

Networkd allows you to specify DNS servers for individual network devices, set DNS server priority, and use different DNS servers per subnet or FQDN.

Bad

• Added a second system log (the journal) to the system, and it's binary, not text.
  [But for reasons: "Introducing the Journal"]
  
  
• Has its own DNS resolver.
  [But maybe it's a good thing: zbyszek's "systemd-resolved: introduction to split DNS"]
  
  From someone on reddit:
  > Why does systemd have DNS in it ?
  
  Because in certain situations they need to have it very early in the boot process, before other stuff is up and running. In most cases it isn't needed and thus isn't used, but when it is needed it is very useful.
  
  That is the big thing a lot of people don't seem to get: systemd provides a lot of optional components that serve niche use-cases but are turned off by default. In most cases those aren't needed and aren't used, but they are there when someone really needs them.
  
  > Something being turned off by default doesn't
  > make it any easier to understand the code,
  > or understand which DNS specification takes
  > precedence over which other one.
  
  
  
  
• Runs all services in one process ? Or dispatches all from one process ? Not sure.
  "ps axjf | grep `pidof systemd`"
  
  
• Now is trying to control the definition of the home directory ?
  [But there's a good case to be made for it: article1, article2]
  
  
• Also has time-sync ?
  
  
• Has a systemd.offline-updates that is very reminiscent of Windows updating, in that it forces two OS restarts to apply a set of updates.
  
  
• An overly-heavy solution for situations where a very simple init system would suffice (such as a container that contains multiple processes).
  
  
• Is Linux-only, so I guess not available on BSD and others.
  
  
• Is glibc-specific, so not supported on systems that use musl or other C standard libs ? So not on Alpine, Void, Android ? musl thread
  
  

Miscellaneous