systemd by example
Part 3: Defining services
Series overview
This article is part of the series systemd by example. The following articles are available.
Part 3: Defining services (this article)
Introduction
This is the third article in a series where I try to understand systemd by creating small containerized examples. In Part 1, we created a minimal systemd setup in a container. In Part 2 we took a close look at systemd’s dependency management. Now we are turning to systemd services. There is a lot to say about services, so I will split the information across the next few posts, starting in this post with the basics of defining a service unit.
To begin with, we should try to understand what a service even is. When I first heard about systemd services, I thought about things like NGINX or PostgreSQL. But while reliably operating services like these is certainly a use case of systemd services, it is a very special use case. systemd services are used for a lot more than this. You could almost say that whenever systemd actually does something, it does it through a service (from now on, when I write service I mean a systemd service). Here are just three example usages of custom services that I use on my machine: running a lock screen before the system goes to sleep so that it is protected when it wakes up again; setting some kernel variables on bootup to turn off annoying LEDs; running a small script every minute to monitor the health of systemd-by-example.
It is hard to concisely define what the essence of a service is, since it has so many uses and so many configuration options. One way (although very simplified and inaccurate) to think about services is to view them as encapsulation of executables. When we define a service, we define the program that should be run, together with its arguments, similar to how we would execute it on the command line.
In this post, we will take a look at how to define basic services. There are currently 195 directives that are valid in the [Service] section of a unit, and an additional 82 that are valid in the [Unit] section, but I think it is neither possible nor useful to try to get through as many as possible. Instead, I will follow the theme of the previous posts and try to keep things simple and minimalistic, focussing on just four directives and leaving more coverage for later posts.
What really makes services powerful is the ability to define when they should be activated. This can be for example when the system reaches a certain state (for example after bootup, or before going to sleep); periodically or at defined times like a cron job; when a file or directory changes; or when there is traffic on a network port. We will cover this in a later post.
All examples in this post are available on the systemd playground, where you can run them directly from the browser, and modify them to try out different things. You can also run them locally on your machine; see Part 1 for details on how to set this up.
Specifying a command to execute
The main directive of a service unit (and the only one that is required) is ExecStart=. With this directive, we specify the command that should be executed when the unit is activated. The syntax to specify the command is similar to the command line of a shell, but there are some differences and restrictions. Let’s start with an example.
[Service]
ExecStart=echo "Hello world!"
Let’s see this in action. Start the system on the playground (or add the service to your container if you want to run this locally) and then activate the service with
systemctl start hello-world
(this is a short-hand form for systemctl start hello-world.service; systemd appends the .service automatically). Checking the journal with
journalctl
we see
Mar 07 11:57:30 3f036c932218 systemd[1]: Started hello-world.service.
Mar 07 11:57:30 3f036c932218 echo[26]: Hello world!
Mar 07 11:57:30 3f036c932218 systemd[1]: hello-world.service: Succeeded.
(To only see the logs of one service we can also use journalctl --unit=hello-world; or use systemctl status hello-world to see the units status and its recent logs.)
The syntax of the command in the ExecStart= directive is intentionally similar to how we would define it on the command line of a shell. In the example above, the binary echo is executed with the single argument Hello world!, just as it would be in a shell. But the how is different. First, there is the way that systemd finds the echo binary. The shell will use the PATH environment variable which contains a list of directories; it will go through this list one by one and check if the echo binary is in any of them, and then call the execve system call with this binary. systemd instead uses a hard-coded list of directories that it searches: /bin, /usr/bin, /usr/local/bin and the corresponding sbin variants. If the binary we want to execute is not in any of those directories, we have to specify an absolute path. Second, while a command that’s executed from the shell will inherit stdout and stderr from the shell process, a systemd service will have its stdout and stderr connected to journald.
Another similarity with a shell command line is the handling of environment variables. systemd will create a custom environment for the process, and we can reference the environment variables with the $VAR syntax. We will see this in more detail later when we take a closer look at the ways to define the environment.
And this is where the similarities end. Other useful shell features like > and >> to redirect output to a file, | to pipe output to another command or command substitution via $(...) are not supported directly. Fortunately, it is easy to work around this by running our desired commands through a shell. For example, it is sometimes useful to change kernel variables after the system starts. We can achieve this with a service like this:
[Service]
ExecStart=bash -c 'echo var > /proc/sys/filename'
Defining the user
By default, the ExecStart= command of a service is executed by the root user. It might make sense to execute the service with a less privileged user. We can do this with the User= directive. The following simple service shows the effect of the directive.
[Service]
ExecStart=whoami
User=nobody
After starting this service with
systemctl start custom-user
the journal shows
Mar 07 11:58:13 af16b82e49d1 systemd[1]: Started custom-user.service.
Mar 07 11:58:13 af16b82e49d1 whoami[26]: nobody
Mar 07 11:58:13 af16b82e49d1 systemd[1]: custom-user.service: Succeeded.
so the service was indeed executed by the user nobody.
It is even possible to create a custom user with the DynamicUser= directive. If this directive is set to true, then systemd creates a new user whenever the service is started. It will also create private mounts for /tmp and /var/tmp which are only accessible to this user. This requires superuser capabilities which are disabled in the systemd playground for security reasons. But you can try this out locally on your machine by adding the --cap-add=CAP_SYS_ADMIN option to the podman run command. You can read more about this directive and why it’s useful on Dynamic Users with systemd on Lennart Poettering’s blog.
Defining the environment
When a new process is created, the default behavior is that it inherits the environment from its parent process. Especially in a shell, this is often used by first enriching the environment of the shell process via the export command, and then calling the desired process which then has access to the exported variables. systemd does not follow this behavior. When it starts a service, the executed process doesn’t inherit the environment of systemd; instead, a new environment is created from scratch. Let’s take a look at the environment that is created by default.
[Service]
ExecStart=env
If env is executed without any parameters, it will simply print its environment to stdout; and as always, when we execute the service, this will end up in the journal. We see that four variables are set in the environment.
Mar 07 11:58:52 9996b9f3c60a env[26]: LANG=C.UTF-8
Mar 07 11:58:52 9996b9f3c60a env[26]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Mar 07 11:58:52 9996b9f3c60a env[26]: INVOCATION_ID=6d62d1a3cb27413bb71ba23148ffa3cd
Mar 07 11:58:52 9996b9f3c60a env[26]: JOURNAL_STREAM=9:292732633
LANG is the system locale, and PATH is the hard coded path that systemd uses internally. INVOCATION_ID is a randomized number that is unique for each service invocation, which allows us to identify this particular run. The id is also passed to the journal, which allows us to query for log lines belonging to one service invocation, using
journalctl _SYSTEMD_INVOCATION_ID=6d62d1a3cb27413bb71ba23148ffa3cd
Finally, JOURNAL_STREAM contains the device and inode numbers of the file descriptor of the journal connection. This enables the service to log using the native journal protocol. For example, it can specify the priority of a log message (like debug, info, warning, error, etc.).
Those four variables are always set, but some directives result in the creation of more environment variables. For example, when the User= directive is used, USER and LOGNAME are defined, and potentially HOME and SHELL if they are defined for that user. The man page for systemd.exec contains the full list.
We can add to the environment with the Environment= directive. We can specify multiple variables in one directive, but we can also have multiple Environment= variables. If the values contain spaces, they need to be quoted. Here is a (rather senseless) example.
[Service]
Environment=FIRST_VAR=first SECOND_VAR=second
Environment=THIRD_VAR="third var"
ExecStart=env
Executing this service produces the logs
Mar 07 11:59:43 da8699a2e854 env[26]: LANG=C.UTF-8
Mar 07 11:59:43 da8699a2e854 env[26]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Mar 07 11:59:43 da8699a2e854 env[26]: INVOCATION_ID=1421745766d547d2901819247ed1d41d
Mar 07 11:59:43 da8699a2e854 env[26]: JOURNAL_STREAM=9:292735872
Mar 07 11:59:43 da8699a2e854 env[26]: FIRST_VAR=first
Mar 07 11:59:43 da8699a2e854 env[26]: SECOND_VAR=second
Mar 07 11:59:43 da8699a2e854 env[26]: THIRD_VAR=third var
The type of a service
With the Type= directive, we can specify the type of the service unit. It can take seven different values. For a long time, it wasn’t really clear to me what the type is used for and which value to use in a given situation. So we are going to spend the rest of this post taking a close look at all possible types and construct examples to highlight their differences.
The main purpose of the type is to determine when a service is considered active. This refers to the states of a unit from Part 2 and is important for dependency management. The type is also used to determine the main process of a service. (A service can consist of multiple processes by forking from the ExecStart= command.) The main process has a special role among all processes. For example, it defines the lifetime of a service. When the main process terminates, the service is considered terminated and all other processes are killed. The main process can also be sent a special signal when a service is restarted, and the PID of the main service can be used when reloading a service.
Before investigating the different available types, we need to take a brief look at how systemd actually executes a command (it’s the same for the shell). It does this in two steps. First, it calls fork(), which creates a copy of the systemd process. In this new process, it then uses the execve() system call to replace systemd by the program that we want to execute. Both steps can fail for different reasons. For example, fork() will fail if no further processes can be created, for instance by hitting the limit specified in /proc/sys/kernel/threads-max; it will also fail if there is not enough memory to allocate the necessary kernel structures. On the other hand, execve() will fail for example if the supplied binary to execute does not exist, or if the executing user does not have permission to execute the binary.
Type=simple
The default type of a service is simple. A service of this type is considered active as soon as systemd has successfully called fork() to create a new process, even if the subsequent execve() call fails. Here are two services to highlight this.
[Service]
Type=simple
ExecStart=/non/existent/binary
[Unit]
Requires=initial.service
After=initial.service
[Service]
ExecStart=echo "My dependencies were activated successfully!"
(We don’t need to specify the Type=simple in initial.service since it’s the default anyway; we only do it for clarity.)
From Part 2 we know that when we try to start follow-up.service, systemd will first start initial.service, and only start follow-up.service once initial.service was successfully activated.
Executing
systemctl start follow-up
and looking at the journal, we see
Mar 07 12:00:14 abe3085d9992 systemd[1]: Started initial.service.
Mar 07 12:00:14 abe3085d9992 systemd[1]: Started follow-up.service.
Mar 07 12:00:14 abe3085d9992 echo[27]: My dependencies were activated successfully!
Mar 07 12:00:14 abe3085d9992 systemd[26]: initial.service: Failed to execute command: No such file or directory
Mar 07 12:00:14 abe3085d9992 systemd[26]: initial.service: Failed at step EXEC spawning /non/existent/binary: No such file or directory
Mar 07 12:00:14 abe3085d9992 systemd[1]: initial.service: Main process exited, code=exited, status=203/EXEC
Mar 07 12:00:14 abe3085d9992 systemd[1]: initial.service: Failed with result 'exit-code'.
Mar 07 12:00:14 abe3085d9992 systemd[1]: follow-up.service: Succeeded.
This means that follow-up.service was indeed activated, even though initial.service failed. But it only failed in the execve() step; in the short time frame between fork() and execve() it was considered active, which was enough for systemd to activate the follow-up service.
Type=exec
We can change this behavior with Type=exec; in this mode, the service is only considered active once the execve() call was successful. Let’s change the type of initial.service above to exec
[Service]
Type=exec
ExecStart=/non/existent/binary
If we are now trying to start follow-up.service with
systemctl start follow-up
it fails with output
A dependency job for follow-up.service failed. See 'journalctl -xe' for details.
and the logs show
Mar 07 12:00:48 fa20954dbd68 systemd[1]: Starting initial.service...
Mar 07 12:00:48 fa20954dbd68 systemd[26]: initial.service: Failed to execute command: No such file or directory
Mar 07 12:00:48 fa20954dbd68 systemd[26]: initial.service: Failed at step EXEC spawning /non/existent/binary: No such file or directory
Mar 07 12:00:48 fa20954dbd68 systemd[1]: initial.service: Main process exited, code=exited, status=203/EXEC
Mar 07 12:00:48 fa20954dbd68 systemd[1]: initial.service: Failed with result 'exit-code'.
Mar 07 12:00:48 fa20954dbd68 systemd[1]: Failed to start initial.service.
Mar 07 12:00:48 fa20954dbd68 systemd[1]: Dependency failed for follow-up.service.
Mar 07 12:00:48 fa20954dbd68 systemd[1]: follow-up.service: Job follow-up.service/start failed with result 'dependency'.
So at first sight, Type=exec seems to be the better alternative. Nevertheless, it is not used by a single service on my machine. If units that depend on the service require the command to be started successfully, they most likely also require that it is initialized successfully (where the definition of what initialized means depends on the service). This is not possible with Type=exec, we would instead use one of the types below.
Type=oneshot
A common pattern for a service is to execute a short-lived command, for example for a one-time setup (like setting up the console keyboard layout with keyboard-setup.service when the system boots), for a periodic task (like man-db.service for a daily regeneration of the man page index caches), or to be executed when the system reaches a certain state (like systemd-halt.service which shuts down the system when it reaches halt.target). For these kinds of services it makes sense to only consider them as successfully activated once the command terminates with a zero status code. This is exactly what Type=oneshot does. (We used this type extensively in Part 2 when we investigated the different states of a systemd unit.)
To highlight the difference between Type=exec and Type=oneshot, replace initial.service above first with
[Service]
Type=exec
ExecStart=false
[Service]
Type=oneshot
ExecStart=false
false always returns status code 1, which is interpreted as a failing command).
If we run both examples, we’ll see that in the first case follow-up.service will be activated, whereas in the second case it won’t. When initial.service is started, systemd can successfully fork and exec (that is start) the false binary; at this point, a service of Type=exec is already considered active. But a Type=oneshot service waits for the command to terminate and evaluates the exit code, which in this case is considered not successful, so the service is marked as failed.
By default, a Type=oneshot service is never marked as active; it is in state activating while the command is running, and afterwards immediately transitions to state deactivating. Any units that have an ordering dependency on the service will be activated when this transition happens. We can change this behavior with the RemainAfterExit= directive. If this is set to true, then the unit is marked as active once the command terminates, and stays in that state until the unit is explicitly deactivated.
Type=oneshot is special in that it is the only unit that does not have a requirement on exactly one ExecStart= directive. It allows multiple ExecStart= directives; in this case, the commands are executed sequentially one after the other until one of them fails or until all are executed. It also allows the specification of no ExecStart= directives at all. In this case, a RemainAfterExit=true directive is mandatory, as well as an ExecStop= directive (which is similar to ExecStart=, but it is executed when the unit is deactivated).
Type=forking
The next three types are intended for long-running processes, like the webserver or database mentioned in the introduction. We start with Type=forking.
Traditionally (before systemd came along), a daemon process was kicked off by forking a short-lived init process. For example, to start an NGINX server, we can execute the nginx binary. The started process then forks itself; the child process lives on to serve http traffic, while the parent process simply exits. If we executed nginx from the command line, we would see that the prompt almost immediately returns and the server is started in the background.
This type of daemon is supported in systemd with Type=forking. The service is finished activating when the ExecStart= command exits with status code 0 (to work reliably, the init process should only exit once the child process is ready to do its job). So far, this is the same behavior as Type=oneshot. But while a oneshot service immediately transitions to an inactive state (unless RemainAfterExit=true), for a forking service systemd now listens to the forked off process: the service is considered active until the forked process terminates, in which case it is marked inactive or failed, depending on the exit code.
Consider the following example.
[Service]
Type=forking
ExecStart=bash -c 'sleep 2; nc -k -l 18 &'
In the terminology above, the bash process is the init process. The sleep 2 is some artificial delay, representing the setup work that’s necessary in a real daemon process. After it sleeps for two seconds, the bash process then forks off a nc process that listens (with the -l option) on port 18 and writes everything it receives to stdout. With the -k option we tell nc to keep listening after the first connection is terminated. (Unfortunately, there are no long-options for nc, so the invocation is rather cryptic.) Note: If you are running these examples locally, you will need to add nc to the container (by adding apt-get install netcat-openbsd to the Dockerfile).
We also change follow-up.service to make use of our nc server.
[Unit]
Requires=initial.service
After=initial.service
[Service]
Type=oneshot
ExecStart=bash -c 'echo "Hello world" | nc -N localhost 18'
This connects to localhost on port 18 and sends the string Hello world to it. Afterwards it closes the connection (with the -N option).
Let’s start follow-up.service as before and look at the logs.
Mar 07 12:01:42 7b97d67845b0 systemd[1]: Starting initial.service...
Mar 07 12:01:44 7b97d67845b0 systemd[1]: Started initial.service.
Mar 07 12:01:44 7b97d67845b0 systemd[1]: Starting follow-up.service...
Mar 07 12:01:44 7b97d67845b0 bash[28]: Hello world
Mar 07 12:01:44 7b97d67845b0 systemd[1]: follow-up.service: Succeeded.
Mar 07 12:01:44 7b97d67845b0 systemd[1]: Finished follow-up.service.
We see that first initial.service is run, which is marked as active two seconds later when the main process returns. Only then is follow-up.service run. We can also see the string Hello world in the logs, logged by bash[28]. This log line belongs to initial.service. We can confirm this by running
systemctl status initial
which outputs
● initial.service
Loaded: loaded (/lib/systemd/system/initial.service; static; vendor preset: enabled)
Active: active (running) since Mon 2022-03-07 12:01:44 UTC; 21s ago
Process: 26 ExecStart=/usr/bin/bash -c sleep 2; nc -k -l 18 & (code=exited, status=0/SUCCESS)
Main PID: 28 (nc)
CGroup: /machine.slice/libpod-7b97d67845b0a0cb1d1ae94ab9768e1056f8917d3d2a62781dab825d077e5760.scope/system.slice/initial.service
└─28 nc -k -l 18
Mar 07 12:01:42 7b97d67845b0 systemd[1]: Starting initial.service...
Mar 07 12:01:44 7b97d67845b0 systemd[1]: Started initial.service.
Mar 07 12:01:44 7b97d67845b0 bash[28]: Hello world
We also see that the unit is currently considered active, and that systemd differentiates between the process that started the unit (here with PID 26) and the main PID (the nc process with PID 28).
In this case, systemd was able to reliably guess the PID of the main process. When there are multiple processes that are forked off this is not always possible. In those cases, it is expected that the original process writes the PID of the main process to a file, which is passed to systemd with the PIDFile= directive.
Type=notify
I mentioned above that starting a daemon via forking is the traditional way. systemd advertises a new way of starting a daemon. Instead of executing an init process which forks off the actual daemon process, the main process will be the daemon. But we now need a way to know when the daemon is to be considered active. For a traditional daemon, we assume that it is ready when the init process returns, but this is not possible if we don’t fork. Instead, systemd provides a library function sd_notify which should be called when the daemon is ready. With Type=notify we tell systemd that it should expect this library function to be called.
Here is the forking example from above rewritten as a new-style daemon.
[Service]
Type=notify
ExecStart=bash -c 'sleep 2; systemd-notify --ready; nc -k -l 18'
NotifyAccess=all
The sleep 2 simulates again the delay of some initialization. We then call systemd-notify, which is a thin wrapper around the sd_notify library function. And finally, we listen on port 18 for incoming calls. This is backwards to what we said above. We should actually first listen for incoming calls and then call sd_notify to signal that we are ready, but this is not possible in a shell script without forking. In this example we also have to specify the NotifyAccess= directive. By default, systemd only accepts calls to sd_notify by the process that’s specified in ExecStart=. In our case, sd_notify is called by a separate process, namely systemd-notify, so we have to let systemd know to allow this.
If we execute follow-up.service in this example (with the same service definition as in the forking example) we get a similar output. But looking at the status of initial.service we see
● initial.service
Loaded: loaded (/lib/systemd/system/initial.service; static; vendor preset: enabled)
Active: active (running) since Mon 2022-03-07 12:46:03 UTC; 2min 23s ago
Main PID: 26 (bash)
CGroup: /machine.slice/libpod-e6d0ff58b7a6c5bfa840b617aec6285796e895ce9f9607e4cdcaa9432e4ae141.scope/system.slice/initial.service
├─26 /usr/bin/bash -c sleep 2; systemd-notify --ready; nc -k -l 18
└─29 nc -k -l 18
Mar 07 12:46:01 e6d0ff58b7a6 systemd[1]: Starting initial.service...
Mar 07 12:46:03 e6d0ff58b7a6 systemd[1]: Started initial.service.
Mar 07 12:46:03 e6d0ff58b7a6 bash[29]: Hello world
So in this case, the bash process is still the main process.
Type=dbus
If a service offers communication via D-Bus, it makes sense to consider it as active as soon as this communication channel is active. This is possible with Type=dbus. systemd will consider such a service as active when it has required the bus name that is specified with the BusName= directive. Similar to calling sd_notify, the bus name should only be acquired once the initialization is done.
[Service]
Type=dbus
BusName=dev.jambor.Test
ExecStart=bash -c 'sleep 2; dbus-test-tool echo --system --name=dev.jambor.Test'
As before, we are adding an artificial delay to simulate some initialization work. We are then using dbus-test-tool from the dbus-tests package (if you run these examples locally, you will need to install it in your Dockerfile). In echo mode, it will always answer with an empty reply. The --system option tells it to connect to the system bus, and the --name= option configures the bus name to acquire; this is the same as in the BusName= directive.
We also have to adapt the follow-up.service to communicate using D-Bus instead of TCP. We are using the dbus-send tool to do this.
[Unit]
Requires=initial.service
After=initial.service
[Service]
Type=oneshot
ExecStart=dbus-send --system --dest=dev.jambor.Test --print-reply /some/path dev.jambor.Test string:'hello world'
Again, we are specifying the --system option so that it uses the system bus, and the --dest= option to specify the bus name. The --print-reply option lets us see the reply that we get from our service, and the remainder of the command line is the actual request.
We need two additional units to make this example work: a dbus.socket unit and a dbus.service unit. They are part of a regular systemd setup, so in the real world we wouldn’t need to add them. But we stripped everything for the minimal systemd setup, so we have to add them now.
[Socket]
ListenStream=/var/run/dbus/system_bus_socket
[Service]
ExecStartPre=bash -c 'echo \'<busconfig><policy user="root"><allow own="dev.jambor.Test"/><allow send_destination="dev.jambor.Test" send_interface="dev.jambor"/></policy></busconfig>\' > /etc/dbus-1/system.d/dev.jambor.Test.conf'
ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
In dbus.socket we specify the socket that the D-Bus daemon will listen under. This is similar to the systemd-journald.socket we defined in Part 1. In dbus.service we start the actual daemon. The ExecStartPre= directive is a small hack to create a configuration file /etc/dbus-1/system.d/dev.jambor.Test.conf on the systemd playground before dbus-daemon is executed. It allows us to actually listen on this bus name and to communicate with it, otherwise the D-Bus daemon would prevent our initial.service from starting. If you are running these examples locally on your machine, you can simply copy this file to your container. Note that we don’t define any dependencies on these units.
Let’s start follow-up.service as before and take a look at the logs.
Mar 07 12:58:53 ab99de899c0c systemd[1]: Listening on dbus.socket.
Mar 07 12:58:53 ab99de899c0c systemd[1]: Starting initial.service...
Mar 07 12:58:55 ab99de899c0c systemd[1]: Starting dbus.service...
Mar 07 12:58:55 ab99de899c0c systemd[1]: Started dbus.service.
Mar 07 12:58:55 ab99de899c0c systemd[1]: Started initial.service.
Mar 07 12:58:55 ab99de899c0c systemd[1]: Starting follow-up.service...
Mar 07 12:58:55 ab99de899c0c dbus-send[31]: method return time=1646657935.819972 sender=:1.1 -> destination=:1.2 serial=3 reply_serial=2
Mar 07 12:58:55 ab99de899c0c systemd[1]: follow-up.service: Succeeded.
Mar 07 12:58:55 ab99de899c0c systemd[1]: Finished follow-up.service.
We can see several interesting things.
First, before initial.service is started, systemd pulls in dbus.socket. This is because every service of Type=dbus automatically has dependencies of type Requires= and After= on dbus.socket. Since dbus.socket was not activated before, systemd starts it automatically.
Next, after initial.service enters the activating state (the Starting initial.service log line), we see a two-second delay (caused by sleep 2), and then dbus.service is started. How can this happen? There is no (visible) dependency on dbus.service. The answer is that when we execute dbus-test-tool in initial.service, it tries to connect to the D-Bus daemon on the socket specified in dbus.socket. systemd recognizes this and automatically starts dbus.service! This is called socket activation, and I find it quite magical. You can read more about it on systemd for Developers I on Lennart Poettering’s blog, and I also plan to cover it in a future blog post.
Once the D-Bus daemon is up, our service can acquire the bus name and is then marked as active (visible by the Started initial.service log line). Now follow-up.service starts and executes dbus-send, which sends a request to the bus name that dbus-test-tool listens to. It sends back an empty reply which we can see in the logs.
Type=idle
This last type is a hack. Similar to Type=simple, the service is considered active as soon as the process in ExecStart= has been forked off. But contrary to Type=simple, this forking only happens once there are no other units that are in state activating. However, systemd doesn’t wait forever, after 5 seconds it forks the process anyway, regardless of whether there are other activating units or not. The use case for this is to avoid interleaving of messages from different services on the console. This is used for example by the getty service that shows a login prompt on the console; to avoid that any output is written by some other service after the login prompt, it uses Type=idle.
We can create an example to highlight the problem and how to “fix” it with an idle service. To do this, we define a service that writes two log lines to the console (using the StandardOutput= directive), but it waits two seconds between the two writes. Since the type is oneshot, the unit is in an activating state until both log lines are written.
[Service]
Type=oneshot
ExecStart=bash -c 'echo "line 1"; sleep 2; echo "line 2"'
StandardOutput=journal+console
Our initial service will also write to the console, but only one second after startup. For now, we use Type=simple.
[Service]
Type=simple
ExecStart=bash -c 'sleep 1; echo "I am the initial service"'
StandardOutput=journal+console
Finally, we change our follow-up service to require both other services.
[Unit]
Requires=initial.service log-writer.service
After=initial.service log-writer.service
[Service]
ExecStart=echo "My dependencies were activated successfully!"
There is no ordering dependency between log-writer.service and initial.service, so they will be started in parallel: executing
systemctl start follow-up
shows something like the following on the console
[3217428.633830] bash[27]: line 1
[3217429.632812] bash[26]: I am the initial service
[3217430.638656] bash[27]: line 2
The log lines are interleaved. This is what the idle type intends to fix. And indeed, when we change the type of initial.service to idle, we get the following output.
[3217462.254129] bash[27]: line 1
[3217464.256919] bash[27]: line 2
[3217465.266314] bash[26]: I am the initial service
We can see from the timestamp that the command of initial.service was started after the log-writer.service left the activating state. But as mentioned above, systemd will wait at most 5 seconds before running initial.service anyway. We can see this as well when we increase the sleep time between the two log lines in log-writer.service from two seconds to seven seconds. Now the log lines are interleaved again.
When to use which type
After seeing the seven available service types, how do you decide which one to use for your particular service? To decide this, consider the following two questions.
- Will any other units depend on the service?
- Is the main process of the service different from the one started by the
ExecStart=directive?
If the answer to both questions is no, then Type=simple is likely the best choice (and the one that’s recommended by the systemd man pages), because it is, well, simple, and it has the least overhead.
If the services main process is not the one specified in ExecStart=, then you’ll need to tell systemd about this, most likely by using Type=forking.
And if other units depend on the service, you will most likely want the service to be fully operational before it is considered active. If the service is a short-lived process (like a one-time setup of kernel variables), choose Type=oneshot; if the service is a traditional daemon process (like described in man 7 daemon), choose Type=forking; if the service provides its functionality via D-Bus, choose Type=dbus; and if the service calls a process that’s designed to use systemd, in particular, it uses sd_notify, choose Type=notify.
Don’t use Type=idle unless the service uses the console and wants to avoid interference of output from other services, like a console login manager.
And lastly, Type=exec. I can’t find a good use case for this type. On my system, not a single service uses it. It seems like you should either use the simpler Type=simple if you don’t expect dependent units, or one of the more sophisticated types. (If you know of a use case, let me know!)
Conclusion
We have seen the basics to define a service: the command that it encapsulates, the user that executes this command, and the environment that it runs in. We have also taken a close look at the Type= directive, which defines how a service transitions between different states.
Services are only useful when they are executed (duh!). So far, we have either executed them manually using systemctl, or defined them as explicit dependencies adding Requires= or Wants= directives in other unit files. But in the majority of cases we would do this differently. We will take a look at different ways to activate services in the next post.
—Written by Sebastian Jambor. Follow me on Mastodon @crepels@mastodon.social for updates on new blog posts.