systemd by example

Part 5: Timers

Series overview

This article is part of the series systemd by example. The following articles are available.

Introduction

This is the fifth article in a series where I try to understand systemd by creating small containerized examples.

Often we might want to run a program periodically, for example to back up some files, check for updates, or to clean up stale data. Traditionally, this was done by adding an entry to /etc/crontab. systemd timers are more powerful and flexible, and in my opinion a lot simpler.

Using a timer

A systemd timer defines one or multiple points in time when another unit should be activated. By default, a timer unit triggers a service unit with the same name.

Here is a basic example.

Follow along on the systemd playground Example 1: A basic timer

[Timer]
OnCalendar=minutely

[Service]
ExecStart=echo "Hello world!"

This tells systemd to activate hello-world.service once every minute.

After the system starts we have to activate the timer with

systemctl start hello-world.timer

We can now check the status of the timer.

systemctl status hello-world.timer

This shows

● hello-world.timer
     Loaded: loaded (/lib/systemd/system/hello-world.timer; static; vendor preset: enabled)
     Active: active (waiting) since Mon 2024-04-01 12:21:30 UTC; 6s ago
    Trigger: Mon 2024-04-01 12:22:00 UTC; 23s left
   Triggers: ● hello-world.service

Apr 01 12:21:30 8338861cc694 systemd[1]: Started hello-world.timer.

So the service should be activated next at 12:22:00. However, nothing happens at that time. It takes 15 more seconds until we see

Apr 01 12:22:15 8338861cc694 systemd[1]: Started hello-world.service.
Apr 01 12:22:15 8338861cc694 echo[42]: Hello world!
Apr 01 12:22:15 8338861cc694 systemd[1]: hello-world.service: Succeeded.

in the journal. Waiting for a few more minutes, we see that the service is indeed activated about every minute, but not on the minute mark. In my example, the service was also activated on 12:23:28, 12:24:01, 12:25:28, and 12:26:28. We’ll investigate why this happens later in the post.

Installing timers

In the previous example, we started the timer manually. In reality, we would probably use an install section, see Part 4: Installing Units. For timers, there is a dedicated target defined to use as a target for timers, namely timers.target. So usually, we would add the following section to our timer unit.

[Install]
WantedBy=timers.target

We then have to enable this unit with systemctl enable. On the playground, this can be achieved by checking the checkmark next to the unit file, then the timer is already enabled when the system starts.

For all future examples, we will use an install section and enable the timer by default.

Calendar events

In the example, we used the OnCalendar= directive, which expects a calendar event which tells systemd when to trigger the timer. In the example, we used minutely, which tells systemd to trigger the timer every minute. Similarly, we can specify hourly to trigger at every full hour, daily to trigger at midnight each day, or weekly to trigger at midnight each Monday. And there’s also monthly, yearly, quarterly, and semiannually.

For more flexibility, we can specify calendar events with a timestamp pattern. Take for example the timestamp Mon 2024-04-22 12:00:00 (the weekday is optional). We can pass this to OnCalendar=, which will cause systemd to trigger the timer exactly once, on April 22 2024 at noon. A timer that triggers exactly once is probably an exception. Most likely, we want it to trigger repeatedly. For this, we can replace any component of the timestamp with a variable indicator. A * means that it will trigger for any possible value for this component. We can specify a list of values separated by commas, or a range of values by specifying the start and end values, connected with ...

Here is an example. You can hover over each component to see how it maps to the trigger.

If we don’t specify a weekday in the timestamp pattern, it matches any weekday. If we don’t specify the date part, it defaults to *-*-*, so it matches every day. And if we don’t specify the time part, it defaults to 00:00:00, meaning midnight.

There is even more flexibility that I won’t go over here; for example, we can start counting from the end of a month, specify a range, but with a step size bigger than one, or specify a timestamp. For the full list of rules, check the man page of system.time.

Very useful when trying to come up with the correct timestamp patterns is the command systemd-analyze calendar. It takes a pattern and writes its normalized form and the next time that the timer is triggered.

For example

systemd-analyze calendar 'Thu *-02-29'

outputs

  Original form: Thu *-02-29                
Normalized form: Thu *-02-29 00:00:00       
    Next elapse: Thu 2052-02-29 00:00:00 CET
       (in UTC): Wed 2052-02-28 23:00:00 UTC
       From now: 27 years 11 months left    

showing that the next February 29 that falls on a Thursday will be in 28 years from now.

Accurate timers

Now let’s get back to why the timer didn’t fire at the minute mark. By default, systemd timers are only accurate up to a minute, meaning that they can fire any time between when they are scheduled to fire and up to a minute later. This is done for power efficiency; by providing slack, the CPU doesn’t have to wake up unnecessarily. The amount of slack can be controlled with the AccuracySec= directive.

AccuracySec= takes a time span, for example 3s, or 1 hour 4 minutes. The smallest unit is us, or microseconds. See the man page of systemd.time for the full specification. If we want our timer to trigger exactly at the time specified by OnCalendar=, we can set AccuracySec= to a small value (the documentation says to set it to 1us, but setting it to 0 also seems to work).

Let’s add the AccuracySec= directive to hello-world.timer.

Follow along on the systemd playground Example 2: Accurate timers

[Timer]
OnCalendar=minutely
AccuracySec=1us

[Install]
WantedBy=timers.target

After starting the system and waiting for a few minutes, we see that the timer now triggers exactly at the minute mark.

Apr 01 12:42:00 c9bab365dc72 systemd[1]: Started hello-world.service.
Apr 01 12:42:00 c9bab365dc72 echo[17]: Hello world!
Apr 01 12:42:00 c9bab365dc72 systemd[1]: hello-world.service: Succeeded.
Apr 01 12:43:00 c9bab365dc72 systemd[1]: Started hello-world.service.
Apr 01 12:43:00 c9bab365dc72 echo[18]: Hello world!
Apr 01 12:43:00 c9bab365dc72 systemd[1]: hello-world.service: Succeeded.

Multiple timers

If there are multiple timers, they each have their own trigger interval, specified by their calendar event and AccuracySec= specification. If those intervals overlap, then systemd will try to fire them at the same time, so that the CPU doesn’t need to wake up unnecessarily.

We can see this in an example.

Follow along on the systemd playground Example 3: Multiple timers

[Timer]
OnCalendar=minutely

[Install]
WantedBy=timers.target

[Timer]
OnCalendar=*:*:30

[Install]
WantedBy=timers.target

As before, hello-world.timer triggers every minute. If we remove the uncertainty of AccuracySec=, it would trigger always at the full minute mark. Similarly, goodbye-world.timer also triggers every minute, but at the half minute mark.

And indeed, when I run this example, I get the following output.

Apr 01 12:46:12 4fc089c8f901 systemd: Startup finished in 72ms.
Apr 01 12:47:28 4fc089c8f901 systemd[1]: Started goodbye-world.service.
Apr 01 12:47:28 4fc089c8f901 systemd[1]: Started hello-world.service.
Apr 01 12:47:28 4fc089c8f901 echo[17]: Goodbye world!
Apr 01 12:47:28 4fc089c8f901 echo[18]: Hello world!
Apr 01 12:47:28 4fc089c8f901 systemd[1]: goodbye-world.service: Succeeded.
Apr 01 12:47:28 4fc089c8f901 systemd[1]: hello-world.service: Succeeded.

The system started at 12:46:12 and activated both timers. goodbye-world.timer was scheduled at 12:46:30, but can trigger up to a minute later (since we didn’t specify AccuracySec= explicitly, the default of one minute is assumed). Similarly, hello-world.timer was scheduled at 12:47:00, but can trigger up to a minute later. The actual triggering happened at 12:47:28, simultaneously for both timers.

Randomized delay

As we have seen, AccuracySec= allows systemd to try to trigger as many timers at the same time as possible, to avoid unnecessary CPU wake ups. But sometimes, we want the exact opposite and spread the triggering as far as possible. Assume for example that we have several heavy duty tasks that we want to run every day. This can be easily achieved by creating timers with OnCalendar=daily. But this will result in all of those heavy duty tasks to be triggered at the exact same time shortly after midnight.

To avoid this, we could manually change the OnCalendar= definitions of each timer, and configure them so that one triggers at midnight, one at 1am, and so forth. But this becomes hard to maintain pretty quickly. Instead, systemd provides the RandomizedDelaySec= directive. For every iteration of the timer, systemd will add a random delay to the calendar event; the delay is somewhere between zero seconds and RandomizedDelaySec=.

Note that systemd also will still apply AccuracySec=, which can cause multiple timers to trigger together, as we saw above. Since we usually use RandomizedDelaySec= to spread out triggers, we would set AccuracySec= to 1us to avoid this grouping.

Let’s see this in action.

Follow along on the systemd playground Example 4: Randomized delay

[Timer]
OnCalendar=minutely
RandomizedDelaySec=15 seconds
AccuracySec=1us

[Install]
WantedBy=timers.target

[Timer]
OnCalendar=minutely
RandomizedDelaySec=15 seconds
AccuracySec=1us

[Install]
WantedBy=timers.target

The first thing to note is that systemctl status shows the randomized delay that was added to the calendar event.

systemctl status hello-world.timer

shows

● hello-world.timer
     Loaded: loaded (/lib/systemd/system/hello-world.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Mon 2024-04-01 12:55:07 UTC; 15s ago
    Trigger: Mon 2024-04-01 12:56:13 UTC; 49s left
   Triggers: ● hello-world.service

Warning: journal has been rotated since unit was started, output may be incomplete.

The calendar event specified a trigger at the minute mark, but a delay of 13 seconds was added. And indeed, the timer triggers at this exact second. If we execute systemctl status again after the timer triggered, we see a different random delay added. We also see from the logs that the timers indeed fire at different times.

Apr 01 12:56:03 4e81dc5e0ef6 systemd[1]: Started goodbye-world.service.
Apr 01 12:56:03 4e81dc5e0ef6 echo[25]: Goodbye world!
Apr 01 12:56:03 4e81dc5e0ef6 systemd[1]: goodbye-world.service: Succeeded.
Apr 01 12:56:13 4e81dc5e0ef6 systemd[1]: Started hello-world.service.
Apr 01 12:56:13 4e81dc5e0ef6 echo[26]: Hello world!
Apr 01 12:56:13 4e81dc5e0ef6 systemd[1]: hello-world.service: Succeeded.
Apr 01 12:57:02 4e81dc5e0ef6 systemd[1]: Started hello-world.service.
Apr 01 12:57:02 4e81dc5e0ef6 echo[35]: Hello world!
Apr 01 12:57:02 4e81dc5e0ef6 systemd[1]: hello-world.service: Succeeded.

You might wonder why systemd shows the delay added by RandomizedDelaySec= in systemctl status, but it doesn’t do the same for AccuracySec=. If you look back at the first example, systemctl status showed that the timer would trigger at the exact minute mark, but it actually fired several seconds later. The reason (I assume) is that doing so would reduce systemd’s flexibility. Suppose for example that there is a timer that runs daily with AccuracySec=12 hours; this means that it triggers some time between midnight and noon. (Such timers do exist. On my system this is true for example for the logrotate.timer and the man-db.timer, even though I think that a RandomizedDelaySec= would have been a better choice.) Let’s now activate a new timer that triggers at 11am. If the other timer hasn’t triggered yet, then systemd can group them together. But if systemd had already applied the AccuracySec= delay in the beginning, it would have lost this flexibility. The question is whether the flexibility of avoiding some CPU cycles in certain cases is really worth it.

Other time specifications

The OnCalendar= directive is not the only way to specify a trigger time. There is for example OnActiveSec= to trigger the timer a certain time after the timer was activated, or OnBootSec= to activate it a certain time after the system was booted up. There is also OnTimezoneChange=, which takes a boolean argument. If set to true, the timer will trigger when the system time zone is modified. You can specify multiple of those directives in the same timer unit, which causes the timer to trigger whenever any of the conditions is met. Check the man page of systemd.timer for the full list.

Conclusion

Timers are probably the units I most personally create most often (along with their service counterparts). They are really useful for recurring tasks, and I find them fairly intuitive, especially when compared to cron jobs. The default value of AccuracySec= can be a bit of a gotcha, but I think that most of the time, pinpoint accuracy is not necessary; and if it is, we can set AccuracySec= to a small value.

—Written by Sebastian Jambor. Follow me on Mastodon @crepels@mastodon.social for updates on new blog posts.