Tips & tricks for systemd and journald on NixOS

This document contains a list of tips and tricks for working with systemd, journalctl, and related tools.

SysVinit vs Upstart vs Systemd

The simplest cheatsheet:

SysVinit Upstart Systemd
/etc/init.d/service start start service systemctl start service
/etc/init.d/service stop stop service systemctl stop service
/etc/init.d/service restart restart service systemctl restart service
/etc/init.d/service status status service systemctl status service

Systemd Unit Types

Systemd has the following unit types you might be concerned with:

  • services: A service unit describes how to manage a typically long-running application process. This includes how to start, stop, reload, etc the service, under which circumstances it should be automatically started, timeout periods or events, and the dependency or ordering relative to other systemd units. In NixOS you can create a new systemd service like so:
  systemd.services.myservice = {
    description = "My service is responsible for ...";
    after = [ "multi-user.target" ];
    wantedBy = [ "multi-user.target" ];
    path = [ pkgs.bash ];
    environment = {
      MY_SERVICE_HOME = "/my/path/here";
      MY_SERVICE_MAX_CONNS = toString myVar;
    };
    serviceConfig = {
      User = "myuser";
      ExecStart = path;
      Restart = "always";
    };
  };
  • paths: This type of unit defines a path to be used for path-based activation. For example, service units could be started, restarted, stopped, reloaded, etc when the file a path unit represents encounters a specific state. inotify is used to monitor the path for state changes.
  • slices: Slice units map to Linux Control Groups. This allows resources to be restricted or assigned to processes associated with the slice. The root slice is named -.slice.
  • sockets: A socket unit describes a network or IPC socket, or a FIFO buffer that systemd uses for socket-based activation. Socket units are associated to services to trigger their start.
  • swaps: This unit describes swap space on the system.
  • targets: A target unit is used to provide synchronization points for other units when booting up or changing states. The target of interest to most systemd service definers will likely be multi-user.target.
  • timers: Timer units define a timer managed by systemd. It represents a periodic or event-based activation. A matching unit, typically a service, will be started when the timer or event requirements are met.

Systemd has other types of units but the above list is a good starting point. For more information please consult man systemctl.

The following commands can be used to query information about systemd units:

# List dependencies for a unit
$ systemctl list-dependencies UNITNAME

# List sockets
$ systemctl list-sockets

# List active systemd jobs
$ systemctl list-jobs

# List all units and their respective states
$ systemctl list-unit-files

# List all loaded or active units
$ systemctl list-units

Systemd Services

Most of the time we will be concerned with systemd services. Below are a list of useful commands for working with these:

# Need to have sudo privileges to stop/start/restart services
$ sudo systemctl stop SERVICE
$ sudo systemctl start SERVICE
$ sudo systemctl restart SERVICE

# Query commands anyone can run
$ systemctl status SERVICE
$ systemctl is-active SERVICE
$ systemctl show SERVICE

You can also run systemctl commands remotely like so:

$ systemctl -H hostname status SERVICE

This works for systemctl commands other than just systemd service specific commands.

Log Accessibility By journalctl

For all services that need logs accessed via journalctl you should log to the console from a systemd unit.

For example, Elasticsearch logging configuration can be set as so:

rootLogger: INFO, console
logger:
  action: INFO
  com.amazonaws: WARN
appender:
  console:
    type: console
    layout:
      type: consolePattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Then you will be able to query logs from the elasticsearch service unit by using:

$ journalctl -f -u elasticsearch

Accessing Logs Via journalctl

# tail "follow" all log messages for elasticsearch unit/service
$ journalctl -f -u elasticsearch

# show last 1000 error messages for elasticsearch unit/service (command
# terminates without ^C)
$ journalctl -fen1000 -u elasticsearch

# only show kernel messages in tail "follow" mode:
$ journalctl -k -f

# only show log messages for service BLA since last "boot"
$ journalctl -b -u BLA

# show all error log messages from all log sources since last "boot"
$ journalctl -xab

Many more permutations of options are available on journalctl. Please consult man journalctl for more information.

User Access To journalctl Logs

All users that are in the systemd-journal group should be able to query logs via journalctl. Ensure your SSH user is in this group via groups USERNAME.

NixOS Configuration for journald

The NixOS expression for a node's configuration contains the following settings that are worth tuning on servers with high frequency events being logged.

As of NixOS 16.03, the defaults for services.journald.rateLimitBurst and services.journald.rateLimitInterval are worth evaluating for your needs:

$ sudo nixos-option services.journald.rateLimitBurst
Value:
100

Default:
100

Description:

Configures the rate limiting burst limit (number of messages per
interval) that is applied to all messages generated on the system.
This rate limiting is applied per-service, so that two services
which log do not interfere with each other's limit.

...

And:


$ sudo nixos-option services.journald.rateLimitInterval
Value:
"10s"

Default:
"10s"

Description:

Configures the rate limiting interval that is applied to all
messages generated on the system. This rate limiting is applied
per-service, so that two services which log do not interfere with
each other's limit. The value may be specified in the following
units: s, min, h, ms, us. To turn off any kind of rate limiting,
set either value to 0.

This means on this system journald will rate limit events per service after 100 messages within 10s. For many servers this is low, and you will want to adjust it with values like the following:

  services.journald.rateLimitBurst = 1000;
  services.journald.rateLimitInterval = 1s;

The above will rate limit services to logging 1000 messages per second.

You can also turn off rate limiting in journald with the following:

  services.journald.rateLimitInterval = 0;

Nix if-then-else expressions

Nix if-then-else expressions

A coworker asked the following question:

I understand how to use an if/else, but I don’t understand how to get an if/elseif/else.

Below is my Slack response (slightly edited) to explain the dissonance in the question itself and then provide a path forward for him. I imagine this might be a common misunderstanding for those coming to Nix without prior understanding of expression languages vs languages that use statements.

Nix being an expression language just has expressions (where the result of each part evaluates to a value [eventually]). Most mainstream imperative languages like Perl, Bash, etc. that most people are familiar with use this notion of statements statements rather than expressions. Statements allow these languages to support if/elseif/.../else as an extension.

In Nix and other expression based languages, this is not the case and a limiting factor of expressions is that every expression must evaluate to a value. So you might write:

{
  key = if builtins.pathExists ./path then "woot" else "bummer";
}

In the case the ./path exists it will evaluate to the value "woot" otherwise it evaluates to the value "bummer".

So the result of the top level expression is:

{ key = "woot"; }

# OR

{ key = "bummer"; }

This does not translate to languages that model ifs as statements, for example:

$ declare bla=$(if true; then "bla"; else "foo"; fi)
$ echo "${bla}"

$ declare bla=$(if true; then echo "bla"; else echo "foo"; fi)
$ echo "${bla}"
bla

Note: that side effects are required inside each clause.

Looking at the equivalent if/else statement for the expression example above we have:

if [ -f ./path ]; then
  declare key="woot"
else
  declare key="bummer"
fi

Here you see that Bash uses side effects to do the assignment in each case, but say we had this:

if [ -d ./path ]; then
  declare key="woot"
elseif [ -x ./path ]; then
  echo "executable"
else
  declare key="bummer"
fi

Now we have a case (where the file is not a directory and also executable) that the variable key is not set. This wouldn't happen in an expression based language.

In short if-then-else is the only way we can build an expression to always evaluate to a value where all logical paths are covered without the program needing to know about the underlying data or condition clauses inspected in the if portion.

You can think of if-then-else as a lambda that is defined as:

  ifThenElse = cond: t: f: if cond then t else f

So to solve the problem you take one of two approaches.

  • If your use case is matching strings exactly in each if/ifelse condition, then you can use an attrset with the keys as the values you need to match:
{ envType, defaultCfg }:
let
  cases = { "dev" = devCfg; "test" = testCfg; "prod" = prodCfg; };
  lookup = attrs: key: default:
    if attrs ? key then attrs."${key}" else default;
in lookup environments envType defaultCfg
  • Use nested if-then-else expressions when you cannot just lookup a key in an attrset.

What's the Nix cookbook?

This is a cookbook-style teaching tool to show how to write Nix expressions for the purposes of packaging, system configuration, provisioning, orchestrated deployment, CI jobs, etc.

Here is some introductory material to get you started:

More to come soon!

Nix Basics

Nix Cookbook

This repository is a collection of Nix expressions and snippets that show you how you can get common tasks done in Nix.

The target audience are those that have read the Nix Manual and found it great reference material but need more examples to be more productive with the following toolset:

Prerequisites

I recommend Nix 1.9+ (although most exercises should work in 1.8) and installing nix-repl:

$ nix-env -i nix-repl

Alternatively just start the nix-shell at the root of this repository:

$ nix-shell

Basics

To get started doing warmups the following snippets provide examples and exercises to get acquainted with the different builtin types in the Nix expression language.

Start by using the nix-repl.

Poke around at the examples and let me know (via pull requests or issues) about any problems or suggestions you have for the material.