Systemd: From Zero to Actually Understanding It

You type systemctl restart nginx. It works. You move on.

That's most people's relationship with systemd. It's the thing that starts services. The thing you blame when boot is slow. The thing you copy-paste commands for without really knowing what's happening.

That ends today.

What Systemd Actually Is

Systemd is PID 1. That's it. That's the starting point.

When your kernel finishes loading, it hands control to one process. On every modern Linux distro, that process is systemd. Everything else — your network, your SSH daemon, your Docker containers, your cron jobs — all of it descends from systemd.

Before systemd, PID 1 was SysV init. A shell script. Literally a bash script that ran other bash scripts in sequence. It worked, but it was slow, fragile, and had no way to express dependencies between services.

Systemd replaced that with a unit-based, dependency-aware, parallel init system. And it also replaced or absorbed cron, syslog, network management, hostname management, login sessions, and more.

That's why people have strong opinions about it. It does a lot.

Units: The Core Concept

Everything in systemd is a unit. A unit is a configuration file that describes something systemd should manage.

There are several unit types:

.service — a daemon or process
.timer — a cron replacement (more on this later)
.socket — socket-activated services
.target — grouping of units (like runlevels)
.mount — filesystem mounts
.path — watch a path and react to changes
.slice — cgroup resource management

When you run systemctl restart nginx, you're telling systemd to restart the nginx.service unit.

Let's look at what a real unit file looks like:

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Three sections. Every unit has them.

[Unit] — metadata and dependencies. After=network.target means "don't start until the network target is reached." It doesn't mean "require network." Just ordering.

[Service] — how to actually run the thing. Type=forking means the process forks and the parent exits (old-school daemon behavior). ExecStartPre runs before the main command — here it validates nginx config before trying to start.

[Install] — when should this unit activate? WantedBy=multi-user.target means "enable me when the system reaches multi-user mode" — which is normal operation for a headless server.

Where Unit Files Live

Three locations, and the precedence matters:

/lib/systemd/system/ ← shipped by packages, don't touch
/etc/systemd/system/ ← your overrides and custom units
/run/systemd/system/ ← runtime units, gone on reboot

If a file exists in /etc/systemd/system/ with the same name as one in /lib/systemd/system/, yours wins. That's how you customize package-provided units without losing changes on upgrade.

But there's a better way for partial overrides: drop-ins.

systemctl edit nginx

This creates /etc/systemd/system/nginx.service.d/override.conf and opens your editor. You only write the sections you want to change. Everything else stays from the original.

Path 1: Drop-in override — use this for small targeted changes.

Example: nginx keeps dying and you want it to restart automatically.

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
Restart=always
RestartSec=5s

That's it. Two lines. The rest of the nginx unit stays untouched. On upgrade, the package updates /lib/systemd/system/nginx.service and your override still applies on top.

Example: add an environment variable to a service without touching its unit.

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
Environment="NODE_ENV=production"
Environment="PORT=3000"

Example: increase the open file limit for a service hitting descriptor exhaustion.

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
LimitNOFILE=65536

You can stack multiple drop-in files too. Everything in /etc/systemd/system/nginx.service.d/ gets merged in alphabetical order. So 10-restart.conf, 20-limits.conf — each file handles one concern.

Path 2: Full copy (systemctl edit --full) — use this when you need to change something that can't be done additively.

systemctl edit --full nginx

This copies the entire unit file to /etc/systemd/system/nginx.service and opens it. Your copy now completely replaces the package version.

The tradeoff: you own it now. If the package ships an update to the unit file, you won't get it automatically. Your copy wins, always.

When does this make sense? When you need to remove a directive rather than add one. Drop-ins can add and override, but they can't delete. If the upstream unit has After=network-online.target and you want to remove that dependency, a drop-in can't do it — you need the full copy.

Example: package ships a unit with ProtectHome=true but your service legitimately needs to read /home. You need to either override the value or remove it entirely.

# /etc/systemd/system/myapp.service (full copy)
[Unit]
Description=My App
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
# ProtectHome removed intentionally — needs /home access
Restart=on-failure

[Install]
WantedBy=multi-user.target

The decision rule:

Adding or changing a directive → drop-in
Removing a directive → full copy
Paranoid about upgrades wiping your changes → drop-in
Unit is so customized it's essentially yours now → full copy

After either change, always reload:

systemctl daemon-reload

Dependencies: Wants vs Requires

This trips everyone up.

Requires=postgresql.service
After=postgresql.service

Wants=postgresql.service
After=postgresql.service

Requires is hard. If postgresql.service fails, your unit fails. If your unit stops, postgresql.service stops too (by default).

Wants is soft. "I'd like this, but if it's not available, start me anyway."

And After is separate from both. It's purely about ordering. You can have After= without Wants= or Requires=. That just means "if we're both starting, start me after that."

Most services should use Wants= + After= rather than Requires=. Tight coupling causes cascading failures. Loose coupling with ordering is usually what you actually want.

The Dependency Graph

Want to see the actual dependency tree?

systemctl list-dependencies nginx

Or reverse it — what depends on this unit?

systemctl list-dependencies --reverse nginx

This is how you debug "why is X starting before Y" or "why does stopping A also stop B".

Targets: The Modern Runlevel

Old SysV had runlevels. 0 = halt, 1 = single user, 3 = multi-user, 5 = graphical, 6 = reboot.

Systemd has targets. They're not 1:1 replacements — they're more flexible.

systemctl get-default                # what target boots into
systemctl set-default multi-user.target  # change it
systemctl isolate rescue.target     # switch to single-user now

The important ones for servers:

multi-user.target — fully operational, no GUI. This is where your services live.
network.target — network interfaces are up (not necessarily configured)
network-online.target — network is actually routable. Use this for services that need real connectivity.
rescue.target — minimal single-user mode

The difference between network.target and network-online.target bites people constantly. network.target is reached very early — interfaces are up but DHCP might not have finished. If your service needs to make outbound connections at startup, use network-online.target.

Journald: Your Actual Logs

journalctl is the interface to systemd's logging daemon, journald. Every service that runs under systemd has its output captured automatically. No log file configuration required.

The commands you actually need:

# Follow a service's logs live
journalctl -u nginx -f

# Show logs since last boot
journalctl -u nginx -b

# Show logs from two boots ago
journalctl -u nginx -b -2

# Show logs for a time range
journalctl -u nginx --since "2026-02-25 10:00" --until "2026-02-25 11:00"

# Show only errors and above
journalctl -u nginx -p err

# Show last 100 lines
journalctl -u nginx -n 100

# Show kernel messages
journalctl -k

# Show everything from this boot
journalctl -b

Journal storage is in /var/log/journal/ if persistent, /run/log/journal/ if volatile (gone on reboot).

Check your config:

cat /etc/systemd/journald.conf | grep Storage

Set Storage=persistent to keep logs across reboots. Set SystemMaxUse=500M to cap disk usage.

Timers: Kill Your Crontab

Systemd timers are cron done right. They log to the journal, they have dependency awareness, they can be triggered by events, not just time.

A timer consists of two files. The timer unit and the service unit it activates.

/etc/systemd/system/backup.service:

[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=backup

/etc/systemd/system/backup.timer:

[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable and start the timer (not the service):

systemctl enable --now backup.timer

Persistent=true is key — if the system was off when the job should have run, it runs immediately on next boot. Cron can't do that.

Check all your timers:

systemctl list-timers

This shows the last run time, next run time, and time until next trigger. Far more useful than crontab -l.

Timer schedule syntax:

OnCalendar=daily              # midnight every day
OnCalendar=weekly             # Monday midnight
OnCalendar=Mon *-*-* 04:00:00 # Every Monday at 4 AM
OnCalendar=*:0/15             # Every 15 minutes
OnBootSec=5min                # 5 minutes after boot
OnUnitActiveSec=1h            # 1 hour after last run

Test your calendar expression before deploying:

systemd-analyze calendar "Mon *-*-* 04:00:00"

Service Hardening

This is the part nobody talks about but everyone should use. Systemd has a rich set of security directives that sandbox your services at the kernel level. No AppArmor profile needed. No manual seccomp configuration. Just add lines to your unit file.

[Service]
# Give the service its own /tmp, isolated from the host
PrivateTmp=true

# Service can't see /home, /root, or /run/user
ProtectHome=true

# Mount /usr, /boot, /etc as read-only
ProtectSystem=strict

# No new privileges via setuid/capabilities
NoNewPrivileges=true

# Restrict which address families it can use
RestrictAddressFamilies=AF_INET AF_INET6

# Kill all processes in the unit on stop
KillMode=control-group

# Run as a specific user
User=myservice
Group=myservice

# Give it only the capabilities it needs
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

Run a security audit on any service:

systemd-analyze security nginx

This gives you a score and a breakdown of what's hardened and what isn't. Not all services can be fully locked down — some legitimately need broad access — but most services are running with far more privileges than they need.

Resource Control with Slices

Every service runs inside a cgroup. Systemd exposes this through slices.

The hierarchy looks like:

system.slice ← system services
└─ nginx.service
└─ postgresql.service
user.slice ← user sessions
machine.slice ← VMs and containers

You can set resource limits directly in unit files:

[Service]
# Limit CPU usage
CPUQuota=50%

# Limit memory
MemoryMax=512M
MemorySwapMax=0

# Limit IO weight (100 is default)
IOWeight=50

# Limit number of tasks (processes + threads)
TasksMax=100

Or create a custom slice for a group of services:

# /etc/systemd/system/myapp.slice
[Slice]
MemoryMax=2G
CPUQuota=200%

Then in your services:

[Service]
Slice=myapp.slice

Now all your app's services share that memory budget. This is how you do resource isolation without containers.

Check real-time resource usage:

systemd-cgtop

Socket Activation

One of systemd's underused features. Instead of a service listening on a port 24/7, systemd holds the socket and starts the service only when a connection arrives.

/etc/systemd/system/myapp.socket:

[Unit]
Description=My app socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

/etc/systemd/system/myapp.service:

[Unit]
Description=My app
Requires=myapp.socket

[Service]
ExecStart=/usr/local/bin/myapp
StandardInput=socket

Benefits: the port is held open immediately at boot (no race condition), the service starts on-demand (saves memory when idle), and if the service crashes the socket stays open so connections queue rather than fail.

Debugging Boot Problems

Boot takes too long? Find out why:

# Total boot time breakdown
systemd-analyze

# Breakdown by service
systemd-analyze blame

# Visual critical path
systemd-analyze critical-chain

# Generate an SVG plot of the entire boot
systemd-analyze plot > boot.svg

critical-chain is the one you actually want. It shows you the chain of dependencies that determined your total boot time. Fix the slowest link in that chain.

systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @12.543s
└─multi-user.target @12.542s
  └─postgresql.service @8.123s +4.201s
    └─network-online.target @8.100s

Here, PostgreSQL waited for network-online.target and then took 4 seconds to start. That's your bottleneck.

The Commands You'll Actually Use

# Start/stop/restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx    # reload config without restart (if supported)

# Enable/disable at boot
systemctl enable nginx
systemctl disable nginx
systemctl enable --now nginx    # enable AND start in one command

# Status and inspection
systemctl status nginx
systemctl is-active nginx
systemctl is-enabled nginx
systemctl is-failed nginx

# Edit units
systemctl edit nginx         # drop-in override
systemctl edit --full nginx  # full copy to /etc

# After editing any unit file
systemctl daemon-reload

# List units
systemctl list-units
systemctl list-units --failed
systemctl list-unit-files

# Logs
journalctl -u nginx -f
journalctl -u nginx -b -p err

# Boot analysis
systemd-analyze blame
systemd-analyze critical-chain

Honest Verdict

Systemd is genuinely good.

The people who hate it usually hate it for political reasons (it's "too complex", it "does too much") or because they had to learn it under pressure.

The timer system is better than cron. The journal is better than scattered log files. The dependency model is better than numbered init scripts. The security directives are a free hardening layer most people leave on the table.

Learn it properly once and you'll stop fighting it.

Go Try It

Start here:

systemd-analyze blame

Find the slowest service on your system. Open its unit file with systemctl cat servicename. Figure out if it actually needs to start at boot or if it can be disabled or made socket-activated.

That one exercise will teach you more about systemd than reading documentation for a week.

Compiled by AI. Proofread by caffeine. ☕