Linux Services & Daemons Management Guide - DevOps

Linux Services & Daemons Management Guide

Published: December 2025 | Topic: Linux Services & Daemons for DevOps

Services are the backbone of Linux systems. Understanding how to manage services with systemd is crucial for deploying, maintaining, and troubleshooting applications. This guide covers systemd, unit files, service lifecycle, and practical management techniques.

1. Understanding Systemd & Services

What is Systemd?

Systemd is a system and service manager for Linux operating systems. It's the first process to start (PID 1) and manages the boot process, service lifecycle, logging, and more.

Key Features: Parallel startup, on-demand activation, service dependencies, snapshot support, and comprehensive logging via journald.

Service Lifecycle States

Active (Running)

Running

Service is currently running

Active (Exited)

Exited

Completed successfully

Inactive

Stopped

Service is not running

Failed

Failed

Service failed to start

Systemctl Basics

Service Status & Information

# Check service status:
$ systemctl status nginx.service
$ systemctl status docker
$ systemctl is-active nginx # Returns: active/inactive
$ systemctl is-enabled nginx # Returns: enabled/disabled
$ systemctl is-failed nginx # Returns: failed/active

# List all units:
$ systemctl list-units
$ systemctl list-units --type=service
$ systemctl list-units --state=failed
$ systemctl list-unit-files # All installed unit files

Service Lifecycle Management

# Start/Stop/Restart services:
$ sudo systemctl start nginx
$ sudo systemctl stop nginx
$ sudo systemctl restart nginx
$ sudo systemctl reload nginx # Reload config without restart
$ sudo systemctl reload-or-restart nginx

# Enable/Disable at boot:
$ sudo systemctl enable nginx
$ sudo systemctl disable nginx
$ sudo systemctl enable --now nginx # Enable and start

# Mask/Unmask services:
$ sudo systemctl mask nginx # Prevent all starts
$ sudo systemctl unmask nginx # Remove mask

Service Types & Categories

Common Service Types

# Web Servers:
nginx.service
apache2.service # Debian/Ubuntu
httpd.service # RHEL/CentOS

# Database Servers:
mysql.service
postgresql.service
mongod.service

# Containerization:
docker.service
containerd.service
podman.service

# Monitoring & Logging:
prometheus.service
grafana-server.service
elasticsearch.service
fluentd.service

# System Services:
sshd.service # SSH Daemon
crond.service # Cron Daemon
systemd-journald.service # Journal Daemon

Finding Services

# Search for services:
$ systemctl list-unit-files | grep nginx
$ systemctl list-units --all | grep mysql

# Find service by process:
$ systemctl status $(pidof nginx)
$ systemctl status $(pgrep mysqld)

# Check service dependencies:
$ systemctl list-dependencies nginx.service
$ systemctl list-dependencies --reverse nginx.service

# Show service properties:
$ systemctl show nginx.service
$ systemctl show --property=Requires nginx.service
$ systemctl show --property=After nginx.service

2. Understanding Unit Files

Unit File Structure

# Unit files are located in:
/lib/systemd/system/ # System-provided units
/etc/systemd/system/ # User/Admin created units
/usr/lib/systemd/system/ # Package-installed units (RHEL)

# Basic unit file structure:
[Unit]
Description=Service Description
Documentation=man:service(8) https://example.com
After=network.target
Requires=network.target
Wants=some-other.service
Conflicts=conflicting.service

[Service]
Type=simple
ExecStart=/usr/bin/command --options
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10
User=service-user
Group=service-group
Environment="KEY=value"
WorkingDirectory=/path/to/workdir

[Install]
WantedBy=multi-user.target
Alias=service-alias.service

Service Section Types

Type=simple (Default)

# ExecStart is main process
# Systemd considers service started immediately
# Good for long-running services

[Service]
Type=simple
ExecStart=/usr/bin/myserver
User=appuser
Restart=always

Type=forking

# Process forks and parent exits
# Systemd needs PIDFile to track
# Traditional daemon behavior

[Service]
Type=forking
PIDFile=/var/run/myserver.pid
ExecStart=/usr/bin/myserver --daemon
ExecStop=/bin/kill $MAINPID
User=appuser

Type=oneshot

# Runs once and exits
# Used for scripts/tasks
# Often combined with RemainAfterExit

[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Type=notify

# Service notifies systemd when ready
# Uses sd_notify() system call
# For services that need to signal readiness

[Service]
Type=notify
ExecStart=/usr/bin/myserver --notify
NotifyAccess=all
TimeoutStartSec=30
Restart=on-failure

Key Directives Explained

[Unit] Section

Description=Human readable description
Documentation=man:page or URL
After=service1.service service2.target
Requires=service1.service # Hard dependency
Wants=service2.service # Soft dependency
Conflicts=conflict.service # Cannot run together
Before=service3.service # Ordering
ConditionPathExists=/path/file
ConditionPathIsDirectory=/path
AssertPathExists=/path/file

[Service] Section

Type=simple|forking|oneshot|dbus|notify
ExecStart=/path/to/command args
ExecStartPre=/path/to/pre-script
ExecStartPost=/path/to/post-script
ExecStop=/path/to/stop-command
ExecReload=/bin/kill -HUP $MAINPID
Restart=no|on-success|on-failure|always
RestartSec=10 # seconds before restart
User=username
Group=groupname
WorkingDirectory=/path
Environment="KEY=value"
EnvironmentFile=/etc/default/service
PIDFile=/var/run/service.pid
TimeoutStartSec=30
TimeoutStopSec=30

[Install] Section

WantedBy=multi-user.target
RequiredBy=some.target
Also=other-unit.service # Install additional units
Alias=service-alias.service

# Common targets:
multi-user.target # Normal multi-user mode
graphical.target # GUI mode
default.target # Default runlevel
network-online.target # When network is ready
sysinit.target # Early boot
shutdown.target # System shutdown

3. Creating Custom Services

Basic Custom Service Example

Simple Python Application Service

# File: /etc/systemd/system/myapp.service

[Unit]
Description=My Python Application
Documentation=https://github.com/user/myapp
After=network.target
Wants=network.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp

# Set environment variables
Environment="APP_ENV=production"
Environment="LOG_LEVEL=INFO"
EnvironmentFile=/etc/default/myapp

# Command to execute
ExecStart=/usr/bin/python3 /opt/myapp/main.py

# Reload sends SIGHUP
ExecReload=/bin/kill -HUP $MAINPID

# Restart policy
Restart=on-failure
RestartSec=10
StartLimitIntervalSec=60
StartLimitBurst=3

# Security options
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target

# Environment file: /etc/default/myapp
DB_HOST=localhost
DB_PORT=5432
DB_NAME=myapp
SECRET_KEY=changeme
DEBUG=false

Advanced Service Examples

Node.js Application Service

[Unit]
Description=Node.js Application
After=network.target mongodb.service
Requires=mongodb.service

[Service]
Type=simple
User=nodeapp
Group=nodeapp
WorkingDirectory=/var/www/nodeapp

# Using npm start
ExecStart=/usr/bin/npm start

# Or directly with node
# ExecStart=/usr/bin/node /var/www/nodeapp/app.js

# Environment
Environment=NODE_ENV=production
Environment=PORT=3000

# Process management
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Docker Container Service

[Unit]
Description=My Docker Container
Requires=docker.service
After=docker.service network.target

[Service]
Type=simple

# Pull and run container
ExecStartPre=-/usr/bin/docker rm -f mycontainer
ExecStartPre=/usr/bin/docker pull myimage:latest
ExecStart=/usr/bin/docker run \
  --name mycontainer \
  -p 8080:80 \
  -v /data:/app/data \
  myimage:latest

ExecStop=/usr/bin/docker stop mycontainer
ExecStopPost=/usr/bin/docker rm -f mycontainer

Restart=always
RestartSec=10
TimeoutStartSec=120

[Install]
WantedBy=multi-user.target

Shell Script Service

[Unit]
Description=Backup Service
After=network.target

[Service]
Type=oneshot
User=backup
Group=backup
WorkingDirectory=/backup

# The actual backup script
ExecStart=/usr/local/bin/backup.sh

# Send email on failure
ExecStopPost=/usr/local/bin/notify-backup-failure.sh

# Log output to syslog
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=backup-service

[Install]
WantedBy=multi-user.target

Timer-based Service

# Service file: /etc/systemd/system/cleanup.service
[Unit]
Description=Daily Cleanup Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/cleanup.sh
User=root

# Timer file: /etc/systemd/system/cleanup.timer
[Unit]
Description=Run cleanup daily at 2 AM

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Service Creation Workflow

# Step 1: Create service file
$ sudo nano /etc/systemd/system/myapp.service

# Step 2: Create application user (optional)
$ sudo useradd -r -s /bin/false myappuser

# Step 3: Set permissions
$ sudo chown root:root /etc/systemd/system/myapp.service
$ sudo chmod 644 /etc/systemd/system/myapp.service

# Step 4: Reload systemd daemon
$ sudo systemctl daemon-reload

# Step 5: Enable and start service
$ sudo systemctl enable myapp.service
$ sudo systemctl start myapp.service

# Step 6: Verify service status
$ systemctl status myapp.service
$ journalctl -u myapp.service -f

4. Advanced Service Management

Dependency Management

Service Dependencies

# View dependencies:
$ systemctl list-dependencies nginx.service
$ systemctl list-dependencies --reverse nginx.service
$ systemctl list-dependencies --before nginx.service
$ systemctl list-dependencies --after nginx.service

# Check what requires a service:
$ systemctl list-dependencies --reverse docker.service

# Create dependency chain:
[Unit]
Description=My Application
After=postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service

# After: starts after these services
# Requires: hard dependency (stops if dependency fails)
# Wants: soft dependency (starts if available)

Targets & Runlevels

# List available targets:
$ systemctl list-units --type=target
$ systemctl list-unit-files --type=target

# Change default target:
$ sudo systemctl set-default multi-user.target
$ sudo systemctl set-default graphical.target

# Switch to different target:
$ sudo systemctl isolate multi-user.target
$ sudo systemctl isolate rescue.target

# Emergency mode:
$ sudo systemctl emergency

# Rescue mode:
$ sudo systemctl rescue

# Map old runlevels:
# runlevel 3 -> multi-user.target
# runlevel 5 -> graphical.target

Service Logging & Debugging

Journalctl for Services

# View service logs:
$ journalctl -u nginx.service
$ journalctl -u nginx.service --since "1 hour ago"
$ journalctl -u nginx.service --since today
$ journalctl -u nginx.service -f # Follow logs

# Filter by priority:
$ journalctl -u nginx.service -p err
$ journalctl -u nginx.service -p warning
$ journalctl -u nginx.service --grep="error"

# Show specific fields:
$ journalctl -u nginx.service -o verbose
$ journalctl -u nginx.service -o json
$ journalctl -u nginx.service --output-fields=MESSAGE,_PID

# Show logs from specific boot:
$ journalctl -u nginx.service -b -1 # Previous boot

Debugging Failed Services

# Check service status:
$ systemctl status failed.service
$ systemctl is-failed failed.service

# View detailed failure info:
$ journalctl -u failed.service -xe
$ journalctl -u failed.service --no-pager | tail -50

# Reset failed state:
$ sudo systemctl reset-failed failed.service

# Test service without starting:
$ sudo systemctl dry-run start myservice

# Check service configuration:
$ systemd-analyze verify /etc/systemd/system/myservice.service
$ systemd-analyze cat-config myservice.service

Security & Isolation

Security Directives

[Service]
# User/Group isolation:
User=appuser
Group=appuser
DynamicUser=yes # Automatic ephemeral user
SupplementaryGroups=www-data,docker

# Filesystem isolation:
ProtectSystem=strict
ReadOnlyPaths=/
ReadWritePaths=/var/lib/myapp
PrivateTmp=true
PrivateDevices=true
NoNewPrivileges=true

# Network isolation:
PrivateNetwork=false
IPAddressDeny=any
IPAddressAllow=localhost
IPAddressAllow=192.168.1.0/24

# Capabilities:
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Resource limits:
LimitNOFILE=65536
LimitNPROC=512
LimitMEMLOCK=infinity

Resource Control

[Service]
# CPU limits:
CPUQuota=50% # Max 50% of one CPU core
CPUWeight=100 # Relative CPU share (1-10000)
CPUShares=1024 # Legacy, use CPUWeight

# Memory limits:
MemoryMax=512M # Hard memory limit
MemoryHigh=256M # Soft memory limit
MemorySwapMax=1G # Swap limit

# I/O limits:
IOWeight=100 # Relative I/O weight
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M

# Block I/O:
BlockIOWeight=100
BlockIOReadBandwidth=/dev/sdb 1M
BlockIOWriteBandwidth=/dev/sdb 1M

# Tasks limit:
TasksMax=100 # Maximum number of tasks

# Check current cgroup:
$ systemctl show --property=ControlGroup nginx.service

5. Troubleshooting & Best Practices

Common Service Issues

Service Won't Start

# Check service status:
$ systemctl status myservice

# View logs:
$ journalctl -u myservice -xe
$ journalctl -u myservice --no-pager | tail -100

# Test configuration:
$ systemd-analyze verify /etc/systemd/system/myservice.service

# Check dependencies:
$ systemctl list-dependencies myservice.service

# Manual start for debugging:
$ sudo -u appuser /path/to/command

# Check permissions:
$ ls -la /path/to/command
$ id appuser

Service Crashes/Restarts

# Check restart count:
$ systemctl show myservice -p NRestarts

# View crash logs:
$ journalctl -u myservice --since "5 minutes ago"

# Check memory limits:
$ systemctl show myservice -p MemoryCurrent

# Adjust restart policy:
[Service]
Restart=on-failure
RestartSec=10
StartLimitIntervalSec=60
StartLimitBurst=3

# Check for signal 9 (OOM killer):
$ dmesg | grep -i "killed process"

# Monitor resource usage:
$ systemd-cgtop

Best Practices

Do's

• Always create a dedicated user for services
• Use WorkingDirectory to set proper context
• Set appropriate Restart policies
• Use EnvironmentFile for configuration
• Implement proper logging with StandardOutput and StandardError
• Use systemctl daemon-reload after modifying unit files
• Test services with systemctl start --dry-run
• Set resource limits to prevent runaway processes

Don'ts

• Don't run services as root unless absolutely necessary
• Don't use Type=simple for forking daemons
• Don't forget to set User and Group
• Don't hardcode secrets in unit files
• Don't use Restart=always for oneshot services
• Don't modify system-provided unit files directly
• Don't ignore service dependencies
• Don't forget to enable services for auto-start

Service Management Script

Automated Service Deployment Script

#!/bin/bash
# deploy_service.sh - Automated service deployment

deploy_service() {
    local SERVICE_NAME="$1"
    local SERVICE_FILE="/etc/systemd/system/$SERVICE_NAME.service"
    local APP_USER="appuser"
    local APP_DIR="/opt/$SERVICE_NAME"

    echo "Deploying service: $SERVICE_NAME"

    # Create application user if doesn't exist
    if ! id "$APP_USER" &>/dev/null; then
        sudo useradd -r -s /bin/false "$APP_USER"
        echo "Created user: $APP_USER"
    fi

    # Create service directory
    sudo mkdir -p "$APP_DIR"
    sudo chown -R "$APP_USER:$APP_USER" "$APP_DIR"

    # Create service file
    sudo tee "$SERVICE_FILE" > /dev/null << EOF
[Unit]
Description=$SERVICE_NAME Service
After=network.target

[Service]
Type=simple
User=$APP_USER
Group=$APP_USER
WorkingDirectory=$APP_DIR
ExecStart=/usr/bin/python3 $APP_DIR/app.py
Restart=on-failure
RestartSec=10
Environment="PYTHONUNBUFFERED=1"

[Install]
WantedBy=multi-user.target
EOF

    # Set permissions
    sudo chmod 644 "$SERVICE_FILE"

    # Reload systemd
    sudo systemctl daemon-reload

    # Enable and start service
    sudo systemctl enable "$SERVICE_NAME"
    sudo systemctl restart "$SERVICE_NAME"

    # Verify deployment
    sleep 2
    sudo systemctl status "$SERVICE_NAME" --no-pager
}

backup_service() {
    local SERVICE_NAME="$1"
    local BACKUP_DIR="/backup/services"
    local TIMESTAMP="$(date +%Y%m%d_%H%M%S)"

    mkdir -p "$BACKUP_DIR"

    # Backup service file
    if [[ -f "/etc/systemd/system/$SERVICE_NAME.service" ]]; then
        cp "/etc/systemd/system/$SERVICE_NAME.service" \
            "$BACKUP_DIR/${SERVICE_NAME}_${TIMESTAMP}.service"
    fi

    echo "Backup created: $BACKUP_DIR/${SERVICE_NAME}_${TIMESTAMP}.service"
}

main() {
    case "$1" in
        deploy)
            deploy_service "$2"
            ;;
        backup)
            backup_service "$2"
            ;;
        *)
            echo "Usage: $0 {deploy|backup} service_name"
            exit 1
        ;;
    esac
}

main "$@"

Systemctl Command Quick Reference

Service Control

$ systemctl start service
$ systemctl stop service
$ systemctl restart service
$ systemctl reload service
$ systemctl enable service
$ systemctl disable service
$ systemctl mask service
$ systemctl unmask service

Service Status

$ systemctl status service
$ systemctl is-active service
$ systemctl is-enabled service
$ systemctl is-failed service
$ systemctl show service
$ systemctl list-units --type=service
$ systemctl list-unit-files --type=service

System Management

$ systemctl daemon-reload
$ systemctl reboot
$ systemctl poweroff
$ systemctl halt
$ systemctl suspend
$ systemctl rescue
$ systemctl emergency
$ systemctl set-default target

Logs & Debugging

$ journalctl -u service
$ journalctl -u service -f
$ journalctl -u service --since "1 hour ago"
$ journalctl -u service -p err
$ journalctl -xe
$ systemd-analyze verify service
$ systemd-analyze cat-config service

Dependencies & Targets

$ systemctl list-dependencies service
$ systemctl list-dependencies --reverse service
$ systemctl list-units --type=target
$ systemctl get-default
$ systemctl isolate target
$ systemctl list-dependencies target

Timers & Sockets

$ systemctl list-timers
$ systemctl list-timers --all
$ systemctl start timer.timer
$ systemctl enable timer.timer
$ systemctl list-sockets
$ systemctl cat timer.timer
$ systemctl status timer.timer

Best Practices Summary

• Use dedicated users: Never run services as root unless absolutely necessary
• Proper restart policies: Choose appropriate Restart= settings based on service type
• Environment files: Store configuration in EnvironmentFile, not in unit files
• Resource limits: Set MemoryMax, TasksMax, and other limits to prevent runaway processes
• Security hardening: Use PrivateTmp, NoNewPrivileges, ProtectSystem, etc.
• Proper logging: Configure StandardOutput and StandardError appropriately
• Test before deploying: Use systemd-analyze verify to check unit files
• Use override files: Don't modify original unit files; use systemctl edit service
• Monitor service health: Set up monitoring for service status and resource usage
• Document services: Include Description and Documentation in unit files

Mastering systemd service management is essential for modern Linux administration. By understanding unit files, dependencies, and proper service configuration, you can ensure reliable, secure, and maintainable service deployments. Always test configurations in a staging environment and implement proper monitoring and alerting for production services.