Linux Security & Permissions for DevOps

Linux Security & Permissions

Published: December 2025 | Topic: Security Hardening for DevOps

Security is not a feature, it's a fundamental requirement. In DevOps, security must be baked into every layer of your infrastructure. Understanding Linux security mechanisms enables you to build secure systems, protect sensitive data, and maintain compliance in production environments.

Why Security Matters for DevOps

Security-first DevOps provides:

• Data Protection: Safeguard sensitive information and customer data
• System Integrity: Prevent unauthorized modifications and breaches
• Compliance: Meet regulatory requirements (GDPR, HIPAA, PCI-DSS)
• Availability: Protect against denial-of-service attacks
• Trust: Build confidence with stakeholders and customers
• Cost Reduction: Prevent expensive security incidents and data breaches
• Automated Security: Implement security as code in your CI/CD pipelines

Linux Security Defense-in-Depth

Network Layer
Firewalls, VPNs

→

Access Control
SSH, Authentication

→

File Permissions
rwx, ACLs

→

Mandatory Access Control
SELinux, AppArmor

→

Monitoring
Logs, Auditing

1. File & Directory Permissions (rwx) Deep Dive

What are Linux Permissions?

Linux Permissions: A security mechanism that controls access to files and directories. Every file and directory has an owner, a group, and a set of permissions that define what actions different users can perform.

The Permission Model is based on three entities:

• Owner (u): The user who owns the file/directory
• Group (g): The group associated with the file/directory
• Others (o): All other users on the system
• All (a): Represents owner, group, and others combined

Understanding Permission Bits

Decoding Permission Strings

Example: -rwxr-xr--

Type: - (regular file)

Owner: rwx (read, write, execute)

Group: r-x (read, execute)

Others: r-- (read only)

File Type Indicators:
- = Regular file
d = Directory
l = Symbolic link
c = Character device
b = Block device
s = Socket
p = Named pipe

Permission Types Explained

File Permissions

• Read (r): View file contents (cat, less, more)
• Write (w): Modify file contents (vim, echo >)
• Execute (x): Run file as a program (scripts, binaries)

# Examples:
-rw------- # Only owner can read/write
-rwxr-xr-x # Everyone can execute
-rw-r--r-- # Everyone can read, only owner can write

Directory Permissions

• Read (r): List directory contents (ls)
• Write (w): Create/delete files in directory (touch, rm)
• Execute (x): Access directory contents (cd)

# Examples:
drwx------ # Only owner can access
drwxr-xr-x # Everyone can list and access
drwxr-x--x # Group can list, others can access but not list

Numeric (Octal) Permission Representation

Octal Permissions: A numeric representation of permissions where each permission type (read, write, execute) is assigned a value, and these values are summed for each entity (owner, group, others).

Permission	Binary	Octal	Meaning
---	000	0	No permissions
--x	001	1	Execute only
-w-	010	2	Write only
-wx	011	3	Write + Execute
r--	100	4	Read only
r-x	101	5	Read + Execute
rw-	110	6	Read + Write
rwx	111	7	Read + Write + Execute

# Common permission combinations:
644 # -rw-r--r-- (files: owner rw, others r)
755 # -rwxr-xr-x (executables: owner rwx, others rx)
600 # -rw------- (private files)
700 # -rwx------ (private executables)
750 # -rwxr-x--- (owner rwx, group rx, others none)
777 # -rwxrwxrwx (dangerous: everyone can do everything)

# How to calculate:
755 = owner(7) + group(5) + others(5)
    = (4+2+1) + (4+0+1) + (4+0+1)
    = rwx + r-x + r-x

Permission Management Commands

chmod - Change Permissions

# Symbolic mode (user/group/others)
$ chmod u+x script.sh # Add execute for owner
$ chmod g-w file.txt # Remove write for group
$ chmod o=r file.txt # Set others to read only
$ chmod a+r file.txt # Add read for everyone
$ chmod u=rwx,g=rx,o= script.sh

# Octal mode (numeric)
$ chmod 755 script.sh # rwxr-xr-x
$ chmod 644 file.txt # rw-r--r--
$ chmod 600 secret.txt # rw-------

# Recursive changes
$ chmod -R 755 directory/ # Apply to all files and subdirectories

chown & chgrp - Change Ownership

# Change file owner
$ sudo chown alice file.txt
$ sudo chown alice:developers file.txt
# user:group format

# Change file group
$ sudo chgrp developers file.txt
$ chown :developers file.txt # Alternative syntax

# Recursive ownership changes
$ sudo chown -R alice:developers project/

# Reference another file's ownership
$ sudo chown --reference=file1.txt file2.txt

umask - Default Permissions

# What is umask?
# A mask that subtracts permissions from default values

# Check current umask
$ umask
0022 # Common default

# Set umask for current session
$ umask 0077 # Most restrictive
$ umask 0022 # Standard restrictive
$ umask 0002 # Permissive (group writable)

# How umask works:
Default file permissions: 666 (rw-rw-rw-)
Default directory permissions: 777 (rwxrwxrwx)
With umask 022 → Files: 644, Directories: 755
With umask 077 → Files: 600, Directories: 700

Real-World Permission Scenarios

# Scenario 1: Web server directory
$ sudo mkdir -p /var/www/html
$ sudo chown www-data:www-data /var/www/html
$ sudo chmod 755 /var/www/html # Readable/executable by all
$ sudo chmod 644 /var/www/html/*.html # HTML files readable by all

# Scenario 2: Shared directory for team
$ sudo mkdir /shared
$ sudo chown root:developers /shared
$ sudo chmod 2770 /shared # SGID + rwx for owner/group
# Files created in /shared will inherit developers group

# Scenario 3: Secure configuration file
$ sudo touch /etc/app/config.conf
$ sudo chown root:root /etc/app/config.conf
$ sudo chmod 600 /etc/app/config.conf # Only root can read/write

# Scenario 4: User's private directory
$ mkdir ~/private
$ chmod 700 ~/private # Only owner can access

⚠️ Common Permission Mistakes

• World-writable files/directories: Anyone can modify/delete
• SUID/SGID on writable scripts: Can lead to privilege escalation
• Incorrect web server permissions: Can expose sensitive data
• Recursive 777 permissions: Extremely dangerous
• Unnecessary execute permissions: Increases attack surface
• Root-owned user files: Users can't modify their own files
• Incorrect umask: Creates files with wrong default permissions

2. Sticky Bit, SUID, SGID

What are Special Permission Bits?

Special Permission Bits: Additional permission settings that provide special functionality beyond the standard read, write, and execute permissions. These include Set User ID (SUID), Set Group ID (SGID), and the Sticky Bit.

Special Bits in Permission Strings

Example: -rwsr-sr-t

SUID: s in owner execute position

SGID: s in group execute position

Sticky Bit: t in others execute position

Special Bits Representation:
s or S = SUID/SGID (lowercase if executable, uppercase if not)
t or T = Sticky Bit (lowercase if executable, uppercase if not)
4000 = SUID bit
2000 = SGID bit
1000 = Sticky bit

Set User ID (SUID)

SUID (Set User ID): When set on an executable file, the program runs with the privileges of the file's owner, rather than the user who executed it. This allows users to run specific programs with elevated privileges.

How SUID Works

Normal execution: Program runs with user's privileges

With SUID: Program runs with owner's privileges

# Example: /usr/bin/passwd
$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 59976 Nov 24 2022 /usr/bin/passwd
# 's' in owner execute position = SUID

# When a normal user runs passwd:
# 1. User executes /usr/bin/passwd
# 2. Because SUID is set, it runs as root
# 3. Can modify /etc/shadow (owned by root)

SUID Security Considerations

• Privilege Escalation Risk: Bugs in SUID programs can be exploited
• Minimize Usage: Only essential programs should have SUID
• Regular Audits: Monitor SUID files on system
• No SUID on Scripts: Most shells ignore SUID on scripts
• Principle of Least Privilege: Use minimal necessary privileges

# Find all SUID files
$ find / -type f -perm -4000 2>/dev/null
# Common SUID binaries:
/usr/bin/passwd
/usr/bin/sudo
/usr/bin/pkexec
/bin/su
/bin/mount

Set Group ID (SGID)

SGID (Set Group ID): Has two different behaviors depending on whether it's set on a file or directory:

• On executable files: Program runs with the privileges of the file's group
• On directories: Files created in the directory inherit the directory's group

SGID on Files

# Example: /usr/bin/wall
$ ls -l /usr/bin/wall
-rwxr-sr-x 1 root tty 35064 Jan 21 2023 /usr/bin/wall
# 's' in group execute position = SGID

# When user runs wall:
# 1. Runs with tty group privileges
# 2. Can write to /dev/tty* devices
# 3. Allows sending messages to all terminals

SGID on Directories

# Create shared directory with SGID
$ sudo mkdir /shared
$ sudo chown root:developers /shared
$ sudo chmod 2770 /shared
# 2 = SGID bit (2000)

# Test SGID behavior
$ cd /shared
$ touch testfile
$ ls -l testfile
-rw-r--r-- 1 alice developers 0 Dec 1 10:00 testfile
# Note: group is 'developers' not alice's primary group
# Files inherit directory's group (developers)

Sticky Bit

Sticky Bit: When set on a directory, prevents users from deleting or renaming files they don't own. Originally used for keeping executables in memory, now primarily used for shared directories like /tmp.

# Example: /tmp directory
$ ls -ld /tmp
drwxrwxrwt 15 root root 4096 Dec 1 10:00 /tmp
# 't' in others execute position = Sticky Bit

# How sticky bit works in /tmp:
# 1. User alice creates file /tmp/alicefile
# 2. User bob cannot delete /tmp/alicefile
# 3. Only root or alice can delete it
# 4. But bob can create his own files in /tmp

# Without sticky bit:
# 1. Anyone could delete anyone else's files in shared directory
# 2. Could cause chaos in collaborative environments

Setting Special Permission Bits

Symbolic Method

# Set SUID on file
$ chmod u+s /path/to/file
# Set SGID on file
$ chmod g+s /path/to/file
# Set sticky bit on directory
$ chmod +t /path/to/directory

# Remove special bits
$ chmod u-s /path/to/file
$ chmod g-s /path/to/file
$ chmod -t /path/to/directory

Numeric (Octal) Method

# Special bits as prefix to regular permissions:
# SUID = 4000, SGID = 2000, Sticky = 1000

# SUID examples:
chmod 4755 file # SUID + rwxr-xr-x
chmod 4711 file # SUID + rwx--x--x

# SGID examples:
chmod 2755 file # SGID + rwxr-xr-x
chmod 2770 dir # SGID on directory

# Sticky bit examples:
chmod 1777 dir # Sticky + rwxrwxrwx (/tmp style)
chmod 1755 dir # Sticky + rwxr-xr-x

# Combined examples:
chmod 6755 file # SUID + SGID (4000+2000+755)
chmod 3777 dir # SGID + Sticky (2000+1000+777)

Finding Special Bits

# Find SUID files
$ find / -type f -perm -4000 2>/dev/null

# Find SGID files
$ find / -type f -perm -2000 2>/dev/null

# Find SGID directories
$ find / -type d -perm -2000 2>/dev/null

# Find sticky bit directories
$ find / -type d -perm -1000 2>/dev/null

# Find world-writable directories with sticky bit
$ find / -type d -perm -1777 2>/dev/null

# Find files with both SUID and SGID
$ find / -type f -perm -6000 2>/dev/null

Practical Use Cases for Special Bits

# Use Case 1: Shared project directory (SGID + Sticky)
$ sudo mkdir /projects
$ sudo chown root:devteam /projects
$ sudo chmod 3770 /projects # SGID + Sticky + rwx for owner/group
# Result: Team members can collaborate, files inherit devteam group,
# members can't delete each other's files

# Use Case 2: Backup script requiring root privileges (SUID)
$ sudo cp backup_script.sh /usr/local/bin/
$ sudo chown root:root /usr/local/bin/backup_script.sh
$ sudo chmod 4750 /usr/local/bin/backup_script.sh
# Result: Backup operators can run script as root to backup system files

# Use Case 3: Log directory for multiple services (SGID)
$ sudo mkdir /var/log/multiservice
$ sudo chown root:loggers /var/log/multiservice
$ sudo chmod 2775 /var/log/multiservice
# Result: Multiple services can write logs, all logs have same group

# Use Case 4: Public upload directory (Sticky Bit)
$ sudo mkdir /var/www/uploads
$ sudo chown www-data:www-data /var/www/uploads
$ sudo chmod 1777 /var/www/uploads
# Result: Web server can write, users can upload, but can't delete others' files

⚠️ Security Best Practices for Special Bits

• Audit regularly: Use find commands to check for special bits
• Minimize SUID/SGID: Remove unnecessary special bits
• Secure SUID programs: Ensure they're properly coded and maintained
• Never SUID shell scripts: Most shells ignore SUID on scripts
• Use capabilities instead: Modern alternative to SUID for specific privileges
• Document special bits: Keep records of why special bits are needed
• Test thoroughly: Ensure special bits work as intended without creating vulnerabilities
• Consider alternatives: sudo, capabilities, or dedicated services may be better

3. Firewall Setup & SSH Hardening

The Importance of Network Security

Defense in Depth: A security strategy that employs multiple layers of security controls. Firewalls and SSH hardening are critical components of the network security layer, protecting systems from unauthorized access and network-based attacks.

Firewall Fundamentals

What is a Firewall?

A network security system that monitors and controls incoming and outgoing network traffic based on predetermined security rules.

# Firewall functions:
1. Packet filtering
2. Stateful inspection
3. Network Address Translation (NAT)
4. Port forwarding
5. Traffic logging

Firewall Strategies

• Default Deny: Block all, allow specific (recommended)
• Default Allow: Allow all, block specific (dangerous)
• Whitelisting: Only explicitly allowed traffic
• Blacklisting: Block known bad traffic

# Default deny example:
Default policy: DROP all incoming
Explicitly allow: SSH, HTTP, HTTPS
Everything else: Blocked

SSH Hardening: Why It's Critical

SSH (Secure Shell): The primary method for remote administration of Linux servers. Because SSH provides direct access to your systems, it's a prime target for attackers and must be properly secured.

Common SSH Attack Vectors

Attack Type	Description	Protection
Brute Force	Repeated login attempts with common passwords	Fail2ban, strong passwords, key auth
Password Spraying	Try few passwords against many users	Account lockout, monitoring
Credential Stuffing	Use leaked credentials from other sites	Unique passwords, MFA
Man-in-the-Middle	Intercept SSH connections	Strict host checking, SSH certificates
Protocol Exploits	Vulnerabilities in SSH protocol	Keep SSH updated, disable weak ciphers

Comprehensive SSH Hardening Guide

Authentication Hardening

# /etc/ssh/sshd_config - Critical settings
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
PermitEmptyPasswords no
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no

# User restrictions
AllowUsers alice bob charlie
AllowGroups ssh-users
DenyUsers baduser hacker
DenyGroups blacklisted

# Login attempts
MaxAuthTries 3
MaxSessions 10
LoginGraceTime 60

Cryptographic Hardening

# Key exchange algorithms
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

# Ciphers (encryption)
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com

# MACs (message authentication)
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Host keys
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

# Disable weak algorithms
# Remove ssh-dss, hmac-md5, etc.

Network & Connection Hardening

# Change default port
Port 2222
# Or use multiple ports
#Port 22
#Port 2222

# Bind to specific interface
ListenAddress 192.168.1.100
#ListenAddress ::: # IPv6

# Connection settings
ClientAliveInterval 300
ClientAliveCountMax 2
TCPKeepAlive yes

# Restrict features
AllowTcpForwarding no
X11Forwarding no
PermitTunnel no
AllowAgentForwarding no

Complete SSH Hardening Implementation

# Step 1: Backup original config
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

# Step 2: Generate strong host keys
$ sudo ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key
$ sudo ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key

# Step 3: Configure /etc/ssh/sshd_config
$ sudo nano /etc/ssh/sshd_config
# Add/change these settings:
Port 2222
AddressFamily inet # IPv4 only, or 'inet6' for IPv6 only
ListenAddress 192.168.1.100
PermitRootLogin no
MaxAuthTries 3
MaxSessions 10
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes
AllowUsers alice bob charlie
X11Forwarding no
PrintMotd no
ClientAliveInterval 300
ClientAliveCountMax 2
# Add cryptographic settings from above

# Step 4: Create SSH user group
$ sudo groupadd ssh-users
$ sudo usermod -a -G ssh-users alice
$ sudo usermod -a -G ssh-users bob

# Step 5: Configure fail2ban for SSH
$ sudo apt install fail2ban
$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
$ sudo nano /etc/fail2ban/jail.local
# Set:
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600

# Step 6: Test configuration
$ sudo sshd -t # Test config syntax
$ sudo systemctl reload sshd

# Step 7: Keep old SSH running for testing
$ sudo systemctl start ssh # Original service on port 22
# Test new SSH on port 2222 before disabling old one

Firewall Integration with SSH

UFW with SSH (Ubuntu/Debian)

# Install and enable UFW
$ sudo apt install ufw
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing

# Allow SSH on non-standard port
$ sudo ufw allow 2222/tcp
# Or limit to specific IPs
$ sudo ufw allow from 192.168.1.0/24 to any port 2222

# Enable rate limiting
$ sudo ufw limit 2222/tcp
# Limits connections to 6 per 30 seconds

$ sudo ufw enable
$ sudo ufw status verbose

firewalld with SSH (RHEL/CentOS)

# Allow SSH on custom port
$ sudo firewall-cmd --permanent --remove-service=ssh
$ sudo firewall-cmd --permanent --add-port=2222/tcp
$ sudo firewall-cmd --reload

# Or create custom service
$ sudo nano /etc/firewalld/services/ssh-custom.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>SSH Custom Port</short>
  <description>SSH on port 2222</description>
  <port protocol="tcp" port="2222"/>
</service>
$ sudo firewall-cmd --permanent --add-service=ssh-custom
$ sudo firewall-cmd --reload

SSH Security Checklist

• ✅ Use SSH keys only - Disable password authentication
• ✅ Change default port - Reduce automated attacks
• ✅ Disable root login - Use sudo instead
• ✅ Use strong cryptography - Ed25519 keys, modern ciphers
• ✅ Implement fail2ban - Block brute force attacks
• ✅ Restrict users/groups - Only allow necessary users
• ✅ Use firewall rules - Limit source IPs if possible
• ✅ Regular updates - Keep SSH server updated
• ✅ Monitor logs - Watch for suspicious activity
• ✅ Test backup access - Ensure you don't lock yourself out

4. Security Logs & Monitoring

Why Logging is Critical for Security

Security Logging: The practice of recording security-relevant events to provide an audit trail for detecting, investigating, and responding to security incidents. Logs are essential for forensic analysis, compliance, and proactive threat detection.

Linux Security Log Locations

Debian/Ubuntu Logs

# Authentication logs
/var/log/auth.log
# All authentication attempts (SSH, sudo, login)

# System logs
/var/log/syslog
/var/log/kern.log
# System and kernel messages

# Application logs
/var/log/apache2/
/var/log/nginx/
/var/log/mysql/

# Failed login attempts
/var/log/faillog
/var/log/btmp

RHEL/CentOS Logs

# Authentication logs
/var/log/secure
# Equivalent to auth.log on Debian

# System logs
/var/log/messages
/var/log/audit/audit.log
# System messages and audit logs

# Authentication databases
/var/log/lastlog
/var/log/wtmp
# Last login times and login records

# Cron logs
/var/log/cron

Key Security Log Files Explained

/var/log/auth.log (Debian/Ubuntu)

# Sample entries:
Dec 1 10:00:01 server sshd[1234]: Accepted publickey for alice from 192.168.1.100 port 2222
Dec 1 10:01:23 server sudo: alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/bin/ls
Dec 1 10:02:45 server sshd[5678]: Failed password for invalid user hacker from 10.0.0.1 port 1234
Dec 1 10:03:12 server sshd[5678]: Connection closed by invalid user hacker 10.0.0.1 port 1234

# Important fields:
Timestamp: When event occurred
Hostname: Which system
Service: sshd, sudo, pam, etc.
PID: Process ID
Message: What happened

/var/log/secure (RHEL/CentOS)

# Similar to auth.log but on RHEL systems
Dec 1 10:00:01 server sshd[1234]: Accepted publickey for alice from 192.168.1.100
Dec 1 10:01:23 server sudo: alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/bin/ls
Dec 1 10:02:45 server sshd[5678]: Failed password for invalid user hacker from 10.0.0.1

# Additional RHEL-specific entries:
# SELinux denials, PAM errors, authentication failures
Dec 1 10:04:56 server pam_unix(sudo:auth): authentication failure; logname=uid=0 euid=0 tty=/dev/pts/0 ruser= rhost= user=alice
Dec 1 10:05:12 server sshd[5678]: error: PAM: Authentication failure for hacker from 10.0.0.1

Authentication Databases

# /var/log/lastlog - Last login times
$ lastlog
Username          Port       From                 Latest
alice          pts/0    192.168.1.100      Mon Dec 1 10:00:01 2023
bob            pts/1    192.168.1.101      Mon Dec 1 09:30:22 2023

# /var/log/wtmp - Login records (binary)
$ last
alice        pts/0        192.168.1.100   Mon Dec 1 10:00  still logged in
bob          pts/1        192.168.1.101   Mon Dec 1 09:30 - 09:45  (00:15)

# /var/log/btmp - Failed login attempts (binary)
$ lastb
hacker      ssh:notty     10.0.0.1        Mon Dec 1 10:02 - 10:02  (00:00)
hacker      ssh:notty     10.0.0.1        Mon Dec 1 10:02 - 10:02  (00:00)

Log Analysis Tools & Commands

Basic Log Viewing

# View logs in real-time
$ sudo tail -f /var/log/auth.log
$ sudo tail -f /var/log/secure

# View specific number of lines
$ tail -100 /var/log/auth.log
$ head -50 /var/log/syslog

# Search for specific patterns
$ grep "Failed password" /var/log/auth.log
$ grep "Accepted publickey" /var/log/secure
$ grep -i "error" /var/log/syslog

Advanced Log Analysis

# Count failed SSH attempts
$ grep "Failed password" /var/log/auth.log | wc -l

# Show unique IPs with failed attempts
$ grep "Failed password" /var/log/auth.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort | uniq

# Show successful logins with timestamps
$ grep "Accepted" /var/log/auth.log | awk '{print $1" "$2" "$3" - "$9" from "$11}'

# Monitor sudo usage
$ grep "sudo:" /var/log/auth.log | grep "COMMAND"

Log Management Tools

# journalctl (systemd systems)
$ sudo journalctl -u sshd
$ sudo journalctl --since "2023-12-01 00:00:00" --until "2023-12-01 23:59:59"
$ sudo journalctl -f # Follow logs

# logwatch - Daily log analysis
$ sudo apt install logwatch
$ logwatch --range yesterday
$ logwatch --detail high

# goaccess - Web log analyzer
$ goaccess /var/log/nginx/access.log

Security Log Monitoring Script

#!/bin/bash
# security-log-monitor.sh
# Monitors security logs for suspicious activity

LOGFILE="/var/log/auth.log"
REPORTFILE="/tmp/security-report-$(date +%Y%m%d).txt"

echo "=== SECURITY LOG REPORT $(date) ===" > $REPORTFILE
echo "" >> $REPORTFILE

# 1. Failed SSH attempts
echo "=== FAILED SSH ATTEMPTS ===" >> $REPORTFILE
grep "Failed password" $LOGFILE | tail -20 >> $REPORTFILE
echo "" >> $REPORTFILE

# 2. Successful SSH logins
echo "=== SUCCESSFUL SSH LOGINS ===" >> $REPORTFILE
grep "Accepted" $LOGFILE | tail -10 >> $REPORTFILE
echo "" >> $REPORTFILE

# 3. Sudo usage
echo "=== SUDO COMMANDS ===" >> $REPORTFILE
grep "sudo:" $LOGFILE | grep "COMMAND" | tail -10 >> $REPORTFILE
echo "" >> $REPORTFILE

# 4. Invalid users
echo "=== INVALID USER ATTEMPTS ===" >> $REPORTFILE
grep "invalid user" $LOGFILE | tail -10 >> $REPORTFILE
echo "" >> $REPORTFILE

# 5. Statistics
echo "=== STATISTICS ===" >> $REPORTFILE
echo "Failed SSH attempts: $(grep -c "Failed password" $LOGFILE)" >> $REPORTFILE
echo "Successful SSH logins: $(grep -c "Accepted" $LOGFILE)" >> $REPORTFILE
echo "Sudo commands: $(grep -c "sudo:" $LOGFILE | grep "COMMAND")" >> $REPORTFILE
echo "Invalid users attempted: $(grep -c "invalid user" $LOGFILE)" >> $REPORTFILE

# Send email alert if too many failures
FAILED_COUNT=$(grep "Failed password" $LOGFILE | wc -l)
if [ $FAILED_COUNT -gt 10 ]; then
    echo "High number of failed login attempts: $FAILED_COUNT" | mail -s "Security Alert" admin@example.com
fi

# Run with cron: */30 * * * * /path/to/security-log-monitor.sh

Log Management Best Practices

• Centralized logging: Use rsyslog or syslog-ng to send logs to central server
• Log rotation: Configure logrotate to manage log file sizes
• Retention policies: Keep logs as long as needed for compliance
• Regular review: Automate log analysis and alerting
• Secure storage: Protect logs from tampering (immutable logs)
• Monitor log integrity: Use tools like auditd to watch for log tampering
• Backup logs: Include logs in your backup strategy
• Compliance alignment: Ensure logs meet regulatory requirements

5. SELinux & AppArmor Basics

What is Mandatory Access Control (MAC)?

Mandatory Access Control (MAC): A security model where access decisions are made by a central authority (security policy) rather than by individual users. In Linux, SELinux and AppArmor implement MAC to provide an additional layer of security beyond traditional discretionary access controls (DAC).

Discretionary Access Control (DAC)

• Control: Owners decide permissions
• Examples: Unix permissions (rwx)
• Limitation: Users can make bad decisions
• Problem: If attacker gains user access, they get user's privileges

# DAC example:
-rwxr-xr-x 1 alice alice script.sh
# Alice can change permissions however she wants
# If script is malicious, it runs with Alice's privileges

Mandatory Access Control (MAC)

• Control: System policy decides permissions
• Examples: SELinux, AppArmor
• Advantage: Policy overrides user decisions
• Protection: Even if attacker gains access, policy restricts what they can do

# MAC example:
# Policy says: httpd_t can only read files labeled httpd_content_t
# Even if file has 777 permissions, policy blocks access
# Even if attacker compromises httpd, they're limited by policy

SELinux (Security-Enhanced Linux)

SELinux: A Linux kernel security module that provides a mechanism for supporting access control security policies, including mandatory access controls. It uses labels (contexts) on all subjects (processes) and objects (files, ports, etc.) to make access decisions.

SELinux Key Concepts

Contexts (Labels)

# View file context
$ ls -Z /etc/passwd
-rw-r--r--. root root system_u:object_r:passwd_file_t:s0 /etc/passwd
# Format: user:role:type:level

# View process context
$ ps -Z $(pgrep httpd)
system_u:system_r:httpd_t:s0 1234 httpd
# httpd process runs with httpd_t type

# Types are most important for access control
# Policy rules define which types can access which other types

SELinux Modes

# Check SELinux status
$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed

# Three modes:
# 1. Enforcing: Policy is active, violations are denied
# 2. Permissive: Policy is checked but not enforced (logs only)
# 3. Disabled: SELinux is turned off

# Change mode temporarily
$ sudo setenforce 0 # Permissive
$ sudo setenforce 1 # Enforcing

SELinux Troubleshooting

# Check SELinux denials
$ sudo ausearch -m avc -ts recent
# or
$ sudo sealert -a /var/log/audit/audit.log

# Temporarily allow access for troubleshooting
$ sudo setsebool -P httpd_can_network_connect on

# Change file context
$ sudo chcon -t httpd_sys_content_t /var/www/html/index.html
$ sudo restorecon -v /var/www/html/index.html # Restore default

# Generate custom policy module
$ sudo audit2allow -a -M mypolicy
$ sudo semodule -i mypolicy.pp

AppArmor (Application Armor)

AppArmor: A Linux kernel security module that allows the system administrator to restrict programs' capabilities with per-program profiles. It's path-based (rather than label-based like SELinux) and is the default MAC on Ubuntu and openSUSE.

AppArmor Key Concepts

Profiles

# Check AppArmor status
$ sudo apparmor_status
apparmor module is loaded.
20 profiles are loaded.
20 profiles are in enforce mode.
0 profiles are in complain mode.

# Profile locations
/etc/apparmor.d/ # System profiles
/etc/apparmor.d/disable/ # Disabled profiles

# Profile example (nginx):
profile nginx /usr/sbin/nginx {
  # Network access
  network inet tcp,
  network inet6 tcp,

  # File access
  /etc/nginx/** r,
  /var/log/nginx/** w,
  /var/www/html/** r,
}

Profile Modes

# Two modes for each profile:
# 1. Enforce: Policy is active, violations are blocked
# 2. Complain: Policy violations are logged but allowed

# Change profile mode
$ sudo aa-complain /usr/sbin/nginx
$ sudo aa-enforce /usr/sbin/nginx

# Disable profile
$ sudo ln -s /etc/apparmor.d/usr.sbin.nginx /etc/apparmor.d/disable/
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.nginx

# Reload all profiles
$ sudo systemctl reload apparmor

AppArmor Troubleshooting

# Check AppArmor logs
$ sudo dmesg | grep apparmor
$ sudo journalctl -u apparmor --since "1 hour ago"

# View denied messages
$ sudo aa-notify
$ sudo grep DENIED /var/log/kern.log

# Generate profile from logs
$ sudo aa-logprof
# Interactive tool to create/update profiles

# Audit a program
$ sudo aa-genprof /path/to/program
# Run program, then in another terminal:
$ sudo aa-logprof

SELinux vs AppArmor

Feature	SELinux	AppArmor
Approach	Label-based (type enforcement)	Path-based (profiles)
Complexity	High learning curve	Easier to understand
Default on	RHEL, CentOS, Fedora	Ubuntu, openSUSE
Policy creation	Complex (te files)	Simpler (plain text)
Granularity	Very fine-grained	Coarser but simpler
Best for	High-security environments	General purpose, easier management

When to Use Which

• Use SELinux if:
  • You need maximum security
  • You're in RHEL/CentOS environment
  • You have experienced security team
  • You need multi-level security (MLS)
• Use AppArmor if:
  • You want easier management
  • You're in Ubuntu/openSUSE environment
  • You need to secure specific applications
  • You want path-based simplicity
• Consider neither if:
  • You have minimal security requirements
  • Resources for management are limited
  • Compatibility issues with legacy apps

Securing Nginx Web Server with AppArmor

# Step 1: Check if AppArmor is enabled
$ sudo apparmor_status
$ sudo systemctl status apparmor

# Step 2: Check existing nginx profile
$ ls -l /etc/apparmor.d/usr.sbin.nginx
$ sudo aa-status | grep nginx

# Step 3: Put nginx in complain mode for learning
$ sudo aa-complain /usr/sbin/nginx

# Step 4: Generate profile from nginx usage
$ sudo aa-genprof /usr/sbin/nginx
# Now use nginx normally (access web pages, etc.)
# In the aa-genprof terminal, press 'S' to scan logs
# Answer questions about allowing/denying accesses

# Step 5: Review generated profile
$ sudo cat /etc/apparmor.d/usr.sbin.nginx
# Profile will include allowed paths, network access, etc.

# Step 6: Put nginx in enforce mode
$ sudo aa-enforce /usr/sbin/nginx

# Step 7: Test nginx functionality
$ curl http://localhost
# Should work if profile is correct

# Step 8: Monitor for denials
$ sudo tail -f /var/log/kern.log | grep DENIED
$ sudo aa-notify

# Step 9: Update profile if needed
$ sudo aa-logprof
# Will suggest updates based on recent denials

MAC Implementation Best Practices

• Start in complain/permissive mode: Learn what applications need before enforcing
• Use existing profiles: Many applications have pre-written profiles
• Monitor logs regularly: Check for denials and adjust policies
• Principle of least privilege: Only allow what's absolutely necessary
• Test thoroughly: Ensure policies don't break legitimate functionality
• Document policies: Keep records of why certain accesses are allowed
• Regular reviews: Update policies as applications change
• Backup policies: Include MAC policies in your configuration management
• Train your team: Ensure everyone understands how MAC works
• Consider automation: Use tools to generate and manage policies

Security Automation for DevOps

Automated Security Hardening with Ansible

# Ansible playbook for Linux security hardening
---
- name: Harden Linux server security
hosts: all
become: yes
tasks:
- name: Install security tools
package:
name:
- fail2ban
- unattended-upgrades
- logwatch
- aide
- rkhunter
state: present

- name: Configure SSH hardening
template:
src: sshd_config.j2
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: "0600"
notify: Restart SSH

- name: Set secure permissions on sensitive files
file:
path: "{{ item.path }}"
owner: "{{ item.owner | default('root') }}"
group: "{{ item.group | default('root') }}"
mode: "{{ item.mode }}"
loop:
- { path: "/etc/passwd", mode: "0644" }
- { path: "/etc/shadow", mode: "0000" }
- { path: "/etc/group", mode: "0644" }
- { path: "/etc/gshadow", mode: "0000" }
- { path: "/etc/sudoers", mode: "0440" }
- { path: "/etc/ssh/sshd_config", mode: "0600" }

- name: Configure UFW firewall
ufw:
state: enabled
policy: deny
direction: incoming
logging: on

- name: Allow necessary ports
ufw:
rule: allow
port: "{{ item }}"
proto: tcp
loop:
- "2222" # SSH custom port
- "80" # HTTP
- "443" # HTTPS

- name: Configure fail2ban for SSH
copy:
dest: /etc/fail2ban/jail.local
content: |
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
owner: root
group: root
mode: "0644"
notify: Restart fail2ban

- name: Configure automatic security updates
copy:
dest: /etc/apt/apt.conf.d/50unattended-upgrades
content: |
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}";
"${distro_id}:${distro_codename}-security";
"${distro_id}ESM:${distro_codename}";
};
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
owner: root
group: root
mode: "0644"
when: ansible_os_family == "Debian"

- name: Disable unnecessary services
systemd:
name: "{{ item }}"
state: stopped
enabled: no
loop:
- telnet.socket
- rlogin.socket
- rexec.socket
- rsh.socket
- daytime.socket
- time.socket
- echo.socket
- discard.socket
- chargen.socket

- name: Configure sysctl security settings
sysctl:
name: "{{ item.name }}"
value: "{{ item.value }}"
sysctl_set: yes
state: present
reload: yes
loop:
- { name: net.ipv4.ip_forward, value: 0 }
- { name: net.ipv4.conf.all.send_redirects, value: 0 }
- { name: net.ipv4.conf.default.send_redirects, value: 0 }
- { name: net.ipv4.conf.all.accept_source_route, value: 0 }
- { name: net.ipv4.conf.default.accept_source_route, value: 0 }
- { name: net.ipv4.conf.all.accept_redirects, value: 0 }
- { name: net.ipv4.conf.default.accept_redirects, value: 0 }
- { name: net.ipv4.conf.all.secure_redirects, value: 0 }
- { name: net.ipv4.conf.default.secure_redirects, value: 0 }
- { name: net.ipv4.conf.all.log_martians, value: 1 }
- { name: net.ipv4.conf.default.log_martians, value: 1 }
- { name: net.ipv4.icmp_echo_ignore_broadcasts, value: 1 }
- { name: net.ipv4.icmp_ignore_bogus_error_responses, value: 1 }
- { name: net.ipv4.conf.all.rp_filter, value: 1 }
- { name: net.ipv4.conf.default.rp_filter, value: 1 }
- { name: net.ipv4.tcp_syncookies, value: 1 }
- { name: net.ipv6.conf.all.accept_redirects, value: 0 }
- { name: net.ipv6.conf.default.accept_redirects, value: 0 }
- { name: net.ipv6.conf.all.accept_source_route, value: 0 }

- name: Install and configure AIDE (file integrity checker)
block:
- name: Initialize AIDE database
command: aideinit
args:
creates: /var/lib/aide/aide.db.new

- name: Install cron job for daily AIDE checks
cron:
name: "Daily AIDE check"
minute: "0"
hour: "2"
job: "/usr/bin/aide --check"

- name: Configure auditd for security monitoring
copy:
dest: /etc/audit/audit.rules
content: |
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change
-a always,exit -F arch=b64 -S clock_settime -k time-change
-a always,exit -F arch=b32 -S clock_settime -k time-change
-w /etc/localtime -p wa -k time-change
-w /etc/group -p wa -k identity
-w /etc/passwd -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/security/opasswd -p wa -k identity
-w /etc/sudoers -p wa -k scope
-w /etc/sudoers.d/ -p wa -k scope
-w /var/log/auth.log -p wa -k identity
-w /var/log/faillog -p wa -k logins
-w /var/log/lastlog -p wa -k logins
-w /var/log/tallylog -p wa -k logins
-w /var/run/faillock -p wa -k logins
-w /etc/ssh/sshd_config -p wa -k sshd
owner: root
group: root
mode: "0640"
notify: Restart auditd

handlers:
- name: Restart SSH
service:
name: sshd
state: restarted

- name: Restart fail2ban
service:
name: fail2ban
state: restarted

- name: Restart auditd
service:
name: auditd
state: restarted

Security Command Cheat Sheet

Permission Management

$ ls -l # View permissions
$ chmod 755 file # Change permissions (numeric)
$ chmod u+x file # Add execute for owner
$ sudo chown user:group file # Change ownership
$ umask 022 # Set default permissions mask

Special Permission Bits

$ chmod u+s file # Set SUID
$ chmod g+s dir # Set SGID on directory
$ chmod +t dir # Set sticky bit
$ find / -perm -4000 2>/dev/null # Find SUID files
$ find / -perm -2000 2>/dev/null # Find SGID files

SSH Security

$ sudo nano /etc/ssh/sshd_config # SSH server config
$ sudo sshd -t # Test SSH config
$ ssh-keygen -t ed25519 # Generate SSH key
$ ssh-copy-id user@host # Copy SSH key to server
$ sudo systemctl restart sshd # Restart SSH service

Security Logs

$ sudo tail -f /var/log/auth.log # Monitor auth logs (Debian)
$ sudo tail -f /var/log/secure # Monitor auth logs (RHEL)
$ last # Show recent logins
$ lastb # Show failed login attempts
$ sudo journalctl -u sshd # View SSH logs (systemd)

SELinux & AppArmor

$ sestatus # Check SELinux status
$ getenforce # Get SELinux mode
$ sudo setenforce 0 # Set SELinux to permissive
$ sudo apparmor_status # Check AppArmor status
$ sudo aa-complain /usr/sbin/nginx # Put in complain mode

Practice Scenarios for DevOps Security

1. A web server has been compromised. How would you investigate using logs and what steps would you take to secure it?
2. You need to implement least privilege for a database backup script that requires root access. What methods would you consider and which is most secure?
3. How would you detect and respond to a brute force SSH attack in real-time?
4. Compare and contrast using SUID vs sudo vs capabilities for granting specific privileges to applications.
5. You're deploying a new web application. What security considerations would you address at each layer (network, OS, application)?
6. How would you automate security compliance checking across 100+ servers?
7. A developer needs write access to log files in /var/log/app/. What's the most secure way to grant this?
8. What would you include in a Linux server security baseline configuration?
9. How would you implement file integrity monitoring for critical system files?
10. Compare SELinux and AppArmor for containerized environments. Which would you choose and why?

Key Takeaways

• Defense in depth: Implement security at multiple layers
• Principle of least privilege: Grant minimum necessary access
Regular audits: Continuously review permissions and configurations
• Secure by default: Start with restrictive settings, then relax as needed
• Monitor everything: Logs are essential for detection and forensics
• Automate security: Use configuration management for consistent security
• Stay updated: Regular patching is critical for security
• Understand your tools: Know how permissions, firewalls, and MAC systems work
• Test security controls: Regularly verify that security measures are working
• Security is a process: Continuous improvement, not a one-time setup

Mastering Linux security is essential for protecting infrastructure, data, and services in DevOps environments. These skills enable you to build secure systems, detect and respond to threats, and maintain compliance with security standards. Remember: security is not a product but a process that requires continuous attention and improvement.