Advanced DevOps Topics Guide - Environment Variables, SSH, Remote Management

Advanced Topics for DevOps: Mastering the Professional Toolbox

Elevate your DevOps skills with these advanced techniques used by professional system administrators.

📅 Published: Feb 2026
⏱️ Estimated Reading Time: 20 minutes
🏷️ Tags: Advanced DevOps, SSH Keys, Environment Variables, Rsync, Tmux, Configuration Management

🌍 Environment Variables & Profiles

What are Environment Variables?

Think of environment variables as post-it notes on your computer's fridge. They store important information that programs can read whenever they need it. These variables contain settings that affect how your shell and applications behave.

Key Environment Variables

# View all environment variables
printenv
# or
env

# View specific variable
echo $PATH
echo $HOME
echo $USER
echo $SHELL

# Common useful variables:
PATH        # Where to look for commands
HOME        # Your home directory
USER        # Your username
SHELL       # Your default shell
PWD         # Current directory
OLDPWD      # Previous directory
PS1         # Command prompt appearance
EDITOR      # Default text editor
LANG        # Language settings

Setting Environment Variables

# Set variable for current session
export MY_VAR="value"

# Set multiple variables
export NAME="Alice" AGE=30 ROLE="DevOps"

# Check if set
echo $MY_VAR

# Use in commands
export BACKUP_DIR="/backup/$(date +%Y%m%d)"
mkdir -p $BACKUP_DIR

# Append to existing variable (like PATH)
export PATH="$PATH:/usr/local/bin"

# Remove/unset variable
unset MY_VAR

Understanding the Startup Files

When you log into Linux, it reads configuration files in this order:

/etc/profile - System-wide settings
~/.bash_profile or ~/.profile - User settings
~/.bashrc - User shell settings

Non-login Shells (new terminal window):

~/.bashrc - User shell settings

The Configuration Files Explained

~/.bashrc - Your Shell Customization

This file runs every time you open a new terminal.

# Common .bashrc additions
nano ~/.bashrc

# Add these useful aliases:
alias ll='ls -la'
alias la='ls -A'
alias l='ls -CF'
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'

# History settings
export HISTSIZE=10000
export HISTFILESIZE=20000
export HISTCONTROL=ignoreboth:erasedups
shopt -s histappend

# Prompt customization
export PS1='\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

# Safety aliases
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Custom functions
mkcd() {
    mkdir -p "$1" && cd "$1"
}

# Source other files
if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# Load completion
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
    . /etc/bash_completion
fi

~/.profile or ~/.bash_profile - Login Settings

This runs only when you first log in.

# Common .profile additions
nano ~/.profile

# Set default editor
export EDITOR=nano
export VISUAL=nano

# Language settings
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

# Add to PATH
export PATH="$HOME/bin:$PATH"
export PATH="$HOME/.local/bin:$PATH"

# Application-specific settings
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
export PATH="$JAVA_HOME/bin:$PATH"

export NODE_PATH=/usr/lib/node_modules

# SSH agent
if [ -z "$SSH_AUTH_SOCK" ] ; then
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/id_rsa
fi

# Source .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
fi

/etc/profile - System-wide Settings

Affects all users on the system.

# System administrators add here:
# Default PATH for all users
# System-wide aliases
# Security settings
# Default umask

Best Practices for Environment Management

# 1. Keep it organized
# Create separate files
~/.bash_aliases    # For aliases
~/.bash_functions  # For functions
~/.bash_exports    # For exports

# 2. Use conditionals
if [ -f /usr/bin/nvim ]; then
    export EDITOR=nvim
else
    export EDITOR=vim
fi

# 3. Keep sensitive data out
# DON'T DO THIS:
export DB_PASSWORD="secret123"  # Visible to all processes!

# Instead, use:
# - Application-specific config files
# - Environment files loaded at runtime
# - Secret management tools (Vault, AWS Secrets Manager)

# 4. Test changes
# After editing .bashrc, reload:
source ~/.bashrc
# or
. ~/.bashrc

# 5. Debug startup
# To see what's being sourced:
bash -x  # Start bash with debugging

Application-specific Environment Files

# Create environment file for your app
nano ~/.myapp.env

# Add variables
export APP_ENV=production
export APP_DEBUG=false
export DATABASE_URL="mysql://user:pass@localhost/db"

# Load it when needed
source ~/.myapp.env

# Or create a script that loads it
nano start-app.sh
#!/bin/bash
source ~/.myapp.env
python app.py

🔑 SSH Keys & Key-based Authentication

Why SSH Keys are Essential

SSH keys are like physical keys to your server. Instead of passwords (which can be guessed or stolen), you use cryptographic keys that are nearly impossible to crack.

Benefits:

More secure than passwords
Can be disabled instantly
Allows automation (scripts, CI/CD)
No typing passwords repeatedly

Generating SSH Key Pairs

# Generate RSA key (4096 bits - secure)
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/id_rsa

# Generate Ed25519 key (modern, more secure)
ssh-keygen -t ed25519 -C "work-laptop" -f ~/.ssh/id_ed25519

# Generate without passphrase (for automation)
ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/deploy_key

# During generation you'll be asked:
# 1. File to save key (press Enter for default)
# 2. Passphrase (recommended for extra security)
# 3. Confirm passphrase

# Keys are saved as:
# Private key: ~/.ssh/id_rsa       (KEEP THIS SECRET!)
# Public key:  ~/.ssh/id_rsa.pub   (Share this)

Managing SSH Keys

# List your keys
ls -la ~/.ssh/

# View public key
cat ~/.ssh/id_rsa.pub

# View fingerprint (unique identifier)
ssh-keygen -l -f ~/.ssh/id_rsa.pub

# Change passphrase (if you set one)
ssh-keygen -p -f ~/.ssh/id_rsa

# Copy key to server (easiest way)
ssh-copy-id -i ~/.ssh/id_rsa.pub user@server-ip

# Manual copy (if ssh-copy-id not available)
cat ~/.ssh/id_rsa.pub | ssh user@server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

# Multiple keys? Use ssh-agent
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
ssh-add ~/.ssh/id_ed25519

# List loaded keys
ssh-add -l

# Remove specific key
ssh-add -d ~/.ssh/id_rsa

# Remove all keys
ssh-add -D

SSH Configuration File

Create ~/.ssh/config for easier connections:

nano ~/.ssh/config

Example configuration:

# Global settings (apply to all hosts)
Host *
    # Security settings
    ServerAliveInterval 60
    ServerAliveCountMax 5
    TCPKeepAlive yes
    Compression yes
    
    # Key management
    AddKeysToAgent yes
    UseKeychain yes  # macOS only
    
    # Security enhancements
    PubkeyAuthentication yes
    PasswordAuthentication no
    
    # Timeouts
    ConnectTimeout 10

# Production server
Host prod
    HostName 192.168.1.100
    User deploy
    Port 22
    IdentityFile ~/.ssh/deploy_key
    ForwardAgent yes

# Development server
Host dev
    HostName dev.example.com
    User developer
    Port 2222
    IdentityFile ~/.ssh/id_ed25519

# Jump host/bastion configuration
Host internal-*
    ProxyJump bastion
    User admin

Host bastion
    HostName bastion.example.com
    User jumper
    IdentityFile ~/.ssh/bastion_key

Host internal-web1
    HostName 10.0.1.10

Host internal-db1
    HostName 10.0.1.20

# GitHub specific
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_key
    IdentitiesOnly yes

Now connect easily:

ssh prod      # Instead of ssh deploy@192.168.1.100
ssh dev       # Instead of ssh -p 2222 developer@dev.example.com
ssh internal-web1  # Automatically jumps through bastion

SSH Security Best Practices

# On the SERVER, harden SSH (edit /etc/ssh/sshd_config):
sudo nano /etc/ssh/sshd_config

# Critical security settings:
Port 2222                      # Change from default 22
PermitRootLogin no             # Never allow root SSH
PasswordAuthentication no      # Keys only
PubkeyAuthentication yes
MaxAuthTries 3                 # Limit login attempts
ClientAliveInterval 300        # Disconnect idle after 5 min
ClientAliveCountMax 2
AllowUsers deploy monitor      # Only specific users
#AllowGroups ssh-users         # Or specific groups

# Restart SSH after changes
sudo systemctl restart ssh

# Keep old session open while testing new connection!

SSH Key Rotation (Important for Security)

# 1. Generate new key pair
ssh-keygen -t ed25519 -f ~/.ssh/new_key -C "rotated-$(date +%Y%m%d)"

# 2. Copy to server
ssh-copy-id -i ~/.ssh/new_key.pub user@server

# 3. Test new key
ssh -i ~/.ssh/new_key user@server

# 4. Update SSH config
# In ~/.ssh/config, change IdentityFile to new key

# 5. Remove old key from server
# On server, edit ~/.ssh/authorized_keys and remove old key

# 6. Delete old local key (optional)
shred -u ~/.ssh/old_key ~/.ssh/old_key.pub

🌐 Managing Remote Servers

The Multi-Server Mindset

When managing multiple servers, think like an air traffic controller. You need visibility and control over all systems simultaneously.

Essential Remote Management Commands

# Execute command on remote server
ssh user@server "ls -la /var/log"

# Multiple commands
ssh user@server "cd /tmp && pwd && ls"

# With sudo on remote
ssh -t user@server "sudo systemctl restart nginx"

# Copy command output from remote
ssh user@server "cat /etc/hostname" > local_file.txt

# Compare files between local and remote
diff <(cat local_file) <(ssh user@server "cat /remote/file")

# Run local script on remote
cat local_script.sh | ssh user@server "bash -s"

# Execute on multiple servers
for server in server1 server2 server3; do
    ssh user@$server "hostname; uptime"
done

SSH ControlMaster: Speed Up Multiple Connections

# Add to ~/.ssh/config
Host *
    ControlMaster auto
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlPersist 10m

# First connection opens master
ssh server1

# Subsequent connections reuse the connection (much faster)
ssh server1  # Fast!
ssh server1 "uptime"  # Also fast!

# Check open connections
ssh -O check server1

# Exit master connection
ssh -O exit server1

Managing Multiple Servers with Parallel SSH

# Install parallel SSH tools
sudo apt install pssh  # Ubuntu/Debian
sudo yum install pssh  # Red Hat/CentOS

# Create server list file
cat > servers.txt << EOF
user@server1:22
user@server2:2222
user@server3
EOF

# Run command on all servers
parallel-ssh -h servers.txt -i "uptime"

# Copy files to all servers
parallel-scp -h servers.txt local_file.txt /remote/path/

# Gather files from all servers
parallel-slurp -h servers.txt -L /local/dir /remote/file local_name

# Alternative: use clustershell
sudo apt install clustershell

# Define groups in /etc/clustershell/groups
echo "web: server1 server2" | sudo tee -a /etc/clustershell/groups
echo "db: server3" | sudo tee -a /etc/clustershell/groups
echo "all: @web @db" | sudo tee -a /etc/clustershell/groups

# Execute on group
clush -g web "systemctl status nginx"
clush -g all "df -h /"

Remote Monitoring Dashboard

#!/bin/bash
# remote-monitor.sh

SERVERS=("web1.example.com" "web2.example.com" "db1.example.com" "db2.example.com")

echo "=== Remote Server Dashboard ==="
echo "Time: $(date)"
echo

for server in "${SERVERS[@]}"; do
    echo "--- $server ---"
    
    # Use timeout to prevent hanging
    if timeout 5 ssh $server "echo -n 'Status: ' && \
        hostname && \
        echo -n 'Uptime: ' && uptime -p && \
        echo -n 'Load: ' && uptime | awk -F'load average:' '{print \$2}' && \
        echo -n 'Memory: ' && free -h | awk '/^Mem:/ {print \$3\"/\"\$2}' && \
        echo -n 'Disk: ' && df -h / | awk 'NR==2 {print \$5}'" 2>/dev/null; then
        echo "✓ Connected"
    else
        echo "✗ Connection failed"
    fi
    
    echo
done

Automated Server Provisioning Script

#!/bin/bash
# provision-server.sh

SERVER_IP=$1
SERVER_USER=$2

if [ -z "$SERVER_IP" ] || [ -z "$SERVER_USER" ]; then
    echo "Usage: $0 <server_ip> <username>"
    exit 1
fi

echo "Provisioning $SERVER_IP as $SERVER_USER..."
echo

# 1. Copy SSH key
ssh-copy-id $SERVER_USER@$SERVER_IP

# 2. Basic setup commands
ssh -t $SERVER_USER@$SERVER_IP << 'EOF'
# Update system
sudo apt update && sudo apt upgrade -y

# Install essential packages
sudo apt install -y \
    curl wget git vim htop \
    net-tools ufw fail2ban \
    nginx mysql-server \
    python3 python3-pip

# Configure firewall
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw --force enable

# Create deploy user
sudo useradd -m -s /bin/bash deploy
sudo usermod -aG sudo deploy
echo "deploy:$(openssl rand -base64 12)" | sudo chpasswd
sudo chage -d 0 deploy

# Secure SSH
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

# Set timezone
sudo timedatectl set-timezone UTC

# Configure automatic updates
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

echo "Provisioning complete!"
EOF

echo
echo "Server $SERVER_IP provisioned successfully."
echo "Next steps:"
echo "1. SSH to server: ssh $SERVER_USER@$SERVER_IP"
echo "2. Test deployment: ssh deploy@$SERVER_IP"
echo "3. Deploy your application"

📁 File Transfer Automation

scp vs rsync: When to Use Which

scp = Simple copy (like cp over SSH)
rsync = Smart sync (copies only changes, resumes interrupted transfers)

scp: Simple Secure Copy

# Copy local file to remote
scp file.txt user@server:/remote/path/

# Copy remote file to local
scp user@server:/remote/file.txt /local/path/

# Copy directory recursively
scp -r /local/dir/ user@server:/remote/dir/

# Preserve permissions and timestamps
scp -p file.txt user@server:/remote/

# Use specific port
scp -P 2222 file.txt user@server:/remote/

# Use specific SSH key
scp -i ~/.ssh/key.pem file.txt user@server:/remote/

# Copy between two remote servers (via local)
scp -3 user1@server1:/file user2@server2:/remote/

# Limit bandwidth (KB/s)
scp -l 1000 largefile.iso user@server:/remote/  # 1000 KB/s = ~1 MB/s

rsync: The Professional's Choice

# Basic sync (copies only differences)
rsync -avz /local/dir/ user@server:/remote/dir/

# Flags explained:
# -a = archive mode (preserves permissions, timestamps, etc.)
# -v = verbose (show what's happening)
# -z = compress during transfer
# -r = recursive (directories)
# -h = human readable sizes

# Dry run (see what would happen)
rsync -avzn /local/dir/ user@server:/remote/dir/

# Delete extra files on destination
rsync -avz --delete /local/dir/ user@server:/remote/dir/

# Exclude files/directories
rsync -avz --exclude='*.tmp' --exclude='cache/' /local/ user@server:/remote/

# Include/exclude from file
rsync -avz --exclude-from='/path/to/exclude-list.txt' /local/ user@server:/remote/

# Limit bandwidth (KB/s)
rsync -avz --bwlimit=1000 /local/ user@server:/remote/

# Resume partial transfers
rsync -avz --partial /local/largefile user@server:/remote/

# With progress indicator
rsync -avz --progress /local/ user@server:/remote/

# Preserve hard links
rsync -avz -H /local/ user@server:/remote/

# Use SSH with specific options
rsync -avz -e "ssh -p 2222 -i ~/.ssh/key.pem" /local/ user@server:/remote/

# Backup with timestamps
rsync -avz --backup --backup-dir=/backup/old-$(date +%Y%m%d) /source/ /dest/

# Sync with checksums (slower but verifies)
rsync -avzc /local/ user@server:/remote/

Real-world rsync Examples

# Website deployment
rsync -avz --delete --exclude='.git' \
    --exclude='.env' \
    --exclude='node_modules' \
    ./ user@server:/var/www/html/

# Database backup transfer
mysqldump -u root -p database | gzip > backup.sql.gz
rsync -avz --progress backup.sql.gz user@backup-server:/backups/

# Incremental backups
# Day 1: Full backup
rsync -avz /data/ /backup/full/

# Day 2: Incremental
rsync -avz --compare-dest=/backup/full/ /data/ /backup/inc-day2/

# Day 3: Another incremental
rsync -avz --compare-dest=/backup/full/ \
    --compare-dest=/backup/inc-day2/ \
    /data/ /backup/inc-day3/

# Mirror a directory with verification
rsync -avzc --delete /important-data/ user@mirror:/backup/

# Sync with remote deletion protection
rsync -avz --backup --suffix=.bak-$(date +%Y%m%d) /local/ user@server:/remote/

Automated Backup Script with rsync

#!/bin/bash
# automated-backup.sh

# Configuration
SOURCE_DIRS="/home /etc /var/www"
BACKUP_SERVER="backup@192.168.1.100"
BACKUP_DIR="/backup/$(hostname)"
RETENTION_DAYS=30
LOG_FILE="/var/log/backup.log"

# Log function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

log "Starting backup..."

# Create backup directory on remote
ssh $BACKUP_SERVER "mkdir -p $BACKUP_DIR"

# Backup each directory
for dir in $SOURCE_DIRS; do
    if [ -d "$dir" ]; then
        log "Backing up $dir..."
        
        rsync -avz --delete \
            --link-dest="$BACKUP_DIR/latest/$dir" \
            "$dir/" \
            "$BACKUP_SERVER:$BACKUP_DIR/incomplete/$dir"
            
        if [ $? -eq 0 ]; then
            log "✓ $dir backed up successfully"
        else
            log "✗ Failed to backup $dir"
        fi
    fi
done

# Move incomplete to latest
ssh $BACKUP_SERVER \
    "mv $BACKUP_DIR/incomplete $BACKUP_DIR/backup-$(date +%Y%m%d) && \
     ln -snf $BACKUP_DIR/backup-$(date +%Y%m%d) $BACKUP_DIR/latest"

# Clean old backups
ssh $BACKUP_SERVER \
    "find $BACKUP_DIR -name 'backup-*' -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;"

log "Backup completed successfully"

💻 Using screen and tmux

Why Use Terminal Multiplexers?

Think of screen and tmux as virtual workspaces for your terminal. They let you:

Run multiple sessions in one window
Disconnect and reconnect later (great for long-running tasks)
Share sessions with colleagues
Never lose work if connection drops

screen: The Classic Multiplexer

# Install screen
sudo apt install screen  # Ubuntu/Debian
sudo yum install screen  # Red Hat/CentOS

# Start new screen session
screen
screen -S session_name  # With custom name

# Inside screen:
# Ctrl+a then c     = Create new window
# Ctrl+a then n     = Next window
# Ctrl+a then p     = Previous window
# Ctrl+a then 0-9   = Switch to window number
# Ctrl+a then d     = Detach (leave session running)
# Ctrl+a then k     = Kill current window
# Ctrl+a then A     = Rename window
# Ctrl+a then "     = List windows
# Ctrl+a then S     = Split screen horizontally
# Ctrl+a then |     = Split screen vertically
# Ctrl+a then Tab   = Switch between splits
# Ctrl+a then X     = Close current split

# Manage sessions
screen -ls              # List all sessions
screen -r               # Reattach to last session
screen -r session_name  # Reattach to specific session
screen -x               # Attach to already attached session (share)

# Start screen with command
screen -dmS mysession command_to_run

# Send commands to screen session
screen -S mysession -X stuff 'ls\n'

# Log screen output
screen -L -S mysession  # Creates screenlog.0 file

tmux: The Modern Multiplexer

# Install tmux
sudo apt install tmux    # Ubuntu/Debian
sudo yum install tmux    # Red Hat/CentOS

# Start new tmux session
tmux
tmux new -s session_name  # With custom name

# Inside tmux:
# Ctrl+b then c     = Create new window
# Ctrl+b then n     = Next window
# Ctrl+b then p     = Previous window
# Ctrl+b then 0-9   = Switch to window number
# Ctrl+b then d     = Detach
# Ctrl+b then &     = Kill current window
# Ctrl+b then ,     = Rename window
# Ctrl+b then w     = List windows
# Ctrl+b then %     = Split vertically
# Ctrl+b then "     = Split horizontally
# Ctrl+b then arrow = Switch between panes
# Ctrl+b then x     = Close current pane
# Ctrl+b then z     | Zoom current pane
# Ctrl+b then [     = Enter scroll mode (q to exit)
# Ctrl+b then t     = Show clock

# Manage sessions
tmux ls                   # List sessions
tmux attach               # Attach to last session
tmux attach -t session_name
tmux kill-session -t session_name

# Send commands to tmux
tmux send-keys -t session_name 'ls' Enter

# Configuration file
nano ~/.tmux.conf

# Useful .tmux.conf settings:
# Enable mouse support
set -g mouse on

# Set prefix to Ctrl+a (like screen)
unbind C-b
set -g prefix C-a
bind C-a send-prefix

# Start windows numbering at 1
set -g base-index 1

# Reload config
bind r source-file ~/.tmux.conf \; display "Reloaded!"

# Status bar customization
set -g status-bg blue
set -g status-fg white

# Enable true color
set -g default-terminal "screen-256color"

Practical tmux/screen Usage

# 1. Long-running process
screen -S database_backup
# Run: mysqldump -u root -p database > backup.sql
# Detach: Ctrl+a then d
# Go home, have dinner, reconnect tomorrow:
screen -r database_backup

# 2. Multiple task organization
tmux new -s project
# Window 0: Code editor
# Window 1: Server logs (tail -f)
# Window 2: Database console
# Window 3: Git operations

# 3. Pair programming/teaching
# Person A starts:
tmux new -s pairing
# Person B connects:
ssh personB@machine
tmux attach -t pairing

# 4. Script automation with screen
#!/bin/bash
# run-with-screen.sh
screen -dmS data_processing python process_data.py
screen -dmS api_server node app.js
screen -dmS monitor ./monitor.sh

# Check all
screen -ls

# 5. tmux for deployment
tmux new -s deployment
# Split screen:
# Top: Run deployment script
# Bottom: Watch logs
# Right: Monitor system resources

⚙️ Config Management Preparation

What is Configuration Management?

Configuration Management is like having recipes for your entire infrastructure. Instead of manually setting up servers, you write code that describes how they should be configured.

Popular Tools

Ansible - Agentless, uses SSH (easiest to start)
Puppet - Master-agent architecture (mature)
Chef - Ruby-based, powerful (flexible)
SaltStack - Fast, event-driven (scalable)

Preparing for Ansible

# 1. Setup SSH key authentication to all servers
# (We covered this earlier)

# 2. Create inventory file
nano ~/ansible/inventory.ini

# inventory.ini example:
[web]
web1 ansible_host=192.168.1.10
web2 ansible_host=192.168.1.11

[db]
db1 ansible_host=192.168.1.20

[all:vars]
ansible_user=deploy
ansible_ssh_private_key_file=~/.ssh/deploy_key

# 3. Test connectivity
ansible all -i inventory.ini -m ping

# 4. Create ansible.cfg
nano ~/ansible/ansible.cfg

# ansible.cfg example:
[defaults]
inventory = ./inventory.ini
host_key_checking = False
retry_files_enabled = False
gathering = smart
fact_caching = memory

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
control_path = ~/.ssh/ansible-%%r@%%h:%%p

# 5. Basic playbook example
nano ~/ansible/setup-webserver.yml

setup-webserver.yml:

---
- name: Setup web servers
  hosts: web
  become: yes
  
  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600
    
    - name: Install nginx
      apt:
        name: nginx
        state: present
    
    - name: Start nginx
      service:
        name: nginx
        state: started
        enabled: yes
    
    - name: Copy website files
      copy:
        src: ../website/
        dest: /var/www/html/
        owner: www-data
        group: www-data
        mode: '0644'

Preparing for Puppet

# 1. Install Puppet on all nodes
# On each server:
curl -O https://apt.puppet.com/puppet7-release-focal.deb
sudo dpkg -i puppet7-release-focal.deb
sudo apt update
sudo apt install puppet-agent

# 2. Configure puppet.conf
sudo nano /etc/puppetlabs/puppet/puppet.conf

[main]
server = puppet-master.example.com
certname = web1.example.com
environment = production

# 3. First run
sudo /opt/puppetlabs/bin/puppet agent --test

# 4. On master, sign certificate
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname web1.example.com

Preparing for Chef

# 1. Install Chef Workstation (local)
# Download from: https://downloads.chef.io/products/workstation

# 2. Install Chef Infra Client on nodes
curl -L https://omnitruck.chef.io/install.sh | sudo bash -s -- -P chef-infra-client

# 3. Generate repository
chef generate repo chef-repo

# 4. Create bootstrap script
knife bootstrap 192.168.1.10 \
  --ssh-user deploy \
  --sudo \
  --identity-file ~/.ssh/deploy_key \
  --node-name web1 \
  --run-list 'recipe[web]'

Common Configuration Patterns

# 1. Directory structure for config management
config-mgmt/
├── ansible/
│   ├── inventory.ini
│   ├── ansible.cfg
│   ├── playbooks/
│   │   ├── webserver.yml
│   │   ├── database.yml
│   │   └── common.yml
│   ├── roles/
│   │   ├── nginx/
│   │   └── mysql/
│   └── group_vars/
│       └── all.yml
├── puppet/
│   ├── manifests/
│   ├── modules/
│   └── hiera/
├── chef/
│   ├── cookbooks/
│   ├── roles/
│   └── environments/
└── scripts/
    ├── bootstrap.sh
    ├── common-setup.sh
    └── monitoring-setup.sh

# 2. Bootstrap script template
nano bootstrap.sh

bootstrap.sh:

#!/bin/bash
# Universal bootstrap script

set -e

# Detect OS
if [ -f /etc/os-release ]; then
    . /etc/os-release
    OS=$ID
    VERSION=$VERSION_ID
else
    echo "Cannot detect OS"
    exit 1
fi

echo "Bootstrapping $OS $VERSION..."

# Common setup
echo "Setting hostname..."
hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null || echo "server-$(openssl rand -hex 4)")

echo "Creating users..."
useradd -m -s /bin/bash deploy
usermod -aG sudo deploy
mkdir -p /home/deploy/.ssh
echo "ssh-rsa AAAAB3NzaC1y..." >> /home/deploy/.ssh/authorized_keys
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys
chown -R deploy:deploy /home/deploy/.ssh

echo "Configuring SSH..."
sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
systemctl restart sshd

echo "Configuring firewall..."
if command -v ufw >/dev/null; then
    ufw allow ssh
    ufw allow http
    ufw allow https
    ufw --force enable
elif command -v firewall-cmd >/dev/null; then
    firewall-cmd --permanent --add-service=ssh
    firewall-cmd --permanent --add-service=http
    firewall-cmd --permanent --add-service=https
    firewall-cmd --reload
fi

echo "Installing monitoring agent..."
curl -sSL https://repos.influxdata.com/influxdb.key | apt-key add -
echo "deb https://repos.influxdata.com/ubuntu focal stable" > /etc/apt/sources.list.d/influxdb.list
apt update
apt install -y telegraf

echo "Bootstrap complete!"

Infrastructure as Code Principles

# 1. Version control everything
git init
git add .
git commit -m "Initial infrastructure code"

# 2. Use variables and templates
# Instead of hardcoding IPs:
# web_servers:
#   - 192.168.1.10
#   - 192.168.1.11

# Use:
# web_servers: "{{ groups['web'] }}"

# 3. Separate configuration from code
# Store sensitive data in vaults, not in code

# 4. Test changes in staging first
# Use identical staging environment

# 5. Document everything
# README.md should explain:
# - How to setup
# - What each file does
# - How to add new servers
# - Troubleshooting guide

Monitoring and Logging Preparation

# Standardize logging across servers
sudo mkdir -p /etc/rsyslog.d/
sudo nano /etc/rsyslog.d/90-app.conf

# Add:
$template AppLogFormat,"%timestamp% %hostname% %app-name% %msg%\n"
if $programname == 'myapp' then /var/log/myapp.log;AppLogFormat
& stop

# Centralized logging preparation
# Install and configure fluentd/vector
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-focal-td-agent4.sh | sh

# Configure to send to central server
sudo nano /etc/td-agent/td-agent.conf

<source>
  @type tail
  path /var/log/myapp.log
  tag app.logs
  <parse>
    @type none
  </parse>
</source>

<match app.logs>
  @type forward
  <server>
    host log-aggregator.example.com
    port 24224
  </server>
</match>

📋 Quick Reference Cheat Sheet

Task	Command/File	Purpose
Set env var	`export VAR=value`	Set for current session
Persistent var	Add to `~/.bashrc` or `~/.profile`	Set for all sessions
Generate SSH key	`ssh-keygen -t ed25519`	Create secure key pair
Copy SSH key	`ssh-copy-id user@host`	Install key on server
SSH config	`~/.ssh/config`	Simplify connections
Secure copy	`scp file user@host:/path`	Simple file transfer
Smart sync	`rsync -avz source/ dest/`	Efficient file sync
Terminal session	`screen` or `tmux`	Multiplex terminals
Detach screen	`Ctrl+a then d`	Leave session running
Reattach	`screen -r` or `tmux attach`	Return to session
Ansible test	`ansible all -m ping`	Test connectivity
Inventory file	`inventory.ini`	List managed servers
Bootstrap	`bootstrap.sh`	Initial server setup
Config backup	`rsync -avz --backup`	Backup with versioning

🚀 Practice Exercises

Exercise 1: Create a Development Environment Setup

# dev-setup.sh
#!/bin/bash
echo "Setting up development environment..."

# 1. Create directory structure
mkdir -p ~/projects/{bin,src,tmp,logs}

# 2. Update .bashrc
cat >> ~/.bashrc << 'EOF'
# Development aliases
alias proj="cd ~/projects"
alias src="cd ~/projects/src"
alias logs="cd ~/projects/logs"

# Git shortcuts
alias gs="git status"
alias ga="git add"
alias gc="git commit"
alias gp="git push"

# Python virtualenv
export WORKON_HOME=~/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh 2>/dev/null || true
EOF

# 3. Install development tools
sudo apt update
sudo apt install -y \
    git vim curl wget \
    python3 python3-pip python3-venv \
    nodejs npm \
    docker.io docker-compose

# 4. Setup SSH for GitHub
ssh-keygen -t ed25519 -C "github-$(date +%Y%m%d)" -f ~/.ssh/github_key
echo "Add this key to GitHub:"
cat ~/.ssh/github_key.pub

# 5. Create project template
mkdir -p ~/projects/template/{src,tests,docs,config}
touch ~/projects/template/README.md
touch ~/projects/template/.gitignore

echo "Development setup complete!"

Exercise 2: Multi-Server Command Execution

# multi-exec.sh
#!/bin/bash
# Execute command on multiple servers

SERVERS=("user@server1" "user@server2" "user@server3")
COMMAND="$1"

if [ -z "$COMMAND" ]; then
    echo "Usage: $0 <command>"
    exit 1
fi

for server in "${SERVERS[@]}"; do
    echo "=== Executing on $server ==="
    ssh -o ConnectTimeout=5 "$server" "$COMMAND"
    echo
done

Exercise 3: Automated File Sync with Monitoring

# sync-with-monitor.sh
#!/bin/bash
SOURCE="/data"
DEST="user@backup-server:/backup"
LOG="/var/log/sync.log"
ALERT_EMAIL="admin@example.com"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG"
}

# Start monitoring
log "Starting sync from $SOURCE to $DEST"

# Run rsync with progress
rsync -avz --progress --stats \
    --log-file="$LOG.tmp" \
    "$SOURCE/" "$DEST/"

SYNC_EXIT=$?

# Check result
if [ $SYNC_EXIT -eq 0 ]; then
    log "Sync completed successfully"
    
    # Get stats
    STATS=$(tail -20 "$LOG.tmp" | grep -E "sent|received|total")
    log "Stats: $STATS"
    
    # Cleanup
    rm -f "$LOG.tmp"
else
    log "ERROR: Sync failed with exit code $SYNC_EXIT"
    
    # Send alert
    echo "File sync failed. Check $LOG for details." | \
        mail -s "Sync Alert" "$ALERT_EMAIL"
    
    exit 1
fi

🔗 Master Advanced DevOps with Hands-on Labs

These advanced topics separate junior from senior DevOps engineers. Practical experience is key to mastering them.

👉 Practice advanced DevOps techniques with real scenarios at:
https://devops.trainwithsky.com/

Our platform provides:

Multi-server management exercises
SSH key and security labs
Configuration management simulations
Real-world automation projects
Expert-guided advanced scenarios

Frequently Asked Questions

Q: Should I use .bashrc or .profile?
A: Use .bashrc for shell-specific settings (aliases, functions). Use .profile for login settings (PATH, environment variables).

Q: What's the most secure SSH key type?
A: Ed25519 is currently the most secure. RSA 4096-bit is also secure and more widely compatible.

Q: When should I use rsync over scp?
A: Always use rsync unless you need a simple one-time copy. Rsync is faster for subsequent transfers (copies only changes).

Q: Is tmux better than screen?
A: Tmux is more modern with better features and active development. Screen is simpler and available everywhere.

Q: Which config management tool should I learn first?
A: Start with Ansible - it's agentless (no setup on servers) and uses YAML which is easy to learn.

Q: How often should I rotate SSH keys?
A: Best practice is every 6-12 months, or immediately if a key is suspected to be compromised.

Q: Can I use environment variables for secrets?
A: For local development yes, but for production use secret management tools (HashiCorp Vault, AWS Secrets Manager, etc.).

Need help with advanced DevOps scenarios or have specific questions? Share your challenge in the comments below! 💬

SKY Tech – Explore Technology!