Linux for DevOps: The Commands You Actually Use, and When to Reach for Them

There are thousands of Linux commands. You will use about forty of them, and maybe fifteen of those every single day.

The trick isn't memorising flags. It's knowing which question you're asking, because each question has one obvious command behind it. Someone says "the server is slow." That's a different first command than "the deploy failed" or "we ran out of space." Once you can map a symptom to a command, the flags follow naturally, and man is right there for the rest.

This guide walks through the commands that actually earn their place, what each one really tells you, and how they get combined during a real incident. If you're new to Linux, read it top to bottom. If you've been doing this a while, skip to the troubleshooting playbooks near the end.

Start with the question, not the command

Almost everything you do on a server starts as one of these four questions.

Keep that table in your head and you're already useful on an unfamiliar box.

Where things live

The filesystem looks intimidating until you realise you only care about six places.

Moving around is four commands: pwd tells you where you are, cd moves, ls lists, and less reads.

ls -lh                 # long listing, human-readable sizes
ls -lhtr               # oldest last, so the newest file is at the bottom
ls -la /etc/nginx      # include dotfiles
less /etc/nginx/nginx.conf   # page through it, q to quit

Two habits worth picking up early. Use ls -lhtr when you want to know what changed recently in a directory, because the newest thing lands next to your prompt. And use less rather than cat for anything long, so a 2GB log doesn't flood your terminal.

/proc deserves a note because it looks weird and it's genuinely useful. Every running process has a directory named after its process ID, and those files answer questions nothing else can:

cat /proc/1234/cmdline | tr '\0' ' '   # the exact command line, args included
ls -l /proc/1234/cwd                   # which directory it's running in
cat /proc/1234/limits                  # its file descriptor and memory limits

What's running: ps, top and htop

ps is a snapshot. top is a live view. That's the whole difference.

ps aux                            # every process, with CPU and memory columns
ps -ef                            # same idea, shows the parent PID clearly
ps aux --sort=-%cpu | head -10    # top 10 CPU consumers, right now
ps aux --sort=-%mem | head -10    # top 10 memory consumers
pgrep -a nginx                    # PIDs matching a name, with command lines
pstree -p 1234                    # what this process spawned

ps aux --sort=-%mem | head is the single most useful process command there is. When someone reports a memory problem, that's your first line.

top refreshes every few seconds and shows the same data live. The columns people misread:

• load average, top right: how many tasks wanted to run, averaged over 1, 5 and 15 minutes. Compare it to your core count from nproc. A load of 4 on a 4-core box is fully busy, not broken.
• %Cpu(s) row: us is your application code, sy is the kernel, wa is time spent waiting on disk, and st is CPU stolen by the hypervisor. High wa means storage. High st on a cloud VM means a noisy neighbour or a throttled instance.
• RES is memory actually in RAM for that process. VIRT is address space it has reserved, which for a JVM or a Go binary can be enormous and mean nothing. Look at RES.

Useful keys inside top: M sorts by memory, P by CPU, 1 breaks the CPU line out per core, H shows individual threads, and q quits.

htop is top with colour, mouse support, a readable tree view (F5), and search (F3). It isn't installed by default on most servers. Install it on boxes you use often, but stay fluent in top, because that's what you get on a locked-down production host at 2am.

Disk space: df vs du

These two get confused constantly, and the difference matters.

df asks the filesystem how much space it has. du walks a directory tree and adds up file sizes. So df answers "am I out of space?" and du answers "what's using it?"

df -h              # free space per filesystem, human readable
df -h /var         # just the filesystem that /var lives on
df -i              # inode usage, which is a separate way to run out
du -xh -d1 /var | sort -h    # size of each subdirectory of /var, smallest first
du -sh /var/log              # one total for a single directory

The -x on du keeps it on one filesystem, so it doesn't wander into a network mount or /proc and waste ten minutes. sort -h sorts those human-readable sizes correctly, which plain sort won't.

Then walk down. du -xh -d1 / shows you /var is huge, du -xh -d1 /var points at /var/log, and two more steps find the actual file. If ncdu is available, ncdu -x / does the same exploration interactively and is much faster to drive.

Three things that catch people out:

df says full, du says there's plenty. Almost always a deleted file that a process still holds open. The space isn't released until the file descriptor closes.

sudo lsof +L1          # open files whose link count is 0, so deleted but still held
sudo lsof -nP | grep deleted

Fix it by restarting the process holding it, not by hunting for a file that no longer has a name.

Disk isn't full but writes fail. Check df -i. Each filesystem has a fixed number of inodes, and millions of tiny files (session files, unrotated logs, a cache gone wrong) can exhaust them while space looks fine.

There's a few percent you can't use. ext4 reserves a slice of the filesystem for root by default, which is why a "100% full" disk sometimes still lets root write. It's a safety margin, not a bug.

To reclaim space from a log that's still being written, don't rm it. The process keeps writing to the deleted inode and you get nothing back:

sudo truncate -s 0 /var/log/huge-app.log   # empties it in place, fd stays valid

Memory: free, and why "free" isn't the number you want

free -h
free -h -s 5    # refresh every 5 seconds

Output is roughly:

               total        used        free      shared  buff/cache   available
Mem:            15Gi       6.2Gi       311Mi       142Mi       8.9Gi       8.5Gi
Swap:          2.0Gi       128Mi       1.9Gi

The column beginners panic about is free, and it's the one to ignore. Linux uses spare RAM to cache files, because idle RAM is wasted RAM. That cache shows under buff/cache and gets handed back the moment an application needs it.

Read the available column. That's the kernel's estimate of what a new process could get without swapping. 8.5Gi available on this box means memory is fine, even though free reads 311Mi.

Swap matters as a trend, not a number. A little swap used is normal. Swap climbing steadily while available drops means you're heading for trouble. And heavy swap activity makes a machine feel frozen long before it actually runs out, so watch the si and so columns in vmstat 1.

When something got killed, the kernel log tells you plainly:

dmesg -T | grep -i -E 'out of memory|killed process'
journalctl -k --since "1 hour ago" | grep -i oom

The OOM killer picks a process, kills it with an untrappable signal, and logs the name and PID. If your app "just disappeared" with no error in its own logs, check this first. It's the most common explanation.

/proc/meminfo has the full detail if you need it, but free -h plus the OOM check answers most questions.

Load and uptime

uptime            # how long it's been up, who's logged in, load averages
w                 # same header, plus what each logged-in user is doing

Load average counts tasks that wanted CPU plus tasks stuck waiting on disk. That second part is specific to Linux, and it's why a box can show load 30 while the CPUs look bored. High load with low CPU usage means something is blocked on I/O, not compute.

To see which it is:

nproc                # how many cores, so you know what "high" means
vmstat 1 5           # CPU, memory, swap and I/O, once a second
iostat -xz 1         # per-disk utilisation and latency (needs the sysstat package)
mpstat -P ALL 1      # per-core breakdown, to spot one pinned core

vmstat 1 is the fastest triage tool on the list. Ignore its first line, which is an average since boot, and read the ones after. Watch the r column (tasks waiting to run), b (tasks blocked), si/so (swap traffic) and wa (I/O wait). Those four columns separate a CPU problem from a memory problem from a disk problem in about ten seconds.

Services: systemctl

On any modern distribution, systemd starts and supervises services. systemctl is how you talk to it.

systemctl status nginx        # running? enabled? recent log lines? main PID?
systemctl start nginx
systemctl stop nginx
systemctl restart nginx       # stop then start, brief downtime
systemctl reload nginx        # re-read config without dropping connections
systemctl enable nginx        # start automatically at boot
systemctl disable nginx
systemctl is-active nginx     # scriptable, prints active or inactive
systemctl --failed            # everything that failed, the best post-reboot check

systemctl status is more informative than people give it credit for. It shows whether the unit is enabled at boot, the main PID, memory use, and the last ten log lines. Nine times out of ten the error you need is already on screen.

Two rules that save a lot of confusion:

Run systemctl daemon-reload after editing a unit file. systemd caches unit definitions. Without the reload, your change simply isn't in effect, and the service restarts with the old config.

reload only works if the unit defines an ExecReload. If it doesn't, use restart. And prefer reload for web servers during business hours, since restart drops in-flight requests.

Don't edit vendor unit files under /lib/systemd/system directly, because a package update overwrites them. Use a drop-in instead:

sudo systemctl edit nginx        # creates an override.conf drop-in for you
systemctl cat nginx              # shows the unit plus every override, in order

Logs from systemd: journalctl

If a service is managed by systemd, its output goes to the journal. journalctl reads it.

journalctl -u nginx                    # everything from one unit
journalctl -u nginx -f                 # follow live, like tail -f
journalctl -u nginx -e                 # jump to the end
journalctl -u nginx --since "30 min ago"
journalctl -u nginx --since "2026-09-12 14:00" --until "2026-09-12 15:00"
journalctl -p err -b                   # errors and worse, this boot only
journalctl -k                          # kernel messages
journalctl -b -1                       # the previous boot, for crash investigation
journalctl -u nginx -n 200 --no-pager  # last 200 lines, no interactive pager

The flags that matter in practice are -u to pick a unit, -f to watch it live, --since to bound the window, and -p err to cut noise. Combine them freely: journalctl -u myapp -p err --since "1 hour ago" is the shape you'll type most.

-b -1 for the previous boot only works if the journal is persisted to disk. If /var/log/journal doesn't exist, logs are in memory and vanish on reboot. Creating that directory and running systemctl restart systemd-journald fixes it, and it's worth doing before you need it.

The journal can also get large:

journalctl --disk-usage
sudo journalctl --vacuum-time=7d       # or --vacuum-size=500M

Log files on disk

Not everything uses the journal. Anything writing its own files puts them in /var/log.

Reading them:

tail -f /var/log/nginx/error.log            # follow one file
tail -n 100 /var/log/syslog                 # last 100 lines
tail -f /var/log/nginx/*.log                # follow several at once
less +F /var/log/syslog                     # follow, Ctrl-C to scroll, F to resume
zcat /var/log/nginx/access.log.2.gz | less  # read a rotated, compressed log

less +F is the underrated one. tail -f can only move forward, but less +F lets you stop, scroll back to find where the errors started, then resume following.

Logs get rotated by logrotate, configured in /etc/logrotate.d/. Two things worth knowing. logrotate -d /etc/logrotate.d/myapp does a dry run so you can check a new rule without waiting a day. And if an app keeps writing to a rotated file, its logrotate config is missing either a postrotate signal or copytruncate.

Reading text fast: grep, awk and sed

These three are how you turn a pile of log lines into an answer. You don't need to master them. You need about six patterns.

grep finds lines

grep "timeout" app.log
grep -i "error" app.log              # case insensitive
grep -c "500" access.log             # count matches instead of printing them
grep -n "database" config.yml        # show line numbers
grep -v "healthcheck" access.log     # invert: everything except health checks
grep -C 5 "Exception" app.log        # 5 lines of context either side
grep -rn "DB_HOST" /etc/myapp/       # recursive search through a directory
grep -E "error|fatal|panic" app.log  # extended regex, several patterns at once

-C is the one people forget and need most. An error message alone rarely explains itself, and the five lines before it usually do. -v is how you strip the noise floor out of an access log so the real traffic is visible.

awk pulls out columns and does maths

Think of awk as "give me field number N, and count things." Fields are whitespace-separated by default, $1 is the first, $0 is the whole line.

awk '{print $1}' access.log                          # just the client IP
awk '{print $1, $7, $9}' access.log                  # IP, path, status code
awk '$9 >= 500 {print}' access.log                   # only 5xx responses
awk '{sum += $10} END {print sum/1024/1024 " MB"}' access.log   # total bytes sent
awk -F: '{print $1}' /etc/passwd                     # custom field separator

The pattern you'll reuse forever is "count occurrences and rank them":

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

That's the top ten IPs hitting your server. Swap $1 for $7 and you get the top ten URLs. Swap it for $9 and you get the distribution of status codes. sort groups identical lines, uniq -c counts each group, sort -rn ranks them highest first. Learn that one chain and you've covered most log analysis you'll ever do by hand.

sed edits streams

sed 's/old-host/new-host/g' config.yml          # print with substitutions, file untouched
sed -i.bak 's/old-host/new-host/g' config.yml   # edit in place, keep config.yml.bak
sed -n '100,150p' huge.log                      # print just lines 100 to 150
sed '/^#/d; /^$/d' nginx.conf                   # strip comments and blank lines

Always run a sed substitution without -i first, look at the output, then add -i. And when you do use -i, give it a suffix like -i.bak so there's a way back. In-place edits on a production config with no backup is how a five-minute change becomes an incident.

The supporting cast

wc -l app.log                     # how many lines
cut -d, -f2,5 data.csv            # simpler than awk for fixed delimiters
sort -u hosts.txt                 # sort and dedupe
tr -d '\r' < win.txt > unix.txt   # strip carriage returns
head -50 file / tail -50 file     # first or last N lines
diff -u old.conf new.conf         # what changed between two configs
jq '.items[].name' data.json      # for JSON, use jq rather than grep

Finding files: find

find walks a tree and matches on properties, not content. That's the split to remember: grep searches inside files, find searches for files.

find /etc -name "*.conf"                   # by name
find / -iname "nginx.conf" 2>/dev/null     # case insensitive, hide permission noise
find /var/log -type f -size +100M          # files over 100MB
find /var/log -type f -mtime -1            # modified in the last 24 hours
find /tmp -type f -mtime +30               # not touched in over 30 days
find /home -type d -name ".ssh"            # directories only
find . -type f -name "*.log" -newer deploy-marker   # changed since a reference file

-size +100M combined with du is how you find the file that filled a disk. -mtime -1 is how you answer "what changed recently" when nobody will admit to deploying.

Acting on the results needs care:

find /var/log -name "*.gz" -mtime +14 -print          # look first, always
find /var/log -name "*.gz" -mtime +14 -delete         # then delete
find /var/log -name "*.gz" -mtime +14 -exec gzip -t {} \;    # run a command per file
find . -name "*.log" -print0 | xargs -0 grep -l "panic"      # safe with odd filenames

Run it with -print before you run it with -delete. Every time. A stray path or a misplaced -o in a find ... -delete on a production box is genuinely dangerous, and the dry run costs you two seconds.

The -print0 with xargs -0 pairing exists because filenames can contain spaces and newlines. Piping plain find output into xargs breaks on those, sometimes destructively.

Permissions: chmod and chown

Every file has an owner, a group, and three sets of permissions: read, write and execute for the owner, the group, and everyone else.

ls -l /usr/local/bin/deploy.sh
# -rwxr-xr-x 1 root root 412 Sep 12 10:14 deploy.sh

Read that as owner rwx, group r-x, others r-x. In numbers, read is 4, write is 2, execute is 1, and you add them per set. So rwxr-xr-x is 755.

The handful you'll actually type:

chmod 600 ~/.ssh/id_ed25519       # numeric
chmod +x deploy.sh                # symbolic: add execute for everyone
chmod u+w,go-w file               # add write for owner, remove for group and others
chmod -R 755 /var/www/html        # recursive, be careful with this one
chown -R appuser:appuser /opt/myapp    # change owner and group
chown :developers shared.txt           # group only

Four points that cause real bugs:

Directories need execute, not just read. Execute on a directory means "can traverse into it." A directory at 644 is unreadable in practice, even though the bits look permissive. This is a classic cause of a web server returning 403 on a path that looks fine.

SSH refuses loose key permissions. If your private key is readable by anyone else, SSH rejects it rather than risking it. chmod 600 on the key and 700 on ~/.ssh fixes the "permissions are too open" error.

chmod -R 777 is never the fix. It makes the symptom go away by making everything world-writable, and you've traded a permissions error for a security hole. Find the right owner and use chown instead.

New files get their mode from umask. The usual default of 022 produces 644 files and 755 directories. If a service creates files with unexpected permissions, its umask is the place to look.

Two special bits you'll meet eventually. The sticky bit on a shared directory means only a file's owner can delete it, which is why /tmp is mode 1777. And setgid on a directory (chmod 2775) makes new files inherit the directory's group, which is how shared team directories stay usable.

Processes vs threads

A process has its own memory. A thread runs inside a process and shares that memory with its siblings.

That difference has practical consequences. Two processes have to work to talk to each other, through sockets, pipes or shared memory, and one crashing doesn't take the other down. Threads share everything by default, so communication is free and a bad memory access can take the whole process with it. Threads are also cheaper to create, which is why a web server handles thousands of connections with threads rather than thousands of processes.

On Linux the kernel schedules both as "tasks," which is why the tooling looks similar:

ps -eLf                              # every thread on the system
ps -o pid,nlwp,comm -p 1234          # nlwp = number of threads in this process
top -H -p 1234                       # live per-thread CPU for one process
cat /proc/1234/status | grep Threads
ls /proc/1234/task/                  # one directory per thread

Where this matters day to day: top -H tells you whether a multi-threaded app is actually using its cores or funnelling everything through one hot thread. And a thread count climbing without limit is a leak, which usually ends in "cannot create native thread" and a dead process.

One gotcha when reading ps output. Threads share memory, so a naive sum of per-thread memory badly overcounts. Trust the per-process RES figure instead.

Signals

Signals are how you tell a running process to do something. There are dozens. You need five.

kill 1234              # sends SIGTERM, the polite default
kill -15 1234          # the same thing, written explicitly
kill -9 1234           # last resort
kill -HUP 1234         # reload config, for daemons that support it
pkill -f "python worker.py"    # match against the whole command line
pgrep -f worker.py             # check what that pattern matches, before you kill it
kill -l                        # list every signal name and number

Don't lead with kill -9. SIGTERM lets a process flush buffers, finish in-flight requests, commit its database transaction, and remove its PID file. SIGKILL gives it no chance, so you can get corrupted state, stale locks, and half-written files. Send SIGTERM, wait a few seconds, and only escalate if it's genuinely stuck.

If kill -9 doesn't work, the process is blocked in the kernel. Look for state D in the STAT column of ps aux, which means uninterruptible sleep. That process is waiting on I/O that isn't coming back, often a hung NFS mount or a failing disk. No signal will move it. Fix the I/O, or reboot.

Always run pgrep -f before pkill -f. A pattern like pkill -f python matches far more than you intended, and there's no confirmation prompt.

In scripts, trap is how you handle signals yourself, which is what makes a job safe to interrupt:

#!/bin/bash
cleanup() { rm -f /tmp/job.lock; echo "cleaned up"; }
trap cleanup EXIT INT TERM

systemd does exactly this on your behalf. systemctl stop sends SIGTERM, waits for TimeoutStopSec (90 seconds by default on most distributions), then sends SIGKILL. If your app needs longer to drain, raise that value rather than hoping.

Environment variables

Environment variables configure a process without touching a config file. They're also where a surprising share of "works on my machine" problems live.

echo $HOME                 # read one
printenv                   # print all of them
printenv DATABASE_URL      # print one, exactly, no shell quoting surprises
export API_KEY=abc123      # set for this shell and anything it starts
unset API_KEY
env FOO=bar ./script.sh    # set for one command only

The rule that explains most confusion: variables are inherited by child processes, never sent back to parents. Your export affects commands you run from that shell afterwards. It does not reach a service that's already running, and a variable set inside a script vanishes when the script ends.

Where they get set persistently:

That last row is the one that bites. A service started by systemd does not read your ~/.bashrc. If your app can't see DATABASE_URL even though echo $DATABASE_URL works in your shell, the variable needs to be in the unit file:

[Service]
Environment=LOG_LEVEL=info
EnvironmentFile=/etc/myapp/env

To see what a running process actually got, ask the kernel rather than guessing:

sudo tr '\0' '\n' < /proc/1234/environ

Two cautions. Anything in your environment shows up in that file and often in process listings, so environment variables are a weak place for real secrets. And a variable you export in an interactive shell lands in your shell history, which is worth remembering before you paste a token.

SSH

SSH is how you reach every server, so a bit of setup pays back daily.

ssh user@host
ssh -i ~/.ssh/deploy_key user@host      # specific key
ssh -p 2222 user@host                   # non-default port
ssh user@host "systemctl status nginx"  # run one command and exit
ssh -v user@host                        # verbose, for debugging auth failures

Generate a modern key and copy it over:

ssh-keygen -t ed25519 -C "you@laptop"
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host

Ed25519 keys are short, fast and secure. Use them unless you're dealing with something old that only understands RSA.

The single best investment is ~/.ssh/config. It turns a long command into a short name and applies the right settings automatically:

Host prod-web
  HostName 10.0.1.24
  User deploy
  IdentityFile ~/.ssh/deploy_key
  ProxyJump bastion
 
Host bastion
  HostName bastion.example.com
  User deploy
  ServerAliveInterval 60

Now ssh prod-web reaches a private host through the bastion in one step. scp and rsync pick up the same config, so rsync -av ./dist/ prod-web:/var/www/ works too.

Things that come up constantly:

Permission denied (publickey). Run ssh -v and read which key it offered. Then check permissions: 700 on ~/.ssh, 600 on the private key, 600 on authorized_keys on the server. Loose permissions are silently rejected.

Connection hangs then times out. That's network or firewall, not authentication. Auth failures are fast and explicit. Check the security group or firewall rule, and test the port directly with nc -zv host 22.

Port forwarding for reaching something that isn't exposed:

ssh -L 5432:db.internal:5432 user@bastion   # local port 5432 tunnels to the database

Then connect to localhost:5432 on your laptop. This is the clean way to reach a private database without opening it to the internet.

Agent forwarding (-A) is convenient and risky. It lets the remote host use your local keys, which means root on that host can use them too. Prefer ProxyJump, which doesn't expose your agent.

For long-running work, run it inside tmux or screen on the server. Then a dropped connection doesn't kill your job:

tmux new -s migration     # start
# Ctrl-b then d to detach
tmux attach -t migration  # reconnect later, even from a different machine

Scheduled jobs: cron, and systemd timers

crontab -l              # list your jobs
crontab -e              # edit them
crontab -l -u appuser   # someone else's, as root

Five fields, then the command:

┌─ minute (0-59)
│ ┌─ hour (0-23)
│ │ ┌─ day of month (1-31)
│ │ │ ┌─ month (1-12)
│ │ │ │ ┌─ day of week (0-6, Sunday is 0)
│ │ │ │ │
* * * * * /path/to/command
 
0 3 * * *        /usr/local/bin/backup.sh      # every day at 03:00
*/5 * * * *      /usr/local/bin/check.sh       # every 5 minutes
0 0 1 * *        /usr/local/bin/monthly.sh     # midnight on the 1st
0 9 * * 1-5      /usr/local/bin/weekday.sh     # 09:00, Monday to Friday

Four gotchas account for nearly every broken cron job:

Cron has almost no PATH. It's typically just /usr/bin:/bin, so aws or python3 from your shell may not resolve. Use absolute paths, or set PATH= at the top of the crontab.

Cron doesn't read your shell config. No ~/.bashrc, so none of your environment variables exist. Set what you need explicitly, or source an env file inside the script.

Output goes nowhere useful. Unmailed output disappears on most servers. Redirect it so you can debug later:

0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

The 2>&1 matters, because errors go to stderr and that's exactly the part you want captured.

A % in a crontab means something. It's treated as a newline, so date +\%F needs the backslash. This mostly bites when you're building a dated filename.

Two more habits. Confirm the job ran by checking /var/log/cron or journalctl -u cron. And wrap anything slow in flock so a long run doesn't overlap with the next one:

0 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

systemd timers are the modern alternative, and worth knowing because you'll meet them. They're more verbose to write, but you get real logs in the journal, dependency handling, and a record of whether the last run succeeded:

systemctl list-timers --all       # every timer, last run and next run
systemctl status backup.timer
journalctl -u backup.service      # the actual output of the last run

For a simple hourly script, cron is fine. For anything you need to debug or that depends on other services, timers are better.

Networking commands you'll need anyway

You can't debug a service without touching the network.

ip a                       # interfaces and IP addresses
ip r                       # routing table, including the default gateway
ss -tulpn                  # every listening TCP/UDP port and the process behind it
ss -tn state established   # current connections
lsof -i :8080              # what has port 8080 open
curl -v https://api.internal/health        # full request and response, headers included
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://site/   # status and timing only
dig +short api.example.com                 # what does DNS actually return
dig @8.8.8.8 api.example.com               # ask a different resolver, to spot local DNS issues
ping -c 4 10.0.1.24                        # basic reachability, if ICMP is allowed
mtr example.com                            # traceroute plus continuous loss stats
nc -zv db.internal 5432                    # is that port open from here

ss -tulpn is the one to memorise. It answers "is my app actually listening, and on which address?" A service bound to 127.0.0.1:8080 works locally and is unreachable from anywhere else, which is one of the most common "but it works on the server" bugs. You need sudo to see process names.

For a TLS certificate question:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -dates

When nothing else explains it, watch the packets:

sudo tcpdump -i any -nn port 5432 -c 50

That's a last resort, not a first move, but "are the packets even arriving?" is sometimes the only question left.

Troubleshooting playbooks

This is where the commands stop being trivia. Below are the situations that actually come up, and the order to work through them.

The first 60 seconds on a box someone says is slow

Run these before forming a theory. They're all read-only and take under a minute:

uptime                              # is load high at all, and rising or falling?
nproc                               # what counts as high here
free -h                             # look at "available"
df -h                               # any filesystem at 100%
top -bn1 | head -20                 # the top consumers, one snapshot
vmstat 1 5                          # is it CPU, memory or I/O
journalctl -p err --since "30 min ago" | tail -30    # what the system is complaining about
dmesg -T | tail -20                 # OOM kills, disk errors, network resets

Those eight commands tell you which of the four resources is the problem: CPU, memory, disk space, or disk I/O. Almost every "slow server" is one of them, and guessing before you look wastes more time than running them.

Disk is full

df -h                          # confirm which filesystem, and how full
df -i                          # rule out inode exhaustion
du -xh -d1 / | sort -h         # follow the biggest directory down
du -xh -d1 /var | sort -h
find /var -type f -size +500M -exec ls -lh {} \;   # the actual large files

Then, depending on what you find:

• Big log files: rotate or truncate with truncate -s 0, never rm on an open file. Then fix logrotate so it doesn't recur.
• Journal has grown: journalctl --disk-usage then journalctl --vacuum-time=7d.
• Docker on the box: docker system df then docker system prune (read what it plans to remove first).
• df and du disagree: sudo lsof +L1 and restart whatever holds the deleted file.
• Old archives: find /var/log -name "*.gz" -mtime +14 -print, check the list, then -delete.

Something got killed and nobody knows why

dmesg -T | grep -i -E 'out of memory|killed process'
journalctl -k --since today | grep -i oom
journalctl -u myapp --since "1 hour ago" | tail -50
free -h
ps aux --sort=-%mem | head -10

If the OOM killer logged it, you have your answer and the question becomes why memory grew. If nothing appears in the kernel log, look at the service's own exit status in systemctl status, which shows the exit code and signal.

CPU pinned at 100%

top                                   # press 1 for per-core, P to sort by CPU
ps aux --sort=-%cpu | head -10
top -H -p <pid>                        # is it one thread or all of them
mpstat -P ALL 1                        # one hot core, or everything busy

One core at 100% with the rest idle usually means a single-threaded loop. All cores busy means genuine load, and the question becomes capacity. If the top consumer is kswapd, your real problem is memory pressure, not CPU.

Load is high but the CPUs look idle

This is the I/O case, and it's worth recognising on sight.

vmstat 1 5                         # high b column, high wa percentage
iostat -xz 1                       # which device, and how saturated
ps aux | awk '$8 ~ /D/ {print}'    # processes stuck in uninterruptible sleep
dmesg -T | tail -30                # disk errors, controller resets, NFS timeouts

Processes in D state plus high wa means storage. Look for a failing disk, a saturated network filesystem, or a cloud volume that's exhausted its IOPS burst credits.

A service won't start

systemctl status myapp -l              # the error is usually right here
journalctl -u myapp -n 100 --no-pager  # the last run's full output
journalctl -u myapp -p err -e
systemctl cat myapp                    # the unit as systemd sees it, overrides included
sudo -u appuser /usr/local/bin/myapp    # run it by hand, as the right user

That last line is the trick people miss. Running the binary manually as the service user turns a vague systemd failure into the application's real error message. The usual causes are a config typo, a missing environment variable, a port already taken (ss -tulpn | grep 8080), a file the service user can't read, or a stale PID file.

And if you edited the unit and nothing changed, you forgot systemctl daemon-reload.

The app is running but nothing can reach it

Work outward from the process.

systemctl status myapp             # is it actually running
ss -tulpn | grep 8080              # is it listening, and on which address
curl -v localhost:8080/health      # does it answer locally
curl -v 10.0.1.24:8080/health      # does it answer on its real IP
sudo iptables -L -n                # local firewall rules
dig +short api.example.com         # is DNS pointing where you think

Each step narrows it down. If localhost works but the IP doesn't, it's bound to 127.0.0.1 only. If the IP works from the box but not from outside, it's a firewall or a security group. If DNS returns an old address, nothing else you check matters.

A process keeps dying and restarting

systemctl status myapp                 # restart count and last exit code
journalctl -u myapp | grep -i -E 'start|stop|fail' | tail -40
dmesg -T | grep -i killed              # OOM again
cat /proc/<pid>/limits                 # file descriptor and memory ceilings
ulimit -n                              # the shell's open-file limit

A crash loop every few seconds is usually a config error. Every few minutes or hours points at a resource leak: memory climbing toward an OOM kill, or file descriptors running out. lsof -p <pid> | wc -l compared against the limit tells you if it's descriptors.

"It worked yesterday"

journalctl --since "yesterday 12:00" -p warning
last -x | head -20                                   # logins, reboots, shutdowns
grep -i -E 'install|upgrade' /var/log/dpkg.log | tail -20   # Debian family
rpm -qa --last | head -20                            # RHEL family
find /etc -type f -mtime -2                          # config files changed in 2 days
ls -lhtr /etc/myapp/                                 # newest config file last
stat /etc/myapp/config.yml                           # exact modification time

find /etc -mtime -2 is the one that finds unannounced changes. Something changed, and the filesystem remembers even when nobody does.

A short list of the rest

Commands that don't need a section but will come up:

The habits matter more than the commands

A few things separate someone who's genuinely good at this from someone who knows a lot of flags.

Look before you touch. df, free, top, journalctl and ss all change nothing. Run them first, and you'll often find the problem before you'd have finished typing a fix for the wrong thing.

Dry-run anything destructive. find with -print before -delete. rsync with --dry-run. sed without -i. These take seconds and save outages.

Change one thing at a time. Three simultaneous changes that fix the problem teach you nothing about which one mattered, and one of the others might be a new bug.

Write down what you ran. Paste your commands and their output into the incident channel as you go. It builds the timeline for free, and it stops you repeating a check you already did twenty minutes ago.

Prefer reversible fixes under pressure. Restart the service now, and read the memory leak properly tomorrow.

Learn the fifteen commands at the top of this list properly, and understand the four questions they answer. That's enough to walk onto almost any Linux box and find out what's wrong. Everything else is detail you can look up.