Learn how Linux leaves clues. You will read journalctl, explore /var/log, inspect service logs, check boot/kernel logs, and follow a simple method to troubleshoot like a calm field engineer.
A log is a timestamped clue. When something starts, fails, restarts, gets blocked, authenticates, times out, or crashes, Linux usually writes a record somewhere. Troubleshooting is the art of asking the right log the right question.
Records of system, service, security, kernel, application, and boot events.
The main viewer for systemd journal logs. Best for service and boot troubleshooting.
Traditional log files. Useful for distro-specific, application, auth, package, and rotated logs.
Kernel ring buffer. Useful for boot, driver, disk, hardware, and kernel messages.
When a server is behaving badly, do not start with random fixes. Start with a fast health scan, capture the time window, then narrow the problem.
Use this when the problem is unclear: slow server, failed login, broken app, or “something is not working.”
date uptime df -h free -h systemctl --failed journalctl -p err -b --no-pager dmesg | tail
Start at the bottom: symptom → health → service → logs → root cause. This stops “command gambling.”
Use this map to decide where to look. Click a log source to see what it means and when to use it.
Pick any block on the left. The explanation will appear here with commands and field use.
Collects logs from services, kernel, boot process, and system components. You usually read it through journalctl.
journalctl --disk-usage journalctl --list-boots
Best first tool for modern Linux troubleshooting, especially for failed services and boot issues.
journalctl -xe journalctl -u sshd --since "15 minutes ago" journalctl -p err
Traditional logs live here. Names differ by Linux family.
| Purpose | Ubuntu/Debian | Rocky/RHEL |
|---|---|---|
| General system | /var/log/syslog | /var/log/messages |
| Authentication | /var/log/auth.log | /var/log/secure |
Shows kernel ring buffer messages. Very useful for boot, drivers, disks, network cards, USB, and hardware clues.
dmesg | tail sudo dmesg -T | tail journalctl -k
Use service-specific logs when one daemon is failing. This gives a cleaner signal than reading every log on the machine.
systemctl status sshd journalctl -u sshd -f journalctl -u sshd --since "today"
Use boot logs when the machine booted slowly, failed to mount something, loaded a bad driver, or had startup service failures.
journalctl -b journalctl -b -1 systemd-analyze blame
Use these for SSH login failures, sudo activity, invalid users, brute-force attempts, and privilege escalation traces.
# Rocky/RHEL sudo tail -f /var/log/secure # Ubuntu/Debian sudo tail -f /var/log/auth.log
A service event moves through the system before you read it. This animation explains why the same event may be visible through journalctl and sometimes also in /var/log.
Beginners often run the right command but miss the clue. Click each part of the log line.
Its meaning will appear here.
Start broad only when you are lost. In real troubleshooting, filter by service, time, boot, and severity as quickly as possible.
# Recent important logs, jump to end journalctl -xe # Logs for current boot journalctl -b # Previous boot journalctl -b -1 # Kernel messages from journal journalctl -k
journalctl -xe is useful, but not magic. For a failing service, journalctl -u <service> is usually cleaner.# SSH logs only journalctl -u sshd # Follow SSH logs live journalctl -u sshd -f # Last 15 minutes journalctl -u sshd --since "15 minutes ago" # Only errors and worse journalctl -p err
journalctl -xe| Part | Meaning |
|---|---|
journalctl | Read systemd journal logs. |
-x | Add explanatory text where available. |
-e | Jump to the end of the logs. |
| Weak approach | Better approach |
|---|---|
journalctl -xe forever | journalctl -u sshd --since "10 minutes ago" |
| Read all logs manually | Filter by service, boot, time, priority |
| Restart repeatedly | Read logs first, then restart once |
The skill is not “run journalctl.” The skill is asking the journal a narrow question: which service, which boot, which time, and which severity?
| Need | Command | Use when |
|---|---|---|
| Current boot | journalctl -b | Problem happened after current startup. |
| Previous boot | journalctl -b -1 | System rebooted or crashed earlier. |
| Time range | journalctl --since "1 hour ago" | You know roughly when it failed. |
| Service | journalctl -u sshd | One daemon is failing. |
| Live follow | journalctl -u sshd -f | You want to watch while reproducing the issue. |
| Kernel | journalctl -k | Hardware, driver, boot, disk, NIC issue. |
| Errors only | journalctl -p err | Logs are too noisy. |
journalctl -b journalctl -b -1 journalctl --since "1 hour ago" journalctl --since today journalctl -u sshd -f journalctl -p warning journalctl -p err..alert journalctl -k journalctl --no-pager journalctl -o short-iso
journalctl -xe is useful, but it is not magic. For real troubleshooting, filter by service and time.Use exact windows during incidents, especially in production or classroom evidence capture.
journalctl --since "2026-08-09 10:00" --until "2026-08-09 10:30" --no-pager
Your lab commands include tail -f /var/log/messages, which is right for Rocky/RHEL. On Ubuntu/Debian, the usual equivalent is /var/log/syslog.
# General system logs sudo tail -f /var/log/messages # Authentication and sudo logs sudo tail -f /var/log/secure # Package manager logs sudo less /var/log/dnf.log
# General system logs sudo tail -f /var/log/syslog # Authentication and sudo logs sudo tail -f /var/log/auth.log # Package manager logs sudo less /var/log/apt/history.log
Traditional file logs are still valuable, especially when applications write to files directly or when older operational runbooks expect file-based logs.
# Watch a file live sudo tail -f /path/to/logfile # Search a file for errors sudo grep -i error /path/to/logfile # Follow last 100 lines live sudo tail -n 100 -f /path/to/logfile
File logs can be huge. Beginners often use cat and flood the terminal. Use reading and filtering tools instead.
less /var/log/messagesgrep -i error filetail -n 50 filetail -f filehead filesudo less /var/log/messages sudo grep -i error /var/log/messages sudo grep -i failed /var/log/secure sudo tail -n 50 /var/log/messages sudo tail -f /var/log/messages
sudo less /var/log/syslog sudo grep -i error /var/log/syslog sudo grep -i failed /var/log/auth.log sudo tail -n 50 /var/log/syslog sudo tail -f /var/log/syslog
When one service fails, follow a repeatable sequence. Do not restart blindly; first collect the failure reason.
systemctl status SERVICEjournalctl -u SERVICE --since "30 minutes ago"systemctl cat SERVICEsudo systemctl restart SERVICEsystemctl status SERVICE journalctl -u SERVICE --since "30 minutes ago" --no-pager systemctl cat SERVICE sudo systemctl restart SERVICE systemctl status SERVICE journalctl -u SERVICE --since "5 minutes ago" --no-pager
SSH and sudo issues are common beginner incidents. The key is knowing the log path and service name for the distro family.
systemctl status sshd journalctl -u sshd --since "30 minutes ago" sudo tail -f /var/log/secure sudo grep -Ei "failed|invalid|accepted|refused" /var/log/secure
systemctl status ssh journalctl -u ssh --since "30 minutes ago" sudo tail -f /var/log/auth.log sudo grep -Ei "failed|invalid|accepted|refused" /var/log/auth.log
| Log clue | Likely meaning | Next check |
|---|---|---|
Failed password | Wrong password or brute-force attempt. | User, source IP, auth method. |
Invalid user | Username does not exist. | id USER, account creation. |
Accepted publickey | Successful key-based login. | Confirm user and source IP. |
Authentication refused | Often permissions or account policy. | Home dir, .ssh permissions, account lock. |
Connection closed | Client disconnected or policy refused session. | Server policy, firewall, client logs. |
Boot logs and kernel logs tell you about startup, drivers, disks, memory pressure, network links, and hardware-level symptoms.
journalctl -b journalctl -b -1 journalctl -p err -b systemd-analyze systemd-analyze blame systemd-analyze critical-chain systemctl --failed
systemd-analyze blame.dmesg | tail sudo dmesg -T | tail dmesg | grep -i error journalctl -k journalctl -k -p err
| Clue | Meaning to investigate |
|---|---|
I/O error | Disk/storage path issue. |
EXT4-fs error | Filesystem problem. |
Out of memory | OOM killer or memory pressure. |
segfault | Process crashed at memory level. |
NIC link down | Network interface/link issue. |
blocked for more than 120 seconds | Possible storage or kernel wait. |
Logs are useful, but they can also fill disks. Learn the safe way to inspect log growth and reduce journal usage.
df -h journalctl --disk-usage sudo du -sh /var/log/* | sort -h ls -lh /var/log
Look for unusually large files, old rotated logs, compressed files like .gz, and services writing too much.
# Reduce systemd journal safely sudo journalctl --vacuum-time=7d sudo journalctl --vacuum-size=500M # Test logrotate config without changing files sudo logrotate -d /etc/logrotate.conf
Use priority filters when logs are too noisy. Click a severity card.
The command and meaning will appear here.
Use the S.C.A.N. method: Symptom → Check service/system state → Analyze logs → Narrow root cause.
What exactly is broken? Since when? Who is affected?
Check service status, disk, network, CPU, and recent changes.
Filter logs by time, service, priority, and boot.
Prove the likely cause before changing the system.
Short field-style scenarios. Read the symptom, inspect the clues, then choose the likely root cause.
Symptom: users cannot SSH into the server.
systemctl status sshd journalctl -u sshd --since "10 minutes ago"
Symptom: application says “No space left on device.”
df -h sudo du -sh /var/log/* journalctl --disk-usage
Symptom: server takes several minutes to boot.
systemd-analyze systemd-analyze blame journalctl -b
Click the best answer. This builds pattern recognition, which is half of troubleshooting.
Which command shows logs for the SSH service on Rocky/RHEL?
Which command shows kernel messages?
What is the Ubuntu/Debian equivalent of /var/log/messages for general system logs?
Run these on a safe training VM. Use two terminals where live-follow commands are involved. The checklist saves progress in your browser.
journalctl -xe
Goal: identify timestamp, hostname, unit/service name, and actual message.
systemctl status sshd journalctl -u sshd journalctl -u sshd -f
systemctl status ssh journalctl -u ssh journalctl -u ssh -f
# Rocky/RHEL sudo tail -f /var/log/messages # Ubuntu/Debian sudo tail -f /var/log/syslog
In another terminal, restart a harmless service and watch log movement.
# Rocky/RHEL sudo systemctl restart sshd # Ubuntu/Debian sudo systemctl restart ssh
journalctl -b dmesg | tail journalctl -k systemd-analyze blame
dmesg may require sudo due to kernel security settings.date uptime df -h free -h systemctl --failed journalctl -p err -b --no-pager
journalctl -b --no-pager | tail -30 journalctl --since "1 hour ago" --no-pager journalctl -p warning --no-pager journalctl -k --no-pager | tail -30
0 of 7 complete
This is useful for classroom labs and real incidents. It collects basic state without changing the system.
cat > collect-troubleshooting-evidence.sh <<'EOF'
#!/usr/bin/env bash
set -u
OUT="troubleshooting-evidence-$(hostname)-$(date +%F-%H%M%S)"
mkdir -p "$OUT"
{
echo "Hostname: $(hostname)"
echo "Date: $(date)"
echo "Kernel: $(uname -a)"
} > "$OUT/summary.txt"
uptime > "$OUT/uptime.txt" 2>&1
free -h > "$OUT/memory.txt" 2>&1
df -h > "$OUT/disk.txt" 2>&1
systemctl --failed > "$OUT/failed-services.txt" 2>&1
journalctl -p err -b --no-pager > "$OUT/current-boot-errors.txt" 2>&1
journalctl -k --no-pager | tail -200 > "$OUT/kernel-tail.txt" 2>&1
dmesg | tail -200 > "$OUT/dmesg-tail.txt" 2>&1
tar -czf "$OUT.tar.gz" "$OUT"
echo "Evidence saved to $OUT.tar.gz"
EOF
chmod +x collect-troubleshooting-evidence.sh
./collect-troubleshooting-evidence.shRead the symptom, decide what you would check, then open the answer. This is where learners start thinking like troubleshooters.
Symptom: User cannot SSH, but the server is reachable.
# Rocky/RHEL sudo grep -Ei "failed|invalid|accepted|refused" /var/log/secure journalctl -u sshd --since "30 minutes ago" # Ubuntu/Debian sudo grep -Ei "failed|invalid|accepted|refused" /var/log/auth.log journalctl -u ssh --since "30 minutes ago"
Likely answer: wrong password, locked account, wrong username, or SSH policy. Match the username, source IP, and exact timestamp.
Symptom: Service worked before a config change, then restart failed.
systemctl status SERVICE journalctl -u SERVICE --since "15 minutes ago" --no-pager systemctl cat SERVICE
Likely answer: syntax error, invalid option, bad path, permission problem, or missing dependency. Fix one line, restart once, verify logs.
Symptom: App errors show No space left on device.
df -h journalctl --disk-usage sudo du -sh /var/log/* | sort -h sudo logrotate -d /etc/logrotate.conf
Likely answer: one log file or journal storage is consuming space. Use logrotate or journal vacuum; do not randomly delete evidence.
Symptom: Boot takes several minutes after adding a mount or network dependency.
systemd-analyze systemd-analyze blame systemd-analyze critical-chain journalctl -b --no-pager systemctl --failed
Likely answer: slow mount, network wait, failed unit, DNS delay, or broken dependency.
Symptom: Application freezes and logs show storage errors.
dmesg | grep -Ei "i/o error|ext4-fs error|blk|reset|timeout" journalctl -k -p warning --no-pager lsblk findmnt
Likely answer: disk, filesystem, controller, SAN/NAS, or VM storage issue. Preserve logs and escalate carefully.
Solve the incident using the proper troubleshooting sequence. Pick the best next step each time.
Users cannot SSH into server01. A change was made 10 minutes ago. Your job is to find the likely cause without random guessing.
Keep this section. It is the “panic calmly” page for real work.
# Recent important logs journalctl -xe # Current boot logs journalctl -b # Previous boot logs journalctl -b -1 # Kernel logs journalctl -k dmesg | tail # Service logs journalctl -u sshd # Follow service logs live journalctl -u sshd -f # Only errors and worse journalctl -p err # Logs since a time journalctl --since "30 minutes ago" # Journal storage usage journalctl --disk-usage
systemctl status SERVICE journalctl -u SERVICE --since "15 minutes ago" systemctl cat SERVICE sudo SERVICE_BINARY --test-config # if supported
journalctl -p err journalctl -u SERVICE journalctl --since "YYYY-MM-DD HH:MM" journalctl -b
Use the browser print button. In print view, navigation and voice controls are hidden.
date uptime df -h free -h systemctl --failed
journalctl -xe journalctl -b journalctl -b -1 journalctl -p err -b journalctl --since "1 hour ago"
systemctl status SERVICE journalctl -u SERVICE journalctl -u SERVICE -f journalctl -k dmesg | tail
# Rocky/RHEL sudo tail -f /var/log/messages sudo tail -f /var/log/secure # Ubuntu/Debian sudo tail -f /var/log/syslog sudo tail -f /var/log/auth.log