Step 107. Exploring Linux Logs — /var/log

Level 2 — Security Introduction and Attack Skill Basics | Difficulty ★★☆☆☆ | Estimated time: 2.5 hours

Prerequisites: grep and find from Step 20, process concepts from Step 26, directory structure from Step 27. A Linux machine (WSL works) where you have root or sudo.

What you need: a Linux terminal. WSL works too — all of today’s measurements were done on WSL Ubuntu.
⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime.
Caution: reading logs is a safe lookup that doesn’t change the system. However, logs contain other users’ records too, so on a shared server, limit your reading to confirming your own work.

Until now we’ve been learning the skills of "the side that breaks through doors." Today we shift our gaze — we read /var/log, the place that records who knocked on the door, and when. Almost everything that happens on Linux leaves a trace in this folder. Login successes and failures, sudo usage, services starting and dying.

Logs are the core battlefield of offense and defense. Attackers try to erase or forge logs after breaking in; defenders read logs to reconstruct intrusions. Today is both sides’ first step — internalizing "where the logs are and how to read one line." And you’ll cause an incident yourself and watch it get logged.

1. Learning Objectives

By the end of this chapter, you will be able to:

State the roles of the main files in /var/log (auth.log, syslog, kern.log, wtmp, etc.)
Dissect a log line into time, host, process, and message
Find login successes/failures and sudo usage records with grep
Know why log rotation (.1, .gz) exists and how to read rotated logs
Explain the concepts of log deletion/forgery and their countermeasures (rotation, remote forwarding)

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Linux shell + text searching
Today’s commands	`ls -la /var/log`, `tail`, `grep -a`, `zcat`, `last`, `journalctl`
Concepts needed	syslog format, log rotation (logrotate), authentication logs, binary logs (wtmp), remote log forwarding
Today’s artifact	A comparison record of "incident I caused → log line"

2-1. /var/log — The Archive of the Incident Ledger

/var is the folder of "variable data," and the log inside it is the records archive. Contents vary by distro and installed services, but the core members of the Ubuntu family are fixed.

File	What it records
`auth.log`	Authentication events like logins, su, sudo (Ubuntu/Debian family)
`syslog`	System-wide messages (everything except authentication)
`kern.log`	Kernel messages — drivers, firewall logs, etc.
`dpkg.log`, `apt/`	Package install/remove records
`wtmp`, `btmp`	Binary records of login successes (wtmp) and failures (btmp)
`journal/`	The systemd journal — a parallel record on modern systems

On the CentOS/Rocky family, auth.log goes by the name secure. Only the name differs; the role is the same.

2-2. Anatomy of a Log Line

A syslog-family line is four pieces.

2026-09-09T15:27:27+09:00  mypc  su:  pam_unix(su:auth): authentication failure; ...
└── time (with timezone) ──┘  └host┘ └actor┘ └──────── message ────────┘

Reading backwards is more comfortable — read the message first to grasp "what kind of event," then the time for when, then the actor for who (which program) left it.

2-3. Log Rotation — Ledgers Are Stored in Torn Pieces

A log can’t grow forever. Before it swallows the disk, the system cuts old logs away and stores them — this is rotation, handled by logrotate. When auth.log fills up, it’s renamed to auth.log.1, and older ones are compressed like auth.log.2.gz. So when hunting for "yesterday’s incident," you have to dig through .1 and .gz too.

2-4. The Battlefield — The One Who Erases and the One Who Guards

Once an attacker breaks in successfully, the logs are their most dangerous witness. So their moves are fixed — erase only their own lines, delete whole files, or stop the program that writes logs (rsyslog). The defender’s counter is fixed too — forward logs to another server the instant they’re created (remote logging), so that even if the compromised machine is wiped, a copy survives. Today is the day we learn this battlefield as a map.

3. Follow Along

3-1. Touring /var/log (measured)

ls -la /var/log/

Output (measured 2026-09-09, WSL Ubuntu 24.04 — varies by environment):

-rw-r--r--   1 root   root    25855 Sep  8 21:12 alternatives.log
drwxr-xr-x   2 root   root     4096 Sep  8 21:12 apt
-rw-r-----   1 syslog adm     63461 Sep  9 15:12 auth.log
-rw-rw----   1 root   utmp        0 Feb 10  2026 btmp
-rw-r--r--   1 root   root   460954 Sep  8 21:13 dpkg.log
-rw-r-----   1 syslog adm   1094645 Sep  9 15:07 kern.log
-rw-r-----   1 syslog adm   4070410 Sep  9 15:12 syslog
-rw-rw-r--   1 root   utmp   130176 Sep  9 15:07 wtmp
drwxr-sr-x+  3 root   systemd-journal 4096 Sep  8 21:11 journal
(partially omitted)

How to read it: start with the permissions. auth.log is -rw-r----- syslog adm — only the owner syslog writes, and only the adm group reads. Authentication records are closed to just anyone. To read them, you need sudo or adm group membership. The permission knowledge from Steps 23–24 is used right here.

3-2. Causing an Incident and Catching It Live (measured)

Logs are understood a hundred times faster by "watching them get written" than by reading them. Run sudo once to cause an incident, then check immediately.

sudo -n true
sleep 1
sudo tail -5 /var/log/auth.log

Output (measured 2026-09-09, paths and hostname adjusted):

2026-09-09T15:12:49+09:00 mypc sudo:     root : TTY=pts/0 ; PWD=/home/user/project ; USER=root ; COMMAND=/usr/bin/true
2026-09-09T15:12:49+09:00 mypc sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
2026-09-09T15:12:49+09:00 mypc sudo: pam_unix(sudo:session): session closed for user root

How to read it: the sudo you just typed was recorded in three lines. The first line’s four pieces — TTY (which terminal), PWD (from which folder), USER (as whom), COMMAND (what). The visceral realization that even the fact that you’re studying this chapter right now goes into a log is today’s core experience. An intruder’s commands are recorded exactly the same way.

3-3. Finding Failed Logins (measured)

This time let’s deliberately create a failure. Attempt su with a wrong password.

su -s /bin/sh nobody -c "echo wrongpw | su -c true root"
sudo grep -a "fail" /var/log/auth.log | tail -2

Output (measured 2026-09-09):

2026-09-09T15:27:27+09:00 mypc su: pam_unix(su:auth): authentication failure; logname= uid=65534 euid=0 tty= ruser=nobody rhost=  user=root
2026-09-09T15:27:30+09:00 mypc su[660]: FAILED SU (to root) nobody on none

How to read it: it’s crystal clear "who (ruser=nobody) failed to become whom (user=root)." In the auth.log of an internet-exposed server, brute-force login attempts in the form Failed password for ... from <IP> pile up by the thousands every day — the first lines a defender looks at.

Notice the -a option here. Plain grep sudo /var/log/auth.log may output this (measured 2026-09-09):

grep: /var/log/auth.log: binary file matches

Cause: if a log file has NUL bytes mixed in, grep judges the file as "binary" and won’t show its contents. Adding -a (force text) solves it.

3-4. Binary Logs — wtmp Is Read with last, Not cat (measured)

wtmp is not text but structured binary, so reading it with cat gives garbage. There’s a dedicated tool.

last -5

Output (measured 2026-09-09):

reboot   system boot  6.18.33.2-micros Wed Sep  9 15:07   still running
reboot   system boot  6.18.33.2-micros Wed Sep  9 14:32 - 14:32  (00:00)
reboot   system boot  6.18.33.2-micros Wed Sep  9 14:29 - 14:31  (00:02)

wtmp begins Tue Sep  8 21:11:41 2026

How to read it: you see "when it booted and how long it stayed up." On systems with user logins, lines of username tty source-IP time appear together — the first material for reconstructing an intrusion timeline. btmp, the binary version of "login failures," is read with lastb.

3-5. Reading Rotated Logs (measured)

ls /var/log/*.gz
zcat /var/log/dmesg.1.gz | head -1
grep "rotate\|weekly" /etc/logrotate.conf

Output (measured 2026-09-09):

/var/log/dmesg.1.gz  /var/log/dmesg.2.gz  /var/log/dmesg.3.gz  ...
[    0.000000] kernel: Linux version 6.18.33.2-microsoft-standard-WSL2 ...
weekly
rotate 4

How to read it: old logs compressed as .gz are read with zcat (decompresses while printing). weekly + rotate 4 in logrotate.conf means "cut weekly, keep up to 4" — in other words, this system’s memory is about 4 weeks. This is why incident investigation must start before logs are deleted.

3-6. journalctl — systemd’s Parallel Record (measured)

Modern distros keep a systemd journal alongside syslog.

journalctl -n 3 --no-pager

Output (measured 2026-09-09, WSL — some environments don’t have it; in that case, 3-1 through 3-5 are enough):

Sep 09 15:12:14 mypc systemd[1]: Stopped user-runtime-dir@65534.service - User Runtime Directory /run/user/65534.
...

How to read it: the same events, but a different tool. journalctl -u servicename (one specific service), -p err (errors and above only), and --since "1 hour ago" (time condition) are the practical trio.

3-7. Looking Again Through the Attacker’s Eyes (concept recap)

Let’s flip what we saw today into an attacker’s to-do list — a reverse reading to make defense visible.

Breaking in leaves my connection in auth.log and wtmp → I want to erase it
But deleting whole files leaves traces of the deletion itself, like a gap in the wtmp begins date
If logs are being forwarded to an external server in real time, erasing is pointless
So mature defense is not "keeping logs" but "smuggling logs out"

We won’t practice log forgery/deletion techniques themselves. Understanding "why they’re made not to work" is today’s defensive literacy.

4. Missions & Exercises

Mission — Incident Reconstruction Practice

Run sudo once in a terminal, find the freshly created sudo log line, and note its four pieces (TTY/PWD/USER/COMMAND)
Deliberately create one failed su and grep out the FAILED SU line
Read recent boot/login records with last -5 and identify one current session (or boot)
Make an "action → log line" comparison table in your wiki — three things you did and the three corresponding log lines

Exercises

Exercise 1. From the auth.log line ... mypc su[660]: FAILED SU (to root) nobody on none, list as many things as you can learn.

Exercise 2. State why wtmp must not be read with cat, and the command used instead.

Exercise 3. With logrotate set to weekly, rotate 4, what problem arises when you try to investigate an authentication incident from 6 weeks ago?

Exercise 4. An attacker deleted the entire auth.log on the breached server, yet the defender can still secure the intrusion records. Describe one structure that makes this possible.

5. Model Answers & Completion Criteria

Mission Model Answer

A model format for the comparison table (based on measurements from 2026-09-09, adjusted):

What I did	Log line (key part)
`sudo -n true`	`sudo: root : TTY=pts/0 ; PWD=... ; COMMAND=/usr/bin/true`
`su root` with a wrong password	`su[660]: FAILED SU (to root) nobody on none`
(system boot)	`reboot system boot ... still running` in `last` output

How to verify: ① does COMMAND in the sudo log exactly match the command you typed? ② did you distinguish ruser (who attempted) from user (the target account) in the FAILED SU line? ③ did you check the "wtmp begins" date in the last output — that date is the starting point of this log’s memory.

Exercise Answers

Answer 1. The time (to the second, with timezone), the hostname, the acting program (su and PID 660), the kind of event (failed identity switch), the attempter (nobody), the target (root), and terminal info (none — meaning it ran without a terminal, a signal of coming through a script). Reading this much out of one line is the fundamentals of log analysis.

Answer 2. wtmp is not text for humans but a sequence of binary structures written by programs, so reading it with cat produces broken characters. last interprets this structure and displays it (the failure record btmp uses lastb).

Answer 3. Since only 4 are kept, logs from 6 weeks ago have most likely already been deleted. In incident investigation, "does the log retention period cover the incident date?" is the first thing to check, and if long-term retention is needed, a remote log server or separate backup is essential.

Answer 4. If logs are forwarded to an external log server the moment they’re created (via rsyslog, etc.), a copy survives even if the original is deleted on the breached server. Unless the attacker takes over both machines, the records live on — this is the substance of the phrase "logs aren’t kept, they’re smuggled out."

Completion Criteria Checklist

[ ] I can state the roles of the 4 core /var/log files (auth.log, syslog, kern.log, wtmp)
[ ] I can explain via permissions why auth.log is closed to regular users
[ ] I can dissect a log line into time, host, actor, and message
[ ] I can explain when grep -a is needed (the binary judgment)
[ ] I know wtmp/btmp must be read with dedicated commands (last/lastb)
[ ] I can read rotated files (.1, .gz) with zcat
[ ] Mission: I completed the incident-reconstruction comparison table

6. Common Pitfalls & Fixes

Wall 1. There’s no auth.log

Symptom: ls: cannot access '/var/log/auth.log': No such file or directory.
Cause: a distro difference — CentOS/Rocky uses /var/log/secure. On WSL or containers, the logging service (rsyslog) may not be running at all, so it may not exist.
Fix: first look at what’s there with ls /var/log/. If there are no syslog-family files at all and only journal/ exists, just read journalctl-centrically. (This chapter’s measurements were taken on WSL Ubuntu 24.04, where auth.log exists.)

Wall 2. grep only says "binary file matches"

Symptom (measured 2026-09-09):

grep: /var/log/auth.log: binary file matches

Cause: NUL bytes mixed into the log made grep judge it as binary.
Fix: add grep -a. It’s a moment when the assumption "logs are always pure text" breaks, so it’s worth remembering.

Wall 3. Permission denied when reading logs

Symptom: tail: cannot open '/var/log/auth.log' for reading: Permission denied.
Cause: the permissions you saw in 3-1 — auth.log is readable only by owner syslog and the adm group.
Fix: read with sudo tail ..., or if it’s a server you administer, check whether you’re in the adm group with groups.

Wall 4. It definitely happened, but it’s not in the logs

Symptom: the record of what you just did isn’t visible.
Cause: ① it already rotated into .1 or .gz, ② you’re looking at the wrong file (authentication is auth.log, services are syslog/journal), ③ some services write their own logs (/var/log/servicename/).
Fix: check the most recently modified logs first with ls -lt /var/log/ | head. The file updated right after the incident is the culprit.

Wall 5. journalctl is missing or empty

Symptom: the command doesn’t exist or shows -- No entries --.
Cause: an environment with restricted systemd, like WSL or containers.
Fix: that’s normal. Read the syslog files instead. Conversely, some systems have only the journal and no files — if you can read both, you’ll never be stuck anywhere.

7. Summary

Today’s Concepts

Concept	One-line explanation
auth.log	The archive of authentication events (Ubuntu) — CentOS calls it secure
syslog	System-wide messages other than authentication
Log rotation	Cutting old logs into `.1`→`.gz` for storage, handled by logrotate
wtmp/btmp	Binary records of login successes/failures — read with `last`/`lastb`
journalctl	A tool for reading the systemd journal — parallel to syslog
Remote logging	Defense that neutralizes deletion attacks by forwarding logs externally at once
Traces of deletion	Erase logs and the "erased interval" itself becomes evidence

Today’s Commands

Command	What it does
`ls -la /var/log/`	Check the log list and permissions
`sudo tail -f /var/log/auth.log`	Watch the authentication log live
`sudo grep -a "pattern" /var/log/auth.log`	Search logs (`-a` = force text)
`last` / `lastb`	Login success/failure history (wtmp/btmp)
`zcat file.gz`	Read compressed old logs
`journalctl -n 10`	Last 10 lines of the systemd journal
`ls -lt /var/log/ \| head`	View most recently updated logs first

An Instinct More Important Than Commands

The ability to read logs is "the ability to ask the system what happened." From the attacker’s side, today’s lesson is clear — every action is recorded. It doesn’t matter in wargames, but in the field, one command line is one line of evidence. From the defender’s side, it’s even simpler — when an incident happens, read auth.log first, in time order, failed records first. And check the log retention period and remote forwarding setup in advance. An eye that has seen one sudo leave three log lines today will suspect "quiet actions" on every system from now on.

Once every box is checked, Step 107 is complete. Click the checkbox in the sidebar to save your progress.