Step 55. ★ Project: Log Analyzer — Pick the Footprints Out of the Pile of Records

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★☆☆ | Estimated time: 4 hours

Prerequisites: Steps 41–54 complete. You know regular expressions (Step 48), dictionaries (Step 42), file I/O (Step 45), exception handling (Step 46), and command-line arguments (Step 51). This is Level 1’s comprehensive project.

What you need: a PC with Python installed, a text editor, a terminal. Nothing new to install — it’s all standard library.
Caution: today’s practice is 100% safe. The logs we analyze are fake logs we make ourselves. We don’t casually take real server logs — other people’s system records can contain personal information.

A security analyst’s day begins with logs. Servers record in rows who did what, when, and from where, and somewhere in that pile of records are the footprints of an attack. The problem is volume — a person can’t read tens of thousands of lines a day with their eyes. So an analyst’s first tool is "a program that automatically picks out the strange things from logs."

Today we build that. Everything you’ve learned so far — regular expressions, dictionaries, file I/O, exception handling, command-line arguments — gathers into this one project. Suspecting "this address might be doing a brute-force attack" when login failures repeat from one address is today’s detection rule. Simple — but real security systems also started from this simple idea.

1. Learning Objectives

By the end of this chapter, you will be able to:

Explain what a log is and why it’s the basic material of security analysis
Parse the IP and result from one log line with a regular expression
Tally counts by IP and by account with Counter
Complete an analyzer that automatically marks targets exceeding a threshold as [SUSPECT]
Save the analysis results as a report file with numbers and meaning written together

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Python 3.12. Standard library only (re, collections, sys)
Today’s syntax	`re.search()` and groups, `Counter` and `most_common()`, `sys.argv` argument handling, safe parsing (`if not m: continue`)
Concepts needed	Logs, the pattern of brute force, parsing, thresholds and false alarms
Today’s artifacts	`log_analyzer.py` + `report.txt` — my first security analysis tool and its report

2-1. Logs — The System’s Diary

A log is a chronological record left by a program or server. Web servers record visits; login systems record successes and failures. Formats vary, but they usually contain "time, who (IP), what, result." When an incident happens, logs are the only material for retracing "what happened that day."

2-2. The Footprints of Brute Force

A brute-force attack leaves a distinctive pattern in logs. From the same IP, in a short time, login failures numbering in the dozens or hundreds. A normal user doesn’t get their password wrong a hundred times. So the rule holds: "count failures per IP, and suspect whatever crosses the bar."

2-3. Parsing and Counter — Pulling Info from Lines and Counting

The work of pulling the needed pieces (IP, result) from one log line is called parsing. Step 48’s regular expressions are the tool for this job. For counting the pulled pieces, the standard library collections‘ Counter is the specialist — just put things into a Counter and it counts per item, and most_common() pulls them out in order of frequency.

2-4. Thresholds — The Mesh Size of the Net

The bar of "from how many times do we suspect" is called the threshold. Lower it and normal users get caught too (false alarms); raise it and you miss real attacks. A threshold is not truth — it’s the mesh size of the net. This balance is the eternal homework of detection systems.

3. Follow Along

3-1. Making a Sample Log

Since real server logs can’t be taken casually, we make our own for study. make_log.py:

import random

users = ["admin", "guest", "lee", "park"]
lines = []
for i in range(30):
    ip = "192.168.0." + str(random.randint(2, 20))
    user = random.choice(users)
    result = "SUCCESS" if random.random() < 0.8 else "FAIL"
    lines.append(f"2026-09-08 10:{i:02d}:00 {ip} login user={user} result={result}")

# Plant records that deliberately look like an attack
for i in range(12):
    lines.append(f"2026-09-08 11:{i:02d}:00 10.0.0.99 login user=admin result=FAIL")

# Mix in one malformed line too (real logs always have them)
lines.append("system restart")

random.shuffle(lines)
with open("auth.log", "w", encoding="utf-8") as f:
    f.write("\n".join(lines))
print("auth.log created:", len(lines), "lines")

auth.log created: 43 lines

(Measured 2026-09-09. The shuffled order and numbers differ every run.)

How to read the output: among thirty normal records, we planted twelve failures from 10.0.0.99, plus one malformed line. The {i:02d} in the f-string is a format meaning "pad to two digits."

Why: in the making, the log’s format sinks into your hands. Knowing the shape of the data you’ll analyze is half of parsing.

3-2. Parsing One Line

parse_test.py:

import re

line = "2026-09-08 10:05:00 192.168.0.7 login user=lee result=FAIL"
m = re.search(r"(\d+\.\d+\.\d+\.\d+).*result=(\S+)", line)
print("IP:", m.group(1))
print("Result:", m.group(2))

IP: 192.168.0.7
Result: FAIL

(Measured 2026-09-09.)

How to read the output: \d+\.\d+\.\d+\.\d+ is "digits.digits.digits.digits" — an IP shape. The parts wrapped in parentheses become groups, pulled out with group(1) and group(2). .* means "whatever’s in between," and result=(\S+) catches the non-whitespace chunk after result=. The r before the pattern is the raw-string marker (Step 48).

Predict: if you apply the same pattern to a SUCCESS line, what does group(2) give? (measured answer 2026-09-09: SUCCESS is caught — the pattern only looks at position, not the content of the result.)

3-3. Counting the Whole File — Counter

count_test.py:

import re
from collections import Counter

fails = Counter()
with open("auth.log", encoding="utf-8") as f:
    for line in f:
        m = re.search(r"(\d+\.\d+\.\d+\.\d+).*result=(\S+)", line)
        if m and m.group(2) == "FAIL":
            fails[m.group(1)] += 1

for ip, count in fails.most_common():
    print(ip, "failed", count, "times")

10.0.0.99 failed 12 times
192.168.0.19 failed 1 times
192.168.0.20 failed 1 times
192.168.0.17 failed 1 times
...

(Measured 2026-09-09. The lower IPs and their order differ every time you make the log — the top one doesn’t change.)

How to read the output: we put only the IPs of failed lines into the Counter, and the planted 10.0.0.99 shot straight to the top. most_common() sorts by frequency. The anomaly among forty-three lines is visible at a glance.

Why: "tally it and the anomaly reveals itself" is the heart of log analysis.

3-4. Drawing the Line — The Suspicion Verdict

Continuing in count_test.py:

threshold = 5
for ip, count in fails.most_common():
    if count >= threshold:
        print("[SUSPECT]", ip, "-", count, "failures: possible brute force")

[SUSPECT] 10.0.0.99 - 12 failures: possible brute force

(Measured 2026-09-09.)

How to read the output: only those crossing the bar (threshold) get filtered out. Try lowering the bar to 3 and running — in the measured log the result was the same even at 3 because normal failures were only 1 each, but in a log where coincidences overlap, you’d see a false alarm with a normal user showing up as [SUSPECT]. Watching how the results change as you change the bar — that’s the only way to grow a feel for thresholds.

3-5. Skipping Malformed Lines — The Seatbelt

Our log has a differently formatted line mixed in: system restart. Let’s measure what happens when code without a seatbelt meets that line:

import re

m = re.search(r"(\d+\.\d+\.\d+\.\d+).*result=(\S+)", "system restart")
print(m.group(1))

AttributeError: 'NoneType' object has no attribute 'group'

(Measured 2026-09-09.)

How to read it: search returns None when it can’t find a match. Calling group on None kills it. So the standard seatbelt of log processing:

m = re.search(r"(\d+\.\d+\.\d+\.\d+).*result=(\S+)", line)
if not m:
    continue  # skip lines that don't match the format
ip, result = m.group(1), m.group(2)

The assumption "every line will come in format" always breaks in log analysis. Just counting malformed lines and quietly skipping them — that’s the skipped variable in the 3-6 finished code.

3-6. Counting Usernames Too — The Account the Attack Targeted

As important as the IP is "which account was targeted." Just change the pattern slightly. user_test.py:

import re
from collections import Counter

users = Counter()
with open("auth.log", encoding="utf-8") as f:
    for line in f:
        m = re.search(r"user=(\S+)", line)
        if m:
            users[m.group(1)] += 1

print(users.most_common())

[('admin', 20), ('guest', 11), ('lee', 7), ('park', 4)]

(Measured 2026-09-09. The numbers differ every time you make the log — but admin being highest is fixed, because of the attack we planted.)

How to read the output: we only changed the pattern to user=(\S+), and out come the attempt counts per account. The reason admin is overwhelming is that the attack we planted targeted admin. "Many attempts targeted the administrator account" is in itself a finding worth writing in a report. If you count accounts only from failed lines (add the result condition), the picture sharpens — in the measurement, admin’s concentration was confirmed as [('admin', 13), ...].

3-7. Completion — The Log Analyzer

Gather the pieces so far and complete log_analyzer.py:

import sys
import re
from collections import Counter

if len(sys.argv) < 2:
    print("Usage: python log_analyzer.py <logfile> [threshold]")
    sys.exit(1)

filename = sys.argv[1]
threshold = int(sys.argv[2]) if len(sys.argv) > 2 else 5

success = Counter()
fails = Counter()
total = 0
skipped = 0

try:
    f = open(filename, encoding="utf-8")
except FileNotFoundError:
    print("File not found:", filename)
    sys.exit(1)

with f:
    for line in f:
        total += 1
        m = re.search(r"(\d+\.\d+\.\d+\.\d+).*result=(\S+)", line)
        if not m:
            skipped += 1
            continue
        ip, result = m.group(1), m.group(2)
        if result == "FAIL":
            fails[ip] += 1
        elif result == "SUCCESS":
            success[ip] += 1

lines = []
lines.append("=== Log Analysis Report ===")
lines.append(f"Target file: {filename}")
lines.append(f"Total lines: {total} / parsed: {total - skipped} / skipped: {skipped}")
lines.append(f"Detection threshold: {threshold} or more failures")
lines.append("")
lines.append("--- Status by IP ---")
all_ips = sorted(set(success) | set(fails), key=lambda ip: fails[ip], reverse=True)
for ip in all_ips:
    mark = "  [SUSPECT]" if fails[ip] >= threshold else ""
    lines.append(f"{ip}: success {success[ip]}, fail {fails[ip]}{mark}")
lines.append("")
suspects = [ip for ip in all_ips if fails[ip] >= threshold]
if suspects:
    lines.append(f"Summary: {len(suspects)} address(es) crossed the threshold.")
    for ip in suspects:
        lines.append(f"- {ip}: repeated failures in a short time — suspected brute-force attempt")
else:
    lines.append("Summary: no address crossed the threshold.")

report = "\n".join(lines)
print(report)
with open("report.txt", "w", encoding="utf-8") as f:
    f.write(report)

Run it:

python log_analyzer.py auth.log

=== Log Analysis Report ===
Target file: auth.log
Total lines: 43 / parsed: 42 / skipped: 1
Detection threshold: 5 or more failures

--- Status by IP ---
10.0.0.99: success 0, fail 12  [SUSPECT]
192.168.0.3: success 2, fail 1
...
192.168.0.2: success 1, fail 0

Summary: 1 address(es) crossed the threshold.
- 10.0.0.99: repeated failures in a short time — suspected brute-force attempt

(Measured 2026-09-09. The same content was saved to report.txt.)

How to read the code: all pieces you know — sys.argv argument handling (Step 51), regex parsing (3-2), the seatbelt (3-5), Counter tallying (3-3), file saving (Step 45), exception handling (Step 46). Only the assembly is new. set(success) | set(fails) gathers all addresses of both Counters into a union, and key=lambda ip: fails[ip] sorts by "most failures first." The final if suspects: part is where meaning gets added to numbers — if "12 failures" is a number, "suspected brute-force attempt" is the interpretation.

Verification points: run with no argument and Usage: python log_analyzer.py <logfile> [threshold] appears; give a nonexistent file and File not found: appears and it ends quietly (both verified in the 2026-09-09 measurement). You can change the threshold with a second argument: python log_analyzer.py auth.log 3.

4. Missions & Exercises

Mission — Toughening Up the Analyzer

Let’s make the finished analyzer sturdier:

Modify make_log.py to grow the scale: 100 normal records, 30 planted attacks
Run python log_analyzer.py auth.log on the bigger log and check that detection doesn’t waver
Run with the threshold changed to 3, 5, and 10, and record how the [SUSPECT] list changes
Change the attack lines’ account from admin to guest, remake the log, and check whether the account tally (user_test.py) catches that change
Open report.txt, check that numbers and meaning (the summary lines) are together, and add one line of impressions by hand

Exercises

Q1. State the three parts of the pattern a brute-force attack leaves in logs (which IP, how much, what result).

Q2. What happens without if not m: continue, and what’s the name of the error that occurs?

Q3. What problem occurs if the threshold is too low, and what problem if too high? State each in security terms (from the false-alarm perspective).

Q4. A good analysis report writes "numbers" and "meaning" together. Write your own "meaning" sentence to attach to the line 10.0.0.99: success 0, fail 12.

5. Model Answers & Completion Criteria

Mission Model Answer

Items 1~2: in make_log.py, change range(30) to range(100) and the attack lines’ range(12) to range(30), then remake the log. Run the analyzer again, and even as the total line count grows to 131, the top is still 10.0.0.99 — tally-based detection doesn’t waver as scale grows.

An example of item 3’s records (based on the 2026-09-09 measured log):

Threshold 3: [SUSPECT] 10.0.0.99 (12 times)
Threshold 5: [SUSPECT] 10.0.0.99 (12 times)
Threshold 10: [SUSPECT] 10.0.0.99 (12 times)

In this log, normal failures are only 1 each, so all three are the same — but in a log with many coincidental failures, threshold 3 produces false alarms. The record that "I observed while changing the threshold" is itself the heart of the mission.

Item 4: change the attack lines to user=guest and user_test.py’s #1 changes to guest. A step confirming that when you change the question (change the tally criterion), the data answers anew.

Item 5: report.txt must contain both "10.0.0.99: success 0, fail 12" (numbers) and "suspected brute-force attempt" (meaning).

Exercise Solutions

Q1 solution. The pattern: from the same IP, in a short time, login failures repeating. Since a normal user doesn’t get their password wrong dozens of times, when this combination holds we suspect a brute-force attempt.

Q2 solution. On a line the regex doesn’t match, you’d call m.group(...), and since m is None, you get AttributeError: 'NoneType' object has no attribute 'group' and the program dies (measured 2026-09-09). One seatbelt line prevents the analysis from stopping.

Q3 solution. Too low, and normal users get caught as [SUSPECT] too — false alarms (false positives) increase until analysts start ignoring alerts. Too high, and you get misses (false negatives), overlooking real attacks. Threshold tuning is balancing between the two, and the right answer differs per environment.

Q4 solution. Example: "With 12 failures repeating and not a single success, this is suspected to be a brute-force attempt rather than normal use. Recommend blocking this address and adding monitoring." — the numbers (12 times, 0 successes) are the grounds, and the sentence is the judgment. The completion of analysis is not code but sentences.

Completion Criteria Checklist

[ ] I can explain what a log is and what pattern brute force leaves
[ ] I can parse the IP and result from one log line with a regular expression
[ ] I can tally and sort with Counter and most_common()
[ ] I can explain why the if not m: continue seatbelt is needed
[ ] I can explain the relationship between thresholds and false alarms
[ ] log_analyzer.py does argument handling, tallying, [SUSPECT] marking, and file saving — all of it
[ ] Mission: I finished the scaled-up log and the threshold-change experiments

6. Common Pitfalls & Fixes

Wall 1. AttributeError: ‘NoneType’ object has no attribute ‘group’

Symptom (measured 2026-09-09):

AttributeError: 'NoneType' object has no attribute 'group'

Cause: you called group on a line the regex doesn’t match (a differently formatted line). search returns None when it can’t find a match.
Fix: the 3-5 seatbelt if not m: continue. In log analysis, the assumption "every line comes in format" always breaks.

Wall 2. The Regex Finds Nothing at All

Symptom: the Counter is completely empty.
Cause: the pattern and the actual log format mismatch. Spaces around result=, differences in the date format, etc.
Fix: pick one line and test it on its own, like 3-2’s parse_test.py. The standard order for regexes is "match it on one line first, then widen to the file."

Wall 3. The Backslash (\d) Behaves Strangely

Symptom: you wrote the pattern, but \d isn’t recognized as digits.
Cause: backslashes are treated specially in Python strings.
Fix: as you learned in Step 48, use a raw string with r in front of the pattern (r"..."). Every example today is written that way.

Wall 4. The File Can’t Be Found

Symptom: it dies with FileNotFoundError, or with the finished code, the File not found: message.
Cause: a typo in the file name, or the folder where you ran the script differs from the folder where the log file is. Relative paths are based on "the folder you ran from" (Step 51 review).
Fix: check the actual file name and location with os.listdir() or File Explorer. When giving a path as an argument, if the folder differs, give it with the path, like python log_analyzer.py logs\auth.log.

Wall 5. Worshipping the Threshold

Symptom: you miss an attack that failed 4 times against threshold 5, and believe 5 times is "confirmed attack."
Cause: you’ve forgotten the essence of thresholds. A threshold is not truth — it’s the mesh size of the net.
Fix: like mission item 3, see for yourself how the results change as you change the threshold. Knowing in your body that "a finer net catches more, and more noise too" is the analyst’s sense.

7. Summary

Today’s Concepts

Concept	One-line description
Log	A chronological record — the basic material of security analysis
Brute-force pattern	Same IP + short time + repeated failures
Parsing	The work of pulling pieces of information from one log line
Counter	The specialist tool for counting per item
Threshold	The line of suspicion — the mesh size of the net
False alarm	Being suspected while normal — when the bar is too low

Today’s Syntax

Syntax	What it does
`re.search(r"pattern", line)`	Find the first match — None if not found
`m.group(1)`, `m.group(2)`	Pull out parenthesized groups
`Counter()` / `c[x] += 1`	Count per item
`c.most_common()`	Sort by frequency
`if not m: continue`	Seatbelt that skips malformed lines
`sys.argv[1]`	Receive the file to analyze as an argument

The Instinct That Matters More Than Commands

Handed a log file, an analyst’s questions are fixed. (a) How much is there — the line count. (b) Who comes most — the tally by IP. (c) What are they after — the tally by account. (d) Are there strange times? (e) Among the successes, is anything strange — one success after a hundred failures is the scariest. Today we did (a)~(c) in code. The tools for the rest are the same — tallying by time slot is, in the end, pulling out the time piece and putting it into a Counter.

Remember two more things. First, what you made today is the prototype of a real security system (the large-scale log analysis system called a SIEM) — collect, parse, tally, alert past the threshold. Only the scale differs; the structure is the same. Someone who has made this prototype can also see the attacker’s position: "if I attacked, what pattern would I leave in the logs?" Second, logs don’t lie, but they don’t tell everything either. Times when recording was off are silence. "The logs are quiet = nothing happened" is not true — engrave this one thing right now.

Once every box is checked, Step 55 is complete. Click the checkbox in the sidebar to save your progress.