Step 95. Level 1 Comprehensive Project — Completing a Network Watchdog Tool

Level 1 — Programming and the Inside of a Computer | Difficulty ★★★★☆ | Estimated time: 4 hours

Prerequisites: Steps 41–94 in full. We especially reuse Step 79 (port scanner), Steps 92–93 (SQLite), and Step 94 (Flask·JSON).

What you need: Python, and the code pieces you’ve made so far (scanner, DB integration). This chapter’s measurements used Python 3.12, with 127.0.0.1 (my own computer) as the only scan target.
Caution: ⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. Scan targets are limited to your own computer (127.0.0.1) and lab ranges you own. This chapter is a test — you’re given only a requirements specification, and no code to follow along with.

You’ve come through more than 50 steps so far. Python, web, networks, databases, Git, documentation. But real skill shows not in "I did the follow-along" — it shows here: building a finished product from requirements alone — that is skill. If you hit a stuck point partway, congratulations. It’s a signal telling you exactly where your hole is — go back to that step, review, and return.

1. Learning Objectives

By the end of this chapter, you will be able to:

Build a finished product from a requirements specification alone
Split a big problem into a pipeline (scan→store→report→compare) and verify each stage independently
Integrate socket scanning + SQLite storage + aggregate reporting + set comparison into one program
Discover and handle edge cases like "first run" yourself
Complete the habit of finishing work with a README and a retrospective document

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Python 3 (socket, sqlite3, sys, datetime — all built-in modules)
Today’s ingredients	Port scanning from Step 79, binding·lastrowid·GROUP BY from Step 93, JSON sense from Step 94
Concepts needed	IP range iteration, one-to-many table design, set operations, command-line arguments (sys.argv), change detection
Today’s artifacts	`netwatch.py` + `netwatch.db` + README.md + a wiki retrospective

2-1. What We’re Building — "NetWatch"

Corporate security teams have watchdog tools that catch "a port that wasn’t on our network yesterday is open today." If an intruder opens a backdoor or someone launches a server without permission — they catch it by change. What you’re building today is a scaled-down version of that, NetWatch: a tool that scans an address range, stores results in a DB, prints a report, and compares against the last scan to warn about "newly opened ports." Don’t underestimate it for being scaled down — the skeleton’s structure is identical to real-world tools.

2-2. How to Split a Big Problem — Think in a Pipeline

If you try to write the whole spec at once, you will collapse without fail. The expert’s secret is splitting. Think of the four stages below as independent small projects.

[Stage 1] Scan only — take a range, get the list of open ports into memory
[Stage 2] DB store — put that list into SQLite with a timestamp
[Stage 3] Report — read from the DB and print a summary
[Stage 4] Compare — diff against the previous scan and warn about new ports

Verify each stage independently. Only after stage 1’s output is printed and confirmed do you move to stage 2. "Write everything and run once" is the beginner’s trap; "write a little and keep checking" is the pro’s rhythm.

2-3. Requirements Specification (This Is All of Today’s Textbook)

■ Program name: netwatch.py
■ Run: python netwatch.py 127.0.0.1 127.0.0.1
  (takes a start IP and an end IP as arguments; prints usage if no arguments)

■ Feature 1 — Scan
  · For each IP in the range, attempt connections on major ports (~10: 22, 80, 443, 445, 3389, etc.)
  · For open ports, attempt banner collection (the greeting a service sends first) (0.5s limit)

■ Feature 2 — Store (netwatch.db)
  · Table scans: id, scan_time (time of scan)
  · Table findings: id, scan_id (which scan), ip, port, banner
  · Binding (?) required — no string concatenation (Step 93's lesson)

■ Feature 3 — Report
  · Print after the scan completes: number of devices found, total open ports, per-device open-port list

■ Feature 4 — Compare
  · If there are "newly opened ports (IP+port combos)" this time compared to the previous scan,
    print a warning in the form "[!] NEW: 127.0.0.1:8001"
  · If it's the first scan, a "baseline created" message

■ Finishing
  · README.md: tool introduction, usage, how it works, cautions (legal use)
  · Commit to a Git repository in meaningful units

Time Allocation Guide

Recommended split of the estimated 4 hours: stage 1 scan 50 min, stage 2 DB 50 min, stage 3 report 30 min, stage 4 compare 60 min, input validation·cleanup·documentation 50 min. It’s perfectly normal for a stage to take longer than planned — that stage is exactly your review point, so write one line about it in your wiki before moving on. This test’s score comes not from the finished product but from that map.

3. Follow Along

This time there’s no code to follow. Instead, here are checkpoints for verifying yourself whether you’ve passed each stage, and how to build the verification environment.

3-1. Preparing the Verification Environment — Launching Test Subjects

To verify a scan tool, "open ports" must exist in advance. Launch two or three test servers.

python -m http.server 8000 --bind 127.0.0.1     # terminal 1
python -m http.server 9000 --bind 127.0.0.1     # terminal 2

How to read it: open several terminal windows and launch one in each. When the experiment ends, be sure to shut each down with Ctrl+C in its window.

Why: before trusting a tool, build an environment that tests the tool — this order is the iron rule of real-world verification.

3-2. Checkpoint 1 — The Scan Stage

Pull out Step 79’s scanner code and wrap it in a function. scan(ip_list, ports) → returns a list of the open ones.
Making the IP range: just iterate the last octet (the fourth number when split on dots). A combination of ip.split('.') and range().
Verify: scan just 127.0.0.1 and confirm the servers you deliberately launched (8000, 9000) get caught. Don’t forget to add 8000 and 9000 to the port list.

3-3. Checkpoint 2 — The DB Store Stage

Put the CREATE statements that make the two tables at the program’s start. Attaching IF NOT EXISTS keeps it safe to run every time.
One scan = one row in scans. The way to pull out that scan’s id is cur.lastrowid (the number of the row just inserted).
Verify: after running, enter with sqlite3 netwatch.db and visually confirm with .tables and SELECT * FROM findings;.

3-4. Checkpoint 3 — The Report Stage

Step 93’s GROUP BY shines here. "Port count per device" is SELECT ip, COUNT(*) FROM findings WHERE scan_id=? GROUP BY ip.
Verify: is the output in a shape a human can read comfortably? Numbers should come first, like "1 device, 4 open ports total."

3-5. Checkpoint 4 — The Compare Stage (Today’s Brain Workout)

How do you find "the previous scan"? The second most recent in scans — think of ORDER BY id DESC LIMIT 2.
Set operations: subtracting the previous one from this scan’s collection of (ip, port) gives "newly opened ones." Python’s set is good at this job.
Verify: scan → scan again with no change (no warning should appear) → launch one more test server and scan ([!] NEW must appear). All three shots must be correct to pass.

3-6. Make a Prediction — The Trap in the Comparison Logic

Prediction: say you wrote code that simply compares this scan’s results with the previous scan’s when computing "newly opened ports." What happens when there is no previous scan (the program’s first run ever)?

(a) Every port gets warned as "NEW"
(b) It errors and dies
(c) It passes quietly

Check for yourself: delete the DB and do a first run. Depending on your code, it’ll be (a) or (b). That’s why the spec has "if it’s the first scan, a baseline-created message" — you need a branch that handles the exceptional situation first. In real work too, edge cases like "first run," "empty file," "list of 0 items" create half of all bugs.

3-7. Checkpoint 5 — Input Validation and Usage

A test also includes "cases where the user uses it weirdly."

When arguments are missing or only one: print usage and exit quietly (check sys.argv length).
When a non-IP argument (abc) comes in: exit with an error message. Check that the result of ip.split('.') is four numbers from 0–255.
When the start IP is larger than the end IP: flip them or refuse — whichever decision you make, write it in the README.

Verify: actually throw the three kinds of weird input and confirm the program doesn’t die but spits "guidance a human can read." Attackers always throw weird input — input validation isn’t a convenience feature, it’s a security feature.

3-8. Checkpoint 6 — Code Cleanup and Commit Units

Even when all features work, it’s not over. Polishing into code others can read is the final stage.

One line of comment per function — "what does this function return."
Gather numeric constants (port list, timeout) at the top of the file. They become easy to change later.
Commit in meaningful units: stamping separately like "scan feature," "DB store," "compare warning," "README" makes this project’s git log itself a page of your portfolio.

Verify: print git log --oneline and check whether each commit reads at a glance as "a commit that did what."

4. Missions & Exercises

Mission — Completing NetWatch and Verifying Three Scenarios

Complete netwatch.py with all of the spec’s features 1–4 working.
Verify all three scenarios: ① first run (baseline created) ② no change (no warning) ③ a new port appears ([!] NEW).
All SQL uses binding; argument validation and usage output are present.
Write introduction·usage·principle·legal-use cautions in README.md, and commit to a Git repository in meaningful units.
Write a retrospective in your wiki — where you got stuck, how you broke through.

Exercises

Exercise 1. Explain why scans and findings are split into two tables, using the words "one-to-many relationship" and "duplication."

Exercise 2. When computing "newly opened ports," explain why set subtraction (now - prev) is more convenient than lists.

Exercise 3. State the cause of the phenomenon where banner collection stalls on a certain port, and the two safety devices you must put on the socket.

Exercise 4. Give two examples of real-world watchdog tools besides port monitoring where this tool’s structure (store baseline → periodic comparison → difference warning) also applies.

5. Model Answers & Completion Criteria

Mission Model Answer

If your code works on its own, that is the correct answer. Below is reference commentary — the implementation actually used for this chapter’s hands-on verification. Don’t read it first; pick out only the parts for the stages where you’re stuck.

"""NetWatch — a tool that watches for port changes on my network (educational)"""
import socket, sqlite3, sys
from datetime import datetime

PORTS = [21, 22, 23, 25, 80, 110, 443, 445, 3306, 3389, 8000, 8001, 9000]
TIMEOUT = 0.3
DB = "netwatch.db"

def make_ip_range(start, end):
    """Turn a range like '127.0.0.1'~'127.0.0.3' into a list of IPs."""
    s, e = start.split("."), end.split(".")
    if len(s) != 4 or len(e) != 4:
        raise ValueError("not a valid IP format.")
    if not all(p.isdigit() and 0 <= int(p) <= 255 for p in s + e):
        raise ValueError("not a valid IP format.")
    prefix = ".".join(s[:3])
    return [f"{prefix}.{i}" for i in range(int(s[3]), int(e[3]) + 1)]

def scan(ip_list, ports):
    """Return open ports as a list of (ip, port, banner) tuples."""
    found = []
    for ip in ip_list:
        for port in ports:
            s = socket.socket()
            s.settimeout(TIMEOUT)
            try:
                s.connect((ip, port))
                banner = ""
                try:
                    banner = s.recv(64).decode("utf-8", "replace").strip()
                except socket.timeout:
                    pass
                found.append((ip, port, banner))
            except OSError:
                pass
            finally:
                s.close()
    return found

def save_scan(conn, found):
    """Store one scan into scans + findings and return the scan id."""
    cur = conn.cursor()
    cur.execute("INSERT INTO scans (scan_time) VALUES (?)",
                (datetime.now().isoformat(timespec="seconds"),))
    scan_id = cur.lastrowid
    for ip, port, banner in found:
        cur.execute(
            "INSERT INTO findings (scan_id, ip, port, banner) VALUES (?, ?, ?, ?)",
            (scan_id, ip, port, banner))
    conn.commit()
    return scan_id

def compare(conn, scan_id):
    """Compare against the previous scan and warn about newly opened ports."""
    ids = [r[0] for r in conn.execute(
        "SELECT id FROM scans ORDER BY id DESC LIMIT 2").fetchall()]
    if len(ids) < 2:
        print("baseline created — first scan, so there's no past to compare against.")
        return
    now = {(ip, p) for ip, p in conn.execute(
        "SELECT ip, port FROM findings WHERE scan_id=?", (ids[0],))}
    prev = {(ip, p) for ip, p in conn.execute(
        "SELECT ip, port FROM findings WHERE scan_id=?", (ids[1],))}
    for ip, port in sorted(now - prev):
        print(f"[!] NEW: {ip}:{port}")
    if now == prev:
        print("no change — identical to the previous scan.")

Three-scenario measurement (2026-09-09, target 127.0.0.1, test-subject servers 8000·9000 running):

### First scan (8000, 9000 open)
Scanning 1 target IP, 13 ports...
1 device found, 4 open ports total
  127.0.0.1: 22, 445, 8000, 9000
baseline created — first scan, so there's no past to compare against.

### Second scan (no change)
1 device found, 4 open ports total
  127.0.0.1: 22, 445, 8000, 9000
no change — identical to the previous scan.

### Third scan (after newly launching 8001)
1 device found, 5 open ports total
  127.0.0.1: 22, 445, 8000, 8001, 9000
[!] NEW: 127.0.0.1:8001

How to read it: watch two things. First, ports 22 and 445 — these aren’t ones I launched; they were originally open on the measurement computer. There were open doors I didn’t know about on my own computer — this is a watchdog tool’s reason for existence. Second, the banner column is empty — Python’s http.server is a service that doesn’t send a greeting first, so having no banner is normal (measured 2026-09-09). Banners only get collected from services that "speak first," like SSH or FTP.

Input validation was measured too — running without arguments prints usage: python3 netwatch.py <start IP> <end IP>, and feeding in abc prints input error: not a valid IP format. and exits quietly (2026-09-09).

Exercise Answers

Answer 1. The scan time is one per scan, but the discovered ports are many per scan — a one-to-many relationship. Cramming them into one table would duplicate the time across every row. So we split into scans (1) and findings (many) and link them by the scan_id number. Being able to paste them back with JOIN in the report is proof of this design.

Answer 2. Because "in A but not in B" is exactly what set difference means. With lists you’d have to ask "was this in the previous list?" one by one inside a double loop, while a set removes even order and duplicate worries in one line, now - prev. The measured implementation also finished with subtraction of two sets of {tuples}.

Answer 3. Because the service doesn’t send a banner and recv waits forever. Two safety devices: set a "maximum wait" with s.settimeout(0.3), and wrap in try/except to handle failure with an empty-string banner. Code that waits must always have a "maximum wait." And as we saw in the measurement, on services that don’t greet (http.server), an empty banner is normal — not treating an empty banner as an error is also part of the design.

Answer 4. Cousins with the same structure: ① file integrity monitoring — store the server’s file list/hashes as a baseline and warn about files secretly changed. ② account monitoring — compare whether an admin account that didn’t exist yesterday has appeared. Only the observed object changes; the skeleton of store baseline → periodic comparison → difference warning is identical.

Completion Criteria Checklist

[ ] I can complete a tool with four features from the spec alone
[ ] I can develop a program split into stages and verify each stage
[ ] I can discover and handle edge cases like "first run" myself
[ ] All SQL uses binding, and argument validation and usage output are present
[ ] I finished the work with a README and a retrospective document
[ ] I can state which step of Level 1 is my weak point
[ ] Mission: I completed verification of all three scenarios

6. Common Pitfalls & Fixes

Wall 1. The scan is too slow

Symptom: the whole range takes several minutes.

Cause: closed ports wait until connection refusal. A long timeout slows things multiplicatively.

Fix: keep the socket timeout short, 0.2–0.5 seconds (review Step 79). If it’s still slow, develop with fewer scan ports and increase after completion. The measured implementation finished within a few seconds with 127.0.0.1 × 13 ports at a 0.3s timeout.

Wall 2. lastrowid comes out None

Symptom: you inserted into scans but can’t get the id.

Cause: if you execute directly with conn.execute, you can’t catch the cursor. You must make a cursor with cur = conn.cursor(), run cur.execute(...), then read cur.lastrowid.

Fix: switch to the flow that makes a cursor explicitly (see save_scan in the model answer).

Wall 3. The comparison gets tangled — everything shows as NEW, or what should appear doesn’t

Symptom: a warning appears even though nothing changed.

Cause: a type mismatch in tuple comparison (port is a number on one side, a string on the other), or you grabbed the wrong scan_id.

Fix: print each of the two scans’ findings with SELECT and compare visually. Printing with print(now, prev) before putting them into sets is the fastest diagnosis.

Wall 4. The program stalls during banner collection

Symptom: it waits forever on a certain port.

Cause: the service doesn’t send a banner and recv waits indefinitely. In the measurement too, Python’s http.server didn’t send a banner — without a timeout it would have stalled right there.

Fix: set a socket timeout before recv, and wrap in try/except so failure means an empty-string banner. Code that waits must always have a "maximum wait."

Wall 5. The stuckness comes not from code but from the mind

Symptom: the situation itself — "a spec but no answer" — is so unfamiliar that your hands stop. This is the point in this chapter where the most people give up.

Cause: it’s normal. Exercises with a visible model answer and work with only a spec use different muscles.

Fix: three tactics. First, stuckness is data — write it in your wiki like "30 minutes at banner recv" and your weakness map gets completed. Second, experiment with small side scripts — peel off just the behavior you’re curious about in a five-line experiment file like test_lastrowid.py, confirm it, then transplant it into the main piece. Third, if you’re stuck on one problem for more than 40 minutes, going back to that step’s main text and redoing the follow-along is faster. It’s not a matter of pride — it’s a matter of strategy.

7. Summary

Today’s Concepts

Concept	One-line explanation
Change detection	Store baseline → periodic comparison → difference warning. The common skeleton of real-world watchdog tools
Pipeline splitting	A design splitting scan→store→report→compare into independently verifiable stages
One-to-many relationship	1 scan : N findings — split the tables in two and link by number
Set difference	`now - prev` — the precise definition of "newly opened"
Edge case	First run, empty list — the exceptional situations that create half of all bugs
Self-checking your attack surface	Your own computer also has open ports you don’t know about (22, 445 in the measurement)

Today’s Commands & Code

Code	What it does
`sys.argv`	Command-line arguments — length checking is the start of input validation
`socket.socket()` + `settimeout` + `connect`	Port connection attempts (reuse of Step 79)
`CREATE TABLE IF NOT EXISTS`	Table preparation safe to run every time
`cur.lastrowid`	The number of the row just INSERTed — the link between scans↔findings
`GROUP BY ip` + `COUNT(*)`	Per-device port count summary
`ORDER BY id DESC LIMIT 2`	The two most recent scans — the material for comparison
`set(...) - set(...)`	Compute newly opened ports

An Instinct More Important Than Commands

Building a finished product from a requirements specification alone — that is this test’s name and the definition of skill. Split big problems into a pipeline, verify each stage independently, and think of edge cases first. And before trusting a tool, build an environment that tests the tool. These four are habits that remain after this project ends.

If you’ve completed it, grade yourself — five items: scan accuracy, DB design, report readability, comparison logic, documentation, each 0–2 points; a total of 7 or more is a pass. Design isn’t a right answer — it’s choices and reasons. If you chose a design different from the spec, write the reason in the README — a different design with a reason isn’t wrong.

Did you pass? Level 1, complete. You opened the inside of a computer, eavesdropped on network conversations, talked with the warehouse keeper, built a server yourself, and finally completed a tool from a spec alone. The watchdog you made today is small, but its skeleton is exactly that of a real-world change detection system. Together with the promise that this tool’s scan targets are only ever your own computer and your own lab, close the door of Level 1.

Once every box is checked, Step 95 is complete. Click the checkbox in the sidebar to save your progress.