Step 76. Crawlers and Automation — A Collector That Works by Itself Every Day

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★★☆ | Estimated time: 4 hours

Prerequisites: Steps 74–75 complete; you can send requests with requests and extract data with BeautifulSoup. You can read and write CSV files in Python (Step 48).

What you need: Python, requests and beautifulsoup4 (the ones installed in Step 74), and half a day of leisure. Today we also build the practice website ourselves — no separate installation, just Python.
Caution: today’s practice is 100% safe. The only site the crawler visits is a practice ground you build yourself inside your own computer (127.0.0.1). It never connects to an external site, not even once.

Until now, our scripts were toys that "work once when switched on, then stop." A real tool is different. It works while its owner sleeps, remembers what it gathered yesterday so it doesn’t fetch the same thing twice, and quietly leaves a record when it fails. Today’s project is evolving the toy into a tool. Page traversal, CSV accumulation, deduplication, error isolation, automatic execution — these five are the basic skeleton of security work, usable as-is whether the target is quote collection or vulnerability-bulletin collection.

1. Learning Objectives

By the end of this chapter, you will be able to:

Launch a local practice site with python -m http.server
Write the while-loop skeleton of a crawler that follows "next page" to the very end
Read an existing CSV into a set and accumulate only new data
Build a collector that "never dies" using try/except and a log file
Register a script with cron or Task Scheduler, and explain why absolute paths are a lifeline

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Python 3 + requests, BeautifulSoup4 (installed in Steps 74–75)
Today’s tools	`python -m http.server` (practice site), the while traversal loop, the `csv` module, `set`, cron / Task Scheduler
Concepts needed	Pagination, accumulation, deduplication, error isolation, absolute paths
Today’s output	`crawler.py` + `quotes.csv` + automatic execution registration — a collector that works by itself every day

2-1. Page Traversal — Turning Until the End Appears

A long list doesn’t fit on one page. You must follow the "next page" link. The skeleton of this traversal is always the same: collect one page → find the next link → stop if there is none. One while loop does it.

2-2. Accumulation and Deduplication — Using a set

Run it every day, and today you gather again what you gathered yesterday. The way to stop that is simple. At startup, read the existing CSV and build a set of "items already present"; if something newly collected is in that set, skip it. A set is a data structure that answers "is it there or not" at lightning speed, so it stays fast even against tens of thousands of entries.

2-3. Error Isolation — A Program That Never Dies

The network can fall ill at any time. If an automatically running program dies on a single error, days pass with nobody knowing. So we wrap the collection part in try/except, and on failure write the date and the reason to a log file, then return an empty result. Failing without dying, and leaving the failure behind as a record — this is the line dividing a toy from a tool.

2-4. Scheduling — The Computer’s Alarm Clock

Every operating system has a device that runs programs automatically at set times.

Linux/WSL: cron. Write 0 9 * * * command in the timetable you open with crontab -e, and it means "run every day at 9:00." The five fields are minute, hour, day, month, weekday, and * means "every."
Windows: Task Scheduler. Find "Task Scheduler" in the Start menu and register via "Create Basic Task."

Either way, the core caution is the same: you must use absolute paths. Automatic execution starts without knowing "which folder it’s running from," unlike when you double-click a file yourself.

3. Follow Along

3-1. Building the Practice Site — A Mini Web Inside My Computer

Instead of an external site, we use a site we build ourselves as the collection target: a three-page "quote practice ground." Make a folder called site in your working folder and create three files inside it.

Input (site/index.html)

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 1</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"The world is a book, and those who do not travel read only one page."</span>
  <small class="author">Saint Augustine</small>
</div>
<div class="quote">
  <span class="text">"Life is what happens when you're busy making other plans."</span>
  <small class="author">John Lennon</small>
</div>
<div class="quote">
  <span class="text">"Security is a process, not a product."</span>
  <small class="author">Bruce Schneier</small>
</div>
<ul class="pager">
  <li class="next"><a href="/page2.html">Next →</a></li>
</ul>
</body>
</html>

Input (site/page2.html)

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 2</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"Simplicity is the soul of efficiency."</span>
  <small class="author">Austin Freeman</small>
</div>
<div class="quote">
  <span class="text">"The only truly secure system is one that is powered off."</span>
  <small class="author">Gene Spafford</small>
</div>
<ul class="pager">
  <li class="next"><a href="/page3.html">Next →</a></li>
</ul>
</body>
</html>

Input (site/page3.html) — the point is that, being the last page, it has no "next" link.

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 3</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"Talk is cheap. Show me the code."</span>
  <small class="author">Linus Torvalds</small>
</div>
<div class="quote">
  <span class="text">"Attackers know no rules. Defenders must follow them."</span>
  <small class="author">Anonymous</small>
</div>
<ul class="pager">
</ul>
</body>
</html>

Now let’s serve this folder as a website. From the folder directly above site (your working folder):

Input (terminal 1 — do not close this window)

python -m http.server 8800 --directory site

Output (measured 2026-09-09):

Serving HTTP on 0.0.0.0 port 8800 (http://0.0.0.0:8800/) ...

How to read it: http.server is a mini web server built into Python. --directory site means "make the site folder the site root," and 8800 is the port number. Open http://127.0.0.1:8800 in a browser; if quotes appear, it’s a success. If you run python -m http.server from inside the site folder, you don’t need --directory.

Why: this structure cuts off at the source any chance of accidentally burdening someone else’s site while learning crawlers. We now have our own practice ground where the rules never change and we can collect freely as many times as we like.

3-2. A Page-Traversing Crawler — Turning to the End

Input (crawler.py)

import time
import requests
from bs4 import BeautifulSoup

BASE = "http://127.0.0.1:8800"

def collect_all():
    rows = []
    url = "/"
    while url:
        r = requests.get(BASE + url, timeout=5)
        soup = BeautifulSoup(r.text, "html.parser")
        for q in soup.select(".quote"):
            text_el = q.select_one(".text")
            author_el = q.select_one(".author")
            if text_el is None or author_el is None:
                continue
            rows.append({"author": author_el.text.strip(),
                         "quote": text_el.text.strip()})
        nxt = soup.select_one("li.next a")
        url = nxt.get("href") if nxt else None
        time.sleep(0.2)
    return rows

if __name__ == "__main__":
    data = collect_all()
    print(f"Collected {len(data)} items")

Input (terminal 2 — leave the server running)

python crawler.py

Output (measured 2026-09-09):

Collected 7 items

How to read it: the while loop runs "while there is a next-page address (url)." It collects one page → finds the next address at li.next a → rests 0.2 seconds → moves to the next page. Page 3 has no next link, so nxt becomes None and the loop ends. 3 + 2 + 2 = 7 items is correct.

Why: "collect, find the next, stop if there is none" — this while’s shape is reused as-is in every pagination crawler. Yesterday’s "collect one page" became today’s "traverse to the end."

3-3. Predict — Will the Same Quote Be Collected Twice?

Time to predict. If the same quote appears on two pages of the site, how does it go into data? And how can we notice it?

Check yourself: add one line to __main__.

    print("Deduplication check:", len(data), "->", len(set(r["quote"] for r in data)))

Output (measured 2026-09-09):

Collected 7 items
Deduplication check: 7 -> 7

How to read it: if the collected count and the count after deduplication via set are equal, it means "there were no duplicates inside the site." If they differ, the set filtered the duplicates for you. Our practice ground is clean, so it’s 7 -> 7.

Why: "collected" and "organized" are different things. This one line of verification habit will shine in the next step’s CSV accumulation.

3-4. CSV Accumulation and Deduplication

Pieces ② and ③. Add two functions to crawler.py and fix __main__.

Input (add to crawler.py)

import csv
import os
from datetime import date

CSV_FILE = "quotes.csv"

def load_seen():
    seen = set()
    if os.path.exists(CSV_FILE):
        with open(CSV_FILE, newline="", encoding="utf-8") as fp:
            for row in csv.DictReader(fp):
                seen.add(row["quote"])
    return seen

def save_new(rows):
    seen = load_seen()
    today = date.today().isoformat()
    fresh = [r for r in rows if r["quote"] not in seen]
    is_new_file = not os.path.exists(CSV_FILE)
    with open(CSV_FILE, "a", newline="", encoding="utf-8") as fp:
        writer = csv.writer(fp)
        if is_new_file:
            writer.writerow(["date", "author", "quote"])
        for r in fresh:
            writer.writerow([today, r["author"], r["quote"]])
    return len(fresh)

Input (make the __main__ part like this)

if __name__ == "__main__":
    data = collect_all()
    added = save_new(data)
    print(f"Collected {len(data)} items, newly saved {added} items")

Input: try running it twice in a row.

python crawler.py
python crawler.py

Output (measured 2026-09-09):

Collected 7 items, newly saved 7 items
Collected 7 items, newly saved 0 items

How to read it: on the first run all 7 are new and get saved; on the second, all of them already exist, so it’s 0. load_seen reads the existing CSV into a set beforehand, and save_new picks only what’s not in that set and appends it to the end of the file ("a" is append mode).

Why: that second output — "collected 7, saved 0" — is the very evidence that an accumulation system is alive. Yesterday’s run and today’s run are talking to each other through the CSV file. Open quotes.csv and you’ll find the header and 7 lines (measured 2026-09-09):

date,author,quote
2026-09-09,Saint Augustine,"""The world is a book, and those who do not travel read only one page."""
2026-09-09,John Lennon,"""Life is what happens when you're busy making other plans."""

(The quotes inside values appearing doubled is CSV’s notation rule. It’s normal.)

3-5. Error Isolation — A Crawler That Never Dies

Piece ④, the survival gear. Add one more function to crawler.py.

Input (add to crawler.py)

def safe_collect():
    try:
        return collect_all()
    except requests.RequestException as e:
        with open("crawler_error.log", "a", encoding="utf-8") as fp:
            fp.write(f"{date.today().isoformat()} collection failed: {type(e).__name__}: {e}\n")
        return []

Change __main__ to call safe_collect() instead of collect_all(). Now let’s verify. Stop the server in terminal 1 with Ctrl+C and run it.

Output (measured 2026-09-09, with the server off):

Collected 0 items, newly saved 0 items

The program exited normally without dying. And a crawler_error.log file has been created (measured 2026-09-09):

2026-09-09 collection failed: ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8800): Max retries exceeded with url: / (Caused by NewConnectionError(... [WinError 10061] No connection could be made because the target machine actively refused it"))

How to read it: it caught the communication error (ConnectionError) thrown by requests, wrote the date and reason to the log, and returned an empty list. With an empty list, save_new has nothing to save, and the program ends quietly. Also note that on a localized Windows, the tail of the error message appears in the local language.

Why: an unmanaged automation is someday silently stopped and nobody knows. The log is the device that breaks that silence. This one function separates "toy" from "tool."

3-6. Registering Automatic Execution — For Tomorrow Morning’s Me

The last piece, ⑤. Do only one, depending on your OS. Registration itself depends on your clock and account, so the outputs below are examples.

Linux (including WSL) — input

crontab -e

Content to enter (add at the bottom line of the file)

0 9 * * * /usr/bin/python3 /home/username/crawler.py >> /home/username/crawler_run.log 2>&1

How to read it: one line meaning "every day at 9:00, with the Python at the absolute path, run the script at the absolute path, and append output and errors to the run log." Replace the username part with the real path — you can check it with pwd. It’s also proper form to change the CSV_FILE and log paths inside the script to absolute paths.

Windows — Task Scheduler: search "Task Scheduler" in the Start menu → "Create Basic Task" on the right → name "QuoteCollector" → trigger "Daily" → start time 09:00 → action "Start a program" → put the full path of python.exe in Program, the full path of crawler.py in Arguments, and the full path of your working folder in Start in.

Why: once registered, tomorrow morning the program works without you touching the keyboard. After registering, proper form is to move the time 2–3 minutes ahead to test "does it really run by itself," then set it back to the original time after confirming. When the test is done, turn off the practice site server too — if the server is off at automatic-run time, the error isolation and logging you just built will work in its place.

4. Missions & Exercises

Mission — Completing the Collector and Making a Variant

Run 3-1 through 3-6 to completion, ending up with crawler.py + quotes.csv + an error log + automatic execution registration
Add page4.html to the site folder (link /page4.html from page 3’s pager), rerun the crawler, and confirm that "only the new page’s quotes" are added to the CSV
Write a separate script stats.py that reads quotes.csv and prints "number of quotes per author"
Record the commands you ran and the observed results (how many items were newly saved) in a README

Exercises

Q1. What is the condition that stops the page-traversal while loop, and which part of our practice site creates that condition?

Q2. Explain the principle by which "newly saved 0 items" comes out on the second run, in terms of the roles of load_seen and save_new.

Q3. Explain why automatic execution must use absolute paths, from the perspective of the "execution folder."

Q4. Explain what happens when you register a crawler without error isolation to cron, using the expression "silent failure."

5. Model Answers & Completion Criteria

Mission Model Answer

After adding page4.html and connecting the pager, running it saves only the new page’s quotes, since the 7 items through page 3 already exist. If you put 2 quotes on page 4, "Collected 9 items, newly saved 2 items" is the correct answer — the meaning of accumulation shows exactly: collection redoes everything, but saving keeps only the new.

The skeleton of stats.py:

import csv
from collections import Counter

with open("quotes.csv", newline="", encoding="utf-8") as fp:
    rows = list(csv.DictReader(fp))

counts = Counter(row["author"] for row in rows)
for author, n in counts.most_common():
    print(f"{author}: {n} items")

Running it prints per-author counts like Saint Augustine: 1 items. With our practice-ground data, everyone has 1 item each.

How to verify: ① Is the new-item count exact on the rerun after adding page4? ② Does the line count of quotes.csv match header + accumulated items? ③ Does it survive with the server off, leaving log entries? ④ Does the crontab or Task Scheduler registration contain only absolute paths? If all four are "yes," it’s complete.

Exercise Solutions

Q1 solution. "There is no next-page link" is the stop condition. When soup.select_one("li.next a") returns None, url becomes None and the while ends. In our practice site, page3.html’s empty <ul class="pager"> creates that condition — page 3 has no li.next.

Q2 solution. load_seen reads the existing quotes.csv and builds a set of "quotes already saved." save_new filters out from the newly collected list whatever is already in that set (fresh) and appends only the remainder to the file. On the second run, all 7 collected items are in the set, so fresh is empty and it becomes 0 items.

Q3 solution. When you run python crawler.py in a terminal, "the folder you’re currently in" becomes the reference, so relative paths like quotes.csv resolve. But when cron or Task Scheduler runs it, the reference folder is different or undefined, so it can’t find relative-path files or creates new files in the wrong place. An absolute path points at the same file whatever the reference folder is, so it is the lifeline of automatic execution.

Q4 solution. A network error kills the crawler with an exception, and cron repeats that tomorrow and the day after. If you didn’t redirect output to a log, the error message remains nowhere, and the owner believes for days that "it must be running fine." This is silent failure. That’s why error isolation — swallowing the failure with try/except but leaving it in a log — is essential.

Completion Criteria Checklist

[ ] I can launch a local practice site with python -m http.server
[ ] I can explain the page-traversal while-loop skeleton (collect → find next → stop if none)
[ ] I ran it twice and confirmed "newly saved 0 items"
[ ] I ran it with the server off and confirmed an error log is left
[ ] I registered automatic execution in cron or Task Scheduler using absolute paths
[ ] Mission: I completed incremental saving for page4 and per-author aggregation

6. Common Pitfalls & Fixes

Wall 1. The crawler dies with a ConnectionError

Symptom (measured 2026-09-09):

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8800): Max retries exceeded with url: / ... [WinError 10061] No connection could be made because the target machine actively refused it

Cause: the practice site server (http.server) is not running. With no collection target, the connection is refused.
Fix: relaunch the server in terminal 1. And this error is exactly the case 3-5’s safe_collect was designed to catch — once isolation is attached, it doesn’t die but flows into the log.

Wall 2. The server won’t start, saying "address already in use"

Symptom (measured 2026-09-09):

OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

Cause: a previously launched http.server is still holding port 8800. A window you never closed with Ctrl+C is alive somewhere.
Fix: find and close the server window, or switch to another port (8801, etc.) — and don’t forget to change the crawler’s BASE port to match.

Wall 3. Non-ASCII text breaks when opening the CSV in Excel

Symptom: it’s fine in Notepad but breaks only in Excel.
Cause: a chronic disease of Windows Excel, which fails to recognize UTF-8 without a BOM (same root as the code-table problem in Step 50).
Fix: change the save encoding to encoding="utf-8-sig", and an invisible mark is added at the very front of the file so Excel recognizes it. Note that pure utf-8 may be better when exchanging with other programs that don’t know this mark.

Wall 4. Duplicates keep piling up

Symptom: you ran it twice, yet the same quotes are saved again.
Cause: the comparison basis and the saved value differ subtly. If you skipped strip() during collection and values were saved with front/back whitespace, the next run compares them against "whitespace-free values" and treats them as different.
Fix: check the strip at the collection step, and verify that the column load_seen reads ("quote") and the column you save use the same basis.

Wall 5. cron silently does nothing

Symptom: you registered it, but the CSV doesn’t change as time passes.
Cause: mostly a path problem. cron runs without knowing where your folder is.
Fix: change not just the script path but also the file paths inside the script (CSV_FILE, log) to absolute paths, and append >> logfile 2>&1 to the end of the cron line to give silent failure an ear. Pulling the time 2–3 minutes ahead right after registering is the standard way to verify.

7. Summary

Today’s Concepts

Concept	One-line description
Pagination	Splitting a long list across multiple pages — traverse by following the "next" link
Accumulation	Reading the existing store and appending only the new (set + append)
Deduplication	Skipping already-collected items via a set’s "is it there" judgment
Error isolation	try/except + logging — fail without dying and leave a record
Scheduling	cron (Linux) / Task Scheduler (Windows) — absolute paths are the lifeline
Silent failure	The state where log-less automation has stopped and nobody knows — prevented by logging

Today’s Commands and Functions

Command/function	What it does
`python -m http.server 8800 --directory site`	Serve a folder as a local website
`soup.select(".quote")` / `select_one("li.next a")`	Find quote blocks / the next-page link
`csv.DictReader` / `csv.writer`	Read CSV (like dictionaries) / write
`open(..., "a", ...)`	Append to the end of a file
`date.today().isoformat()`	Today’s date in "2026-09-09" form
`crontab -e`	Edit the Linux automatic-execution timetable

The Instinct That Matters More Than Commands

The skeleton we completed today — traversal, accumulation, deduplication, error isolation, automatic execution — becomes a "threat-intelligence collector" just by changing the target. Security bulletins, CVE updates, monitoring mentions of our organization — all of it stands on these five pieces. Conversely, aimed at someone else’s service without permission, this skeleton becomes burden and trespass, so collection targets must always be "places where collecting is allowed" (practice grounds you built like today’s, places robots.txt and the terms of service permit).

The completion of automation includes not "making it run" but "the habit of checking." That 10 seconds tomorrow morning — opening quotes.csv to see the line count, checking that crawler_error.log is empty — is the automation owner’s job. And when the data grows to tens of thousands of items and needs searching, that’s the time to move the CSV to a database (SQLite, later in this book) — the skeleton stays the same; only the storehouse changes.

Once every box is checked, Step 76 is complete. Click the checkbox in the sidebar to save your progress.