Step 76. Crawlers and Automation — A Collector That Works by Itself Every Day

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★★☆ | Estimated time: 4 hours

Prerequisites: Steps 74–75 complete; you can send requests with requests and extract data with BeautifulSoup. You can read and write CSV files in Python (Step 48).

• What you need: Python, requests and beautifulsoup4 (the ones installed in Step 74), and half a day of leisure. Today we also build the practice website ourselves — no separate installation, just Python.
• Caution: today’s practice is limited to an owned local practice environment. The only site the crawler visits is a practice ground you build yourself inside your own computer (127.0.0.1). It never connects to an external site, not even once.

Until now, our scripts were toys that “work once when switched on, then stop.” A real tool is different. It works while its owner sleeps, remembers what it gathered yesterday so it doesn’t fetch the same thing twice, and quietly leaves a record when it fails. Today’s project is evolving the toy into a tool. Page traversal, CSV accumulation, deduplication, error isolation, automatic execution — these five are the basic skeleton of security work, usable as-is whether the target is quote collection or vulnerability-bulletin collection.

1. Learning Objectives

By the end of this chapter, you will be able to:

• Launch the practice site on loopback.
• Traverse a bounded set of pages and reject cycles or outside links.
• Deduplicate against old records and the current batch.
• Record failures and return a nonzero exit status.
• Optionally configure and verify a single scheduled instance.

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. Page Traversal — Turning Until the End Appears

A long list doesn’t fit on one page. You must follow the “next page” link. The skeleton of this traversal is always the same: collect one page → find the next link → stop if there is none. One while loop does it.

2-2. Accumulation and Deduplication — Using a set

Run it every day, and today you gather again what you gathered yesterday. The way to stop that is simple. At startup, read the existing CSV and build a set of “items already present”; if something newly collected is in that set, skip it. A set is a data structure that answers “is it there or not” at lightning speed, so it stays fast even against tens of thousands of entries.

Report Collection Failures

The corrected entry point records a failure and exits with a nonzero status. It saves rows only after a complete successful traversal. Returning an empty list on a failed request would hide the difference between failure and a valid empty result.

2-4. Scheduling — The Computer’s Alarm Clock

Every operating system has a device that runs programs automatically at set times.

• Linux/WSL: cron. Write 0 9 * * * command in the timetable you open with crontab -e, and it means “run every day at 9:00.” The five fields are minute, hour, day, month, weekday, and * means “every.”
• Windows: Task Scheduler. Find “Task Scheduler” in the Start menu and register via “Create Basic Task.”

Either way, the core caution is the same: you must use absolute paths. Automatic execution starts without knowing “which folder it’s running from,” unlike when you double-click a file yourself.

3. Follow Along

3-1. Building the Practice Site — A Mini Web Inside My Computer

Instead of an external site, we use a site we build ourselves as the collection target: a three-page “quote practice ground.” Make a folder called site in your working folder and create three files inside it.

Input (site/index.html)

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 1</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"The world is a book, and those who do not travel read only one page."</span>
  <small class="author">Saint Augustine</small>
</div>
<div class="quote">
  <span class="text">"Life is what happens when you're busy making other plans."</span>
  <small class="author">John Lennon</small>
</div>
<div class="quote">
  <span class="text">"Security is a process, not a product."</span>
  <small class="author">Bruce Schneier</small>
</div>
<ul class="pager">
  <li class="next"><a href="/page2.html">Next →</a></li>
</ul>
</body>
</html>

Input (site/page2.html)

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 2</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"Simplicity is the soul of efficiency."</span>
  <small class="author">Austin Freeman</small>
</div>
<div class="quote">
  <span class="text">"The only truly secure system is one that is powered off."</span>
  <small class="author">Gene Spafford</small>
</div>
<ul class="pager">
  <li class="next"><a href="/page3.html">Next →</a></li>
</ul>
</body>
</html>

Input (site/page3.html) — the point is that, being the last page, it has no “next” link.

<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground Page 3</title></head>
<body>
<h1>Quote Practice Ground</h1>
<div class="quote">
  <span class="text">"Talk is cheap. Show me the code."</span>
  <small class="author">Linus Torvalds</small>
</div>
<div class="quote">
  <span class="text">"Attackers know no rules. Defenders must follow them."</span>
  <small class="author">Anonymous</small>
</div>
<ul class="pager">
</ul>
</body>
</html>

Now let’s serve this folder as a website. From the folder directly above site (your working folder):

Input (terminal 1 — do not close this window)

python -m http.server 8800 --bind 127.0.0.1 --directory site

Output (measured 2026-09-09):

Serving HTTP on 0.0.0.0 port 8800 (http://0.0.0.0:8800/) ...

How to read it: http.server is a mini web server built into Python. --directory site means “make the site folder the site root,” and 8800 is the port number. Open http://127.0.0.1:8800 in a browser; if quotes appear, it’s a success. If you run python -m http.server from inside the site folder, you don’t need --directory.

Why: this structure cuts off at the source any chance of accidentally burdening someone else’s site while learning crawlers. We now have our own practice ground where the rules never change and we can collect freely as many times as we like.

3-2. A Page-Traversing Crawler — Turning to the End

Save this listing as crawler_core.py.

import time
from urllib.parse import urljoin, urlsplit
import requests
from bs4 import BeautifulSoup

BASE = "http://127.0.0.1:8800"


def collect_all(base=BASE, max_pages=20):
    origin = urlsplit(base)
    expected = (origin.scheme, origin.hostname, origin.port)
    if expected[0] != "http" or expected[1] != "127.0.0.1":
        raise ValueError("this exercise permits the owned loopback server only")
    rows, visited = [], set()
    url = base + "/"
    with requests.Session() as session:
        session.trust_env = False  # no environment proxy for this local lab
        while url:
            target = urlsplit(url)
            if (target.scheme, target.hostname, target.port) != expected:
                raise ValueError("next link leaves the permitted origin")
            if target.username or target.password:
                raise ValueError("credentials in URLs are not permitted")
            url = target._replace(fragment="").geturl()
            if url in visited or len(visited) >= max_pages:
                raise ValueError("pagination cycle or page limit")
            visited.add(url)
            with session.get(url, timeout=(2, 3), allow_redirects=False,
                             stream=True) as response:
                if response.status_code != 200:
                    raise ValueError(f"unexpected HTTP {response.status_code}")
                chunks, size = [], 0
                for chunk in response.iter_content(8192):
                    size += len(chunk)
                    if size > 262144:
                        raise ValueError("page exceeds 256 KiB")
                    chunks.append(chunk)
                body = b"".join(chunks).decode("utf-8", errors="strict")
            soup = BeautifulSoup(body, "html.parser")
            for quote in soup.select(".quote"):
                text = quote.select_one(".text")
                author = quote.select_one(".author")
                if text is not None and author is not None:
                    rows.append({"author": author.get_text(strip=True),
                                 "quote": text.get_text(strip=True)})
            nxt = soup.select_one("li.next a")
            href = nxt.get("href") if nxt else None
            url = urljoin(url, href) if href else None
            if url:
                time.sleep(0.2)
    return rows


if __name__ == "__main__":
    data = collect_all()
    print(f"Collected {len(data)} items")

Input (terminal 2 — leave the server running)

python crawler_core.py

Output (measured 2026-09-09):

Collected 7 items

How to read it: the while loop runs “while there is a next-page address (url).” It collects one page → finds the next address at li.next a → rests 0.2 seconds → moves to the next page. Page 3 has no next link, so nxt becomes None and the loop ends. 3 + 2 + 2 = 7 items is correct.

Why: “collect, find the next, stop if there is none” — this while’s shape is reused as-is in every pagination crawler. Yesterday’s “collect one page” became today’s “traverse to the end.”

3-3. Predict — Will the Same Quote Be Collected Twice?

Time to predict. If the same quote appears on two pages of the site, how does it go into data? And how can we notice it?

Check yourself: add one line to __main__.

print("Deduplication check:", len(data), "->",
      len({(row["author"], row["quote"]) for row in data}))

Output (measured 2026-09-09):

Collected 7 items
Deduplication check: 7 -> 7

How to read it: if the collected count and the count after deduplication via set are equal, it means “there were no duplicates inside the site.” If they differ, the set filtered the duplicates for you. Our practice ground is clean, so it’s 7 -> 7.

Why: “collected” and “organized” are different things. This one line of verification habit will shine in the next step’s CSV accumulation.

3-4. CSV Accumulation and Deduplication

Save the CSV functions in a second file, crawler_archive.py, in the same folder. The identity key is (author, quote), including duplicates inside the current batch.

Input: crawler_archive.py

import csv
from pathlib import Path
from datetime import date

CSV_FILE = Path(__file__).resolve().with_name("quotes.csv")


def load_seen(path=CSV_FILE):
    if not path.exists():
        return set()
    with path.open(newline="", encoding="utf-8") as fp:
        return {(r["author"], r["quote"]) for r in csv.DictReader(fp)}


def save_new(rows, path=CSV_FILE):
    # One writer at a time; not a concurrent database replacement.
    seen = load_seen(path)
    fresh = []
    for row in rows:
        key = (row["author"], row["quote"])
        if key not in seen:
            seen.add(key)  # also deduplicate within this batch
            fresh.append(row)
    if not fresh:
        return 0
    new_file = not path.exists() or path.stat().st_size == 0
    with path.open("a", newline="", encoding="utf-8") as fp:
        writer = csv.writer(fp)
        if new_file:
            writer.writerow(["date", "author", "quote"])
        for row in fresh:
            writer.writerow([date.today().isoformat(),
                             row["author"], row["quote"]])
    return len(fresh)

For a first successful run, create crawler_run.py with these imports and calls.

from crawler_core import collect_all
from crawler_archive import save_new

if __name__ == "__main__":
    rows = collect_all()
    print("New rows:", save_new(rows))

Input: try running it twice in a row.

python crawler_run.py
python crawler_run.py

Output (measured 2026-09-09):

Collected 7 items, newly saved 7 items
Collected 7 items, newly saved 0 items

How to read it: on the first run all 7 are new and get saved; on the second, all of them already exist, so it’s 0. load_seen reads the existing CSV into a set beforehand, and save_new picks only what’s not in that set and appends it to the end of the file ("a" is append mode).

Why: that second output — “collected 7, saved 0” — is the very evidence that an accumulation system is alive. Yesterday’s run and today’s run are talking to each other through the CSV file. Open quotes.csv and you’ll find the header and 7 lines (measured 2026-09-09):

date,author,quote
2026-09-09,Saint Augustine,"""The world is a book, and those who do not travel read only one page."""
2026-09-09,John Lennon,"""Life is what happens when you're busy making other plans."""

(The quotes inside values appearing doubled is CSV’s notation rule. It’s normal.)

Report Collection Failures

Replace crawler_run.py with the complete failure-reporting entry point below. Keep the other two files.

Input: crawler_run.py (complete replacement)

from pathlib import Path
from datetime import datetime, timezone
import requests
from crawler_core import collect_all
from crawler_archive import save_new


def main():
    try:
        rows = collect_all()
        print("New rows:", save_new(rows))
        return 0
    except (requests.RequestException, OSError, ValueError) as exc:
        path = Path(__file__).resolve().with_name("crawler_error.log")
        with path.open("a", encoding="utf-8") as fp:
            fp.write(f"{datetime.now(timezone.utc).isoformat()} "
                     f"collection failed: {type(exc).__name__}\n")
        print("Collection failed; no empty-success result was saved")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

Stop the owned practice server, run python crawler_run.py, and inspect the nonzero exit status and error log. A failed collection must not be recorded as a successful empty collection.

Expected pattern with the server stopped (exact error type depends on the environment):

Collection failed; no empty-success result was saved

The program returns exit status 1 and writes an error category to crawler_error.log.

2026-09-09 collection failed: ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8800): Max retries exceeded with url: / (Caused by NewConnectionError(... [WinError 10061] No connection could be made because the target machine actively refused it"))

The entry point catches the supported failures and exits unsuccessfully. save_new runs only after a complete successful traversal, so a network failure does not append a misleading empty-success record.

Why: an unmanaged automation is someday silently stopped and nobody knows. The log is the device that breaks that silence. This one function separates “toy” from “tool.”

3-6. Registering Automatic Execution — For Tomorrow Morning’s Me

The last piece, ⑤. Do only one, depending on your OS. Registration itself depends on your clock and account, so the outputs below are examples.

Linux (including WSL) — input

crontab -e

Content to enter (add at the bottom line of the file)

0 9 * * * cd /absolute/lab && /absolute/lab/.venv/bin/python crawler_run.py >> crawler_run.log 2>&1

Replace both absolute paths with your own lab and its Python environment. Keep all three scripts together. CSV and error-log locations are anchored to the script folder. Configure only one scheduled instance at a time; this CSV example has no cross-process writer lock. WSL scheduling also requires the distribution and scheduler to be running.

On Windows Task Scheduler, select the lab environment’s Python executable and pass the absolute path to crawler_run.py as its argument. Keep crawler_core.py and crawler_archive.py beside it. Set the working directory explicitly, prevent overlapping runs, and verify the exit status and log.

Why: once registered, tomorrow morning the program works without you touching the keyboard. After registering, proper form is to move the time 2–3 minutes ahead to test “does it really run by itself,” then set it back to the original time after confirming. When the test is done, turn off the practice site server too — if the server is off at automatic-run time, the error isolation and logging you just built will work in its place.

4. Missions & Exercises

Mission — Completing the Collector and Making a Variant

1. Complete all three crawler files, verify CSV deduplication and failure logging, then optionally configure and verify a single scheduled run.
2. Add page4.html to the site folder (link /page4.html from page 3’s pager), rerun the crawler, and confirm that “only the new page’s quotes” are added to the CSV
3. Write a separate script stats.py that reads quotes.csv and prints “number of quotes per author”
4. Record the commands you ran and the observed results (how many items were newly saved) in a README

Exercises

Q1. What is the condition that stops the page-traversal while loop, and which part of our practice site creates that condition?

Q2. Explain the principle by which “newly saved 0 items” comes out on the second run, in terms of the roles of load_seen and save_new.

Q3. Explain why automatic execution must use absolute paths, from the perspective of the “execution folder.”

Q4. Explain what happens when you register a crawler without error isolation to cron, using the expression “silent failure.”

import csv
from collections import Counter

with open("quotes.csv", newline="", encoding="utf-8") as fp:
    rows = list(csv.DictReader(fp))

counts = Counter(row["author"] for row in rows)
for author, n in counts.most_common():
    print(f"{author}: {n} items")

6. Common Pitfalls & Fixes

Wall 1. The crawler dies with a ConnectionError

Symptom (measured 2026-09-09):

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8800): Max retries exceeded with url: / ... [WinError 10061] No connection could be made because the target machine actively refused it

Ensure your own loopback server is running, then rerun crawler_run.py. A failed run is logged and returns a failure status; it does not pretend to have collected zero valid rows.

Wall 2. The server won’t start, saying “address already in use”

Symptom (measured 2026-09-09):

OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

Cause: a previously launched http.server is still holding port 8800. A window you never closed with Ctrl+C is alive somewhere.
Fix: find and close the server window, or switch to another port (8801, etc.) — and don’t forget to change the crawler’s BASE port to match.

Wall 3. Non-ASCII text breaks when opening the CSV in Excel

For Excel, import the UTF-8 CSV through its text/CSV import controls. If you deliberately switch to UTF-8 with BOM, update both reading and writing to utf-8-sig and handle existing files consistently; changing only the writer can break header-based deduplication. Excel behavior varies by version and import method.

Wall 4. Duplicates keep piling up

Symptom: you ran it twice, yet the same quotes are saved again.
Cause: the comparison basis and the saved value differ subtly. If you skipped strip() during collection and values were saved with front/back whitespace, the next run compares them against “whitespace-free values” and treats them as different.
Fix: check the strip at the collection step, and verify that the column load_seen reads (“quote”) and the column you save use the same basis.

Wall 5. cron silently does nothing

Symptom: you registered it, but the CSV doesn’t change as time passes.
Cause: mostly a path problem. cron runs without knowing where your folder is.
Fix: change not just the script path but also the file paths inside the script (CSV_FILE, log) to absolute paths, and append >> logfile 2>&1 to the end of the cron line to give silent failure an ear. Pulling the time 2–3 minutes ahead right after registering is the standard way to verify.

7. Summary

Today’s Concepts

Today’s Commands and Functions

The Instinct That Matters More Than Commands

The skeleton we completed today — traversal, accumulation, deduplication, error isolation, automatic execution — becomes a “threat-intelligence collector” just by changing the target. Security bulletins, CVE updates, monitoring mentions of our organization — all of it stands on these five pieces. Conversely, aimed at someone else’s service without permission, this skeleton becomes burden and trespass, so collection targets must always be “places where collecting is allowed” (practice grounds you built like today’s, places robots.txt and the terms of service permit).

The completion of automation includes not “making it run” but “the habit of checking.” That 10 seconds tomorrow morning — opening quotes.csv to see the line count, checking that crawler_error.log is empty — is the automation owner’s job. And when the data grows to tens of thousands of items and needs searching, that’s the time to move the CSV to a database (SQLite, later in this book) — the skeleton stays the same; only the storehouse changes.

Once every box is checked, Step 76 is complete.