InfoSec 3001 — Information Security from the Offensive Perspective

Step 75. Crawling with BeautifulSoup — Pulling Only What You Want from HTML

Step 75. Crawling with BeautifulSoup — Pulling Only What You Want from HTML

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★☆☆ | Estimated time: 3 hours

Prerequisites: Step 74 complete; you can send requests and handle responses with requests, and you have the lab.py practice server. You know HTML tag structure and CSS selectors (Step 71).

  • What you need: Python, requests (already installed), beautifulsoup4 which we install today, and the lab.py from Step 74. No internet needed — today’s collection target is our home practice server again.
  • Caution: today’s practice is 100% safe. The crawling target is 127.0.0.1 — your own server only.

Last time we succeeded in fetching a whole page with requests. But what we fetched is just an HTML string of thousands of characters. How would we pull out only "the three quotes" from inside it? String searching hits its limit because tags are tangled together. BeautifulSoup is a library that turns this long string into a navigable tree structure. This technique is called crawling (crawling around and collecting), and it is a daily fundamental in security work — threat-intelligence gathering and leak monitoring.

1. Learning Objectives

By the end of this chapter, you will be able to:

  • Turn an HTML string into a navigable structure with BeautifulSoup(string, "html.parser")
  • Use the two families of search tools — find/find_all and CSS selectors (select/select_one) — appropriately for the situation
  • Pull text and attributes out of found elements with .text and ["href"]/get()
  • Write defensive code so the program doesn’t die when something isn’t found (None)
  • Organize collected items into a "list of dictionaries," a form ready for saving

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category Details
Language/environment Python 3 + beautifulsoup4 (pip install beautifulsoup4). Collection target: the lab.py server from Step 74 (127.0.0.1:8010)
Today’s tools The BeautifulSoup class, the find family, the select family
Today’s functions BeautifulSoup(html, "html.parser"), soup.find / find_all, soup.select / select_one, element.text, element.get("attribute")
Concepts needed HTML tag structure, CSS selectors (.class, #id, tag descendant), Python lists and dictionaries
Today’s output quotes.py — a quote collector

2-1. Parsing — The Magic of Turning a String into Structure

The work of turning a long string into a navigable structure is called parsing. Hand an HTML string to BeautifulSoup, and the same tree you saw in the DevTools Elements tab is built inside Python. Now orders like "give me the one h1 tag" or "give me all the a tags" become possible. The variable name soup is a convention.

2-2. Two Kinds of Search Tools

BeautifulSoup’s search tools come in two families.

  • The find family: soup.find("h1") returns the first single h1 tag; soup.find_all("a") returns all the a tags (as a list).
  • The select family: soup.select(".quote") searches using exactly the CSS selector syntax learned in Step 71. .name means class, #name means id. select_one finds only the first one.

From a found element, you pull the contents with .text (the text inside) and .get("href") (an attribute value). "Find the tag → pull out the text/attribute" — this two-beat rhythm is the whole of crawling.

2-3. Crawling Etiquette — robots.txt and Commas

Scraping data from someone else’s site puts a burden on their server. So sites post a notice at the door — robots.txt. Open "site address/robots.txt" and anyone can read the rules: "collecting is not permitted on these paths." Reading and respecting it, and inserting commas between requests with time.sleep — these two are both the collector’s etiquette and their credential. Today the partner is our home server, so there’s no burden, but habits are made in the practice ground.

3. Follow Along

Start the server first. In terminal 1, run python lab.py (the very file from Step 74). All the experiments below happen in terminal 2.

3-1. Installing and the First Parse

Input

pip install beautifulsoup4

Now we experiment in Python. Create soup1.py.

Input

import requests
from bs4 import BeautifulSoup

r = requests.get("http://127.0.0.1:8010/")
soup = BeautifulSoup(r.text, "html.parser")
print(soup.title.text)

Output (measured 2026-09-09):

Quote Practice Ground

How to read it: the one line BeautifulSoup(string, "html.parser") is the whole of parsing. From the parsed soup, soup.title means the title tag, and .text means the text inside it. The package name is beautifulsoup4, but the name you import in Python is bs4 — the first spot where beginners trip.

Why: this is a rite for confirming the moment a heap of string turns into a navigable structure. Everything gets easier after this one line.

3-2. find and find_all — One and All

Input

h1 = soup.find("h1")
print("First h1:", h1.text)

links = soup.find_all("a")
print("Number of links:", len(links))
for a in links:
    print(a.text.strip(), "->", a.get("href"))

Output (measured 2026-09-09):

First h1: Quotes of the Day
Number of links: 4
life -> /tag/life
wisdom -> /tag/wisdom
design -> /tag/design
wisdom -> /tag/wisdom

How to read it: find returns the first one; find_all returns all of them as a list. Since it’s a list, you can count with len() and loop over it. a.get("href") is the a tag’s href attribute — the link’s destination. All the tag links attached to lab.py’s quote page were caught.

Why: "pulling every link on a page" is crawling’s national example, and also the first step of security reconnaissance that draws a map of a site.

3-3. Sniping Precisely with CSS Selectors

One quote on our server had this structure.

<div class="quote">
  <span class="text">The greatest danger is a life without danger.</span>
  <small class="author">Elbert Hubbard</small>
  <div class="tags"><a class="tag" href="/tag/life">life</a></div>
</div>

Input

quotes = soup.select(".quote")
print("Number of quotes:", len(quotes))
for q in quotes:
    text = q.select_one(".text").text
    author = q.select_one(".author").text
    print(author, ":", text[:20], "...")

Output (measured 2026-09-09):

Number of quotes: 3
Elbert Hubbard : The greatest danger  ...
Proverb : Seeing once is bette ...
Antoine de Saint-Exupéry : Perfection is achiev ...

How to read it: we grabbed the three quote boxes whole with the class selector .quote, then sniped .text and .author again inside each box. This nested search — "grab the box first, then pull the parts out of it" — is the standard pattern of real-world crawling. The point is that a select result can serve again as the starting point of a select_one.

Why: this is training in picking out exactly "that part I want," not the whole page. The more precise the selector, the cleaner the harvest.

3-4. Predict — What Happens When You Search for Something That Isn’t There

Time to predict. If you search for a tag the page doesn’t have — say soup.find("marquee") (an old tag) — what happens? ① an error is raised ② an empty string ③ None is returned. And what if you immediately attach .text to that result?

Input

nothing = soup.find("marquee")
print("find result:", nothing)
print("find_all result:", soup.find_all("marquee"))
print(nothing.text)

Output (measured 2026-09-09):

find result: None
find_all result: []
AttributeError: 'NoneType' object has no attribute 'text'

How to read it: when find can’t find something, it quietly returns None (the nothing object), and only at the moment you attach .text to None does the error explode. find_all instead returns an empty list, so there is no error. Half of all crawling errors come from exactly this None — when the page structure differs from your expectation, find hands you None without a sound.

Why: that’s why the habit of checking "did I find it?" before .text is survival gear. Was your prediction right?

3-5. Organizing the Harvest into a Table — A List of Dictionaries

It’s a waste to print the extracted data and throw it away. Let’s organize it into a form that’s easy to save and analyze.

Input

rows = []
for q in soup.select(".quote"):
    text_el = q.select_one(".text")
    author_el = q.select_one(".author")
    if text_el is None or author_el is None:
        print("skipped")
        continue
    tags = [t.text for t in q.select(".tag")]
    rows.append({"author": author_el.text, "quote": text_el.text, "tags": tags})

print(len(rows), "items collected")
print(rows[0])
print(rows[2])

Output (measured 2026-09-09):

3 items collected
{'author': 'Elbert Hubbard', 'quote': 'The greatest danger is a life without danger.', 'tags': ['life']}
{'author': 'Antoine de Saint-Exupéry', 'quote': 'Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.', 'tags': ['design', 'wisdom']}

How to read it: we made each collected item into one dictionary (name: value) and stacked them neatly in a list. Notice the None check (if … continue) went in together — the lesson of 3-4 became code right away. Things like tags, where one box holds several, were gathered into a list.

Why: this trains separating "extracting" from "stacking." Screen output is for people; this list of dictionaries is for programs. Later, when saving to a CSV file, this structure becomes the table’s rows as-is.

3-6. Predict — Does soup Execute JS?

One last prediction. If a page’s data is not in the HTML but is drawn in later by JavaScript, will it be collected with requests + BeautifulSoup? ① yes ② no.

How to check yourself: add <script>document.body.innerHTML += "<p>Text drawn by JS</p>";</script> at the very bottom of lab.py’s QUOTES_PAGE, restart the server, and try finding that text — visible in the browser — with soup.find("p"). It is not caught in soup.

How to read it: requests fetches only the raw HTML string. What executes JS is the browser, not requests. So data that’s "visible in the browser but absent in soup" was drawn later by JS.

Why: when you meet such a site, there are two paths. Check in the DevTools Network tab whether that data arrives as a separate request (JSON) and request that address directly (the proper way), or use a browser automation tool (the heavy way). Knowing the cause is more than half the solution.

4. Missions & Exercises

Mission — Completing the Quote Collector

Create quotes.py to perform the following (the lab.py server is required).

  1. Collect quote, author, and tags from the first page at 127.0.0.1:8010 and print them in the format "1. author — quote (tags)"
  2. If an element can’t be found at any step, the program must not die — it prints "skipped" (None check)
  3. Print the number of collected items at the end, and verify by eye that the number matches the actual number of quotes on the server page
  4. At the top comment of the script, write this pledge: "Collection target: my practice server. For someone else’s site, checking robots.txt and time.sleep come first"
  5. (Challenge) Try saving the collected list of dictionaries to a quotes.csv file with import csv — the moment a list of dictionaries becomes table rows

Exercises

Q1. State the difference in return values between soup.find("h1") and soup.find_all("h1"), and explain why the latter is safe without errors.

Q2. In soup.select(".quote"), what does the dot (.) mean, and what do #menu and div.quote each search for?

Q3. In 3-4, find quietly returned None and the error exploded at .text. Explain why this "quietness" makes debugging hard.

Q4. What kind of data cannot be collected with requests + BeautifulSoup, and what are the two paths to check in that case?

5. Model Answers & Completion Criteria

Mission Model Answer

# Collection target: my practice server. For someone else's site, checking robots.txt and time.sleep come first
import csv
import requests
from bs4 import BeautifulSoup

r = requests.get("http://127.0.0.1:8010/", timeout=3)
soup = BeautifulSoup(r.text, "html.parser")

rows = []
for q in soup.select(".quote"):
    text_el = q.select_one(".text")
    author_el = q.select_one(".author")
    if text_el is None or author_el is None:
        print("skipped")
        continue
    tags = [t.text for t in q.select(".tag")]
    rows.append({"author": author_el.text.strip(),
                 "quote": text_el.text.strip(),
                 "tags": tags})

for i, row in enumerate(rows, 1):
    print(f"{i}. {row['author']} — {row['quote']} ({', '.join(row['tags'])})")
print("Total", len(rows), "items collected")

with open("quotes.csv", "w", newline="", encoding="utf-8-sig") as f:
    writer = csv.DictWriter(f, fieldnames=["author", "quote", "tags"])
    writer.writeheader()
    writer.writerows(rows)

Execution result (measured 2026-09-09):

1. Elbert Hubbard — The greatest danger is a life without danger. (life)
2. Proverb — Seeing once is better than hearing a hundred times. (wisdom)
3. Antoine de Saint-Exupéry — Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. (design, wisdom)
Total 3 items collected

How to verify: ① Does the printed count (3) match the number of div.quote blocks in lab.py’s QUOTES_PAGE? ② Running it with the server off raises requests’ ConnectionError — handling even this gracefully (Step 74’s try/except) is icing on the cake. ③ When you open quotes.csv, is the text intact? — utf-8-sig is the encoding choice that keeps non-ASCII characters from breaking in Excel.

Exercise Solutions

Q1 solution. find returns the first single element (None if absent); find_all returns a list of all of them (an empty list [] if absent) (measured 2026-09-09). For an empty list, a loop or len() simply "runs zero times" with no error, so it doesn’t die even when the absent case occurs.

Q2 solution. The dot (.) means "search by the alias called class." #menu searches for the one element whose id is menu, and div.quote searches for "among div tags, those whose class is quote" — exactly the CSS selector syntax from Step 71.

Q3 solution. Because the error explodes not at "the real cause (the moment it wasn’t found)" but at "the moment its result is used." The longer the code between the two, the harder the cause is to trace. That’s why checking for None immediately after find is the fix — it drags the error back near its cause.

Q4 solution. Data that JavaScript draws in later — because requests fetches only the raw HTML and does not execute JS (the 3-6 experiment). There are two paths: ① check in the Network tab whether that data arrives as a separate JSON request and request that address directly with requests, ② if real browser execution is needed, use a browser automation tool.

Completion Criteria Checklist

  • [ ] I can parse an HTML string with BeautifulSoup
  • [ ] I can explain the difference between find and find_all (one/None vs list/empty list)
  • [ ] I can snipe the elements I want with CSS selectors (select, select_one)
  • [ ] I can pull text and attributes with .text and .get("href")
  • [ ] I can explain the need for a None check and apply it in code
  • [ ] I can organize collected items into a list of dictionaries
  • [ ] Mission: I completed the quote collector

6. Common Pitfalls & Fixes

Wall 1. The import itself fails

Symptom: ModuleNotFoundError: No module named 'bs4'

Cause: you didn’t install beautifulsoup4, or you installed it into a different Python. For the record, the package name (beautifulsoup4) and the import name (bs4) being different is also a seed of confusion.
Fix: run python -m pip install beautifulsoup4 with the very Python you’re using. This makes the "which Python was it installed into?" problem disappear.

Wall 2. NoneType errors keep happening

Symptom (measured 2026-09-09): AttributeError: 'NoneType' object has no attribute 'text'

Cause: find/select_one found nothing, but you called .text right away. The page structure differs from your expectation.
Fix: make the pattern if element is None: skip right after finding a habit. And re-inspect the actual raw HTML with your eyes (print(r.text)).

Wall 3. The text is clearly on the screen, but it won’t be collected

Symptom: data visible in the browser can’t be found in soup.
Cause: that data was drawn later by JS, not present in the HTML (the 3-6 experiment). requests fetches only the raw document.
Fix: check in the Network tab whether that data arrives as a separate request (JSON). If it comes as JSON, requesting that address directly is the correct answer.

Wall 4. The selector grabs less than expected

Symptom: you expected 3 but only 1 comes — or conversely, junk comes along too.
Cause: similar class names are mixed into other elements, or the selector is too narrow or too wide.
Fix: experiment while narrowing the selector one step at a time. Writing the tag and class together, like "div.quote", raises accuracy. The trick is printing print(len(…)) as you tune.

Wall 5. Strange whitespace comes along with the text

Symptom: the extracted text has heaps of newlines and spaces at its front and back.
Cause: the HTML source’s indentation came along as-is.
Fix: attaching strip (removing front/back whitespace), like .text.strip(), is crawling’s basic finishing touch. It’s in the mission model answer too.

7. Summary

Today’s Concepts

Concept One-line description
Parsing The work of turning a long string into a navigable tree structure
BeautifulSoup An HTML parsing library — its import name is bs4
Crawling/scraping The act of roaming pages / the act of tearing data out
None find’s quiet answer when it can’t find — must check before .text
robots.txt The collection-rules notice at a site’s door — the collector’s first thing to check
Nested search The standard pattern: grab the box (.quote) first, then pull out the parts (.text) inside

Today’s Functions

Function What it does
BeautifulSoup(html, "html.parser") HTML string → navigable soup
soup.find("tag") The first one (None if absent)
soup.find_all("tag") All as a list ([] if absent)
soup.select("selector") All matching a CSS selector
soup.select_one("selector") The first one matching a CSS selector
element.text The text inside the tag
element.get("href") Pull out an attribute value
element.text.strip() With front/back whitespace removed

The Instinct That Matters More Than Commands

Today’s core instinct is "look at the structure before extracting." The procedure of successful crawling is always the same — ① check the raw document (print(r.text)) ② grasp the structure (which tag, which class) ③ snipe with a selector ④ None check ⑤ organize into a list of dictionaries. And remember — if a site has an official window for handing out data (an API), that is the proper way, better than crawling. There’s no reason to climb the wall when the front gate is open. The security connection: "collect → structure → watch" is the daily pattern of security work like threat-intelligence gathering and leak monitoring, and as great as that power is, etiquette (checking robots.txt, time.sleep, authorized targets) is the credential. You now hold both "the hand that fetches (requests)" and "the hand that extracts (BeautifulSoup)" on the web.

Once every box is checked, Step 75 is complete.

← 이전 스텝
다음 스텝 →