Step 74. The requests Library — Talking to the Web with Python

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★☆☆ | Estimated time: 3 hours

Prerequisites: Step 73 complete; you know the HTTP request/response structure, GET and POST, headers, and status codes. You can write and run Python scripts and have used pip.

What you need: Python, two terminals (one for the server + one for experiments). An internet connection is not needed — today again, our conversation partner is a localhost server we build ourselves.
Caution: today’s practice is 100% safe. Every request travels only inside 127.0.0.1.

Last time we talked with a server using curl. One request per command line. Excellent, but limited — scraping a hundred pages means typing the command a hundred times, and pulling just the values you want out of a response means finding them by eye. Without automation we are not collectors but typing slaves. requests is the most famous library for sending HTTP requests from Python, and with this one library you handle GET/POST, parameters, headers, and cookies in a few lines of code. This library is the beating heart of web scanners, collectors, and testing tools.

1. Learning Objectives

By the end of this chapter, you will be able to:

Build your own practice server (lab.py) in Python that understands POST and cookies
Send requests with requests.get/post and read the response object’s status_code, headers, and text
Distinguish and use params (address) and data (body) in code
Experiment with the principle of "acting continuously as if logged in" by keeping cookies in a Session
Write request code that never hangs, using timeout and exception handling

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Python 3 + the requests library (`pip install requests`). Practice partner: our self-built server at 127.0.0.1:8010
Today’s tools	`requests.get/post`, the response object, `requests.Session`, and the `http.server` module on the server side
Today’s functions	`requests.get(URL, params=, headers=, timeout=)`, `requests.post(URL, data=)`, `r.status_code / r.headers / r.text / r.json() / r.url / r.history`
Concepts needed	Request/response structure (Step 73), Python dictionaries, cookies and sessions
Today’s output	`lab.py` — a practice server, `analyze.py` — a response analyzer

2-1. The Response Object, a Gift Box

What comes back when you call requests.get(...) is a response object. Think of it as "a gift box with many pockets (attributes) attached."

Attribute	Contents	Example
`r.status_code`	The status code (number)	200
`r.headers`	Response headers (used like a dictionary)	`r.headers["Content-Type"]`
`r.text`	The response body (string)	`"<!DOCTYPE html>..."`
`r.json()`	Unpacks the body as JSON and returns a dictionary	Only for JSON responses
`r.url`	The final address the request actually went to	For checking redirects
`r.history`	The record of redirects passed through	For tracing the route

What you read with your eyes as > and < in curl -v last time has now become attributes you pull out with a dot (.) in code.

2-2. params and data — A Difference of Riding Position

The GET/POST difference learned in Step 73 splits like this in code.

params={"id": "admin"} → attached after the address (?id=admin). This is a GET request’s data.
data={"pw": "1234"} → goes into the body. This is a POST request’s data.

Hand over a Python dictionary (a bundle of name-value pairs) as-is, and requests packs it into HTTP format by itself. There is no need to assemble ?id=admin&pw=1234 by hand.

2-3. Sessions — The Hand That Keeps Presenting the ID

HTTP is originally a protocol "with no memory." The server treats each request as a separate guest. Even after a successful login, the next request is treated as a stranger again. What solves this is the cookie (an ID slip issued by the server), and what automatically takes care of cookies in code is requests.Session(). Make requests through a session object, and the cookie received in the first request is automatically attached to the next one — an imitation of what the browser has been doing.

3. Follow Along

3-1. Installing and Building the Practice Server

Input

pip install requests

This time, let’s build the server that will be our conversation partner. Last chapter’s python -m http.server only knew GET. This server also receives POST and plants cookies. Create lab.py.

from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.parse import urlparse, parse_qs

QUOTES_PAGE = """<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground</title></head>
<body>
  <h1>Quotes of the Day</h1>
  <div class="quote">
    <span class="text">The greatest danger is a life without danger.</span>
    <small class="author">Elbert Hubbard</small>
    <div class="tags"><a class="tag" href="/tag/life">life</a></div>
  </div>
  <div class="quote">
    <span class="text">Seeing once is better than hearing a hundred times.</span>
    <small class="author">Proverb</small>
    <div class="tags"><a class="tag" href="/tag/wisdom">wisdom</a></div>
  </div>
  <div class="quote">
    <span class="text">Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.</span>
    <small class="author">Antoine de Saint-Exupéry</small>
    <div class="tags"><a class="tag" href="/tag/design">design</a> <a class="tag" href="/tag/wisdom">wisdom</a></div>
  </div>
</body>
</html>"""

class LabHandler(BaseHTTPRequestHandler):
    def _send(self, code, body, ctype="text/html; charset=utf-8", extra=None):
        data = body.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        for k, v in (extra or {}).items():
            self.send_header(k, v)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        path = urlparse(self.path).path
        if path == "/":
            self._send(200, QUOTES_PAGE)
        elif path == "/headers":
            lines = [f"{k}: {v}" for k, v in self.headers.items()]
            self._send(200, "\n".join(lines), "text/plain; charset=utf-8")
        elif path == "/set":
            self._send(200, "A cookie has been planted.",
                       extra={"Set-Cookie": "session_id=abc123; Path=/"})
        elif path == "/who":
            cookie = self.headers.get("Cookie", "(no cookie)")
            self._send(200, f"Cookie received by the server: {cookie}",
                       "text/plain; charset=utf-8")
        elif path == "/old":
            self.send_response(301)
            self.send_header("Location", "/")
            self.end_headers()
        else:
            self._send(404, "<h1>404 — no such page</h1>")

    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        raw = self.rfile.read(length).decode("utf-8")
        fields = parse_qs(raw)
        flat = {k: v[0] for k, v in fields.items()}
        self._send(200, f"POST body received by the server: {raw}\nUnpacked result: {flat}",
                   "text/plain; charset=utf-8")

if __name__ == "__main__":
    print("Practice server started: http://127.0.0.1:8010  (stop: Ctrl+C)")
    HTTPServer(("127.0.0.1", 8010), LabHandler).serve_forever()

Input (terminal 1 — for the server)

python lab.py

Output (measured 2026-09-09):

Practice server started: http://127.0.0.1:8010  (stop: Ctrl+C)

How to read it: don’t be intimidated. It’s long, but what it does is simple — a set of branches that returns a fixed answer for each path. /headers is a mirror that returns the headers it received, /set is a counter that plants a cookie, /who is a counter that shows the cookie it received, and /echo is a counter that returns the POST body. We’ll keep using this server in this chapter and the crawling chapter.

Why: building a server yourself leaves in your body the fact that "a server is a program that reads the request line and headers and returns a fixed response." To see a server through an attacker’s eyes, you must once have stood in the server’s position.

3-2. The First Request and Dissecting the Response

Open terminal 2 and create req1.py.

Input

import requests

r = requests.get("http://127.0.0.1:8010/")
print("Status code:", r.status_code)
print("Content-Type:", r.headers["Content-Type"])
print("First 60 chars of body:", repr(r.text[:60]))

Output (measured 2026-09-09):

Status code: 200
Content-Type: text/html; charset=utf-8
First 60 chars of body: '<!DOCTYPE html>\n<html>\n<head><meta charset="utf-8"><title>Quote'

How to read it: with three lines of code a GET request flew off, and from the returned box we pulled out and printed the status code, a header, and the body. The body is the very HTML we wrote into lab.py. Last chapter’s single curl line has merely become a few lines of code — but what differs is that this code can now go inside a loop.

3-3. Predict — Where on the Address Do the params Attach?

Time to predict. In the code below, what will the final address printed in r.url look like?

Input

r = requests.get("http://127.0.0.1:8010/headers", params={"q": "security", "page": "2"})
print("Final address:", r.url)

Output (measured 2026-09-09):

Final address: http://127.0.0.1:8010/headers?q=security&page=2

How to read it: the two name-values passed as a dictionary attached after the ?, joined with &. requests built for us the GET address we used to assemble by hand. For the record, our server’s /headers ignores the tail of the address and returns only the headers — but as you can see, the address itself flew off fully assembled.

Why: this is the identity of the phenomenon where the address grows long when you type a word into a search box. And the first question of web testing — "if I change this address’s parameters, will the server respond differently?" — is born right here.

3-4. POST and Header Disguise

Input

r = requests.post("http://127.0.0.1:8010/echo", data={"id": "admin", "pw": "1234"})
print(r.text)

r2 = requests.get("http://127.0.0.1:8010/headers",
                  headers={"User-Agent": "MySecurityStudyBot/1.0"})
print(r2.text)

Output (measured 2026-09-09):

POST body received by the server: id=admin&pw=1234
Unpacked result: {'id': 'admin', 'pw': '1234'}

Host: 127.0.0.1:8010
User-Agent: MySecurityStudyBot/1.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

How to read it: the first part is evidence that the data riding in the POST body arrived at the server intact and was unpacked — you can also see that it traveled in name=value&name=value form. The second part is the list of headers the server saw. The User-Agent I changed shows up as-is, and the headers requests attaches by default (Accept-Encoding, Connection, etc.) appear together.

Why: this is the stage where you gain the conviction that "every part of a request can be assembled by me." The essence of a web testing tool is exactly this freedom of assembly.

3-5. Sessions — An Experiment in Whether Cookies Persist

Input

s = requests.Session()
s.get("http://127.0.0.1:8010/set")          # the counter that plants a cookie
r = s.get("http://127.0.0.1:8010/who")      # check with the same session
print("With session:", r.text)

r2 = requests.get("http://127.0.0.1:8010/who")   # a fresh request without a session
print("Without session:", r2.text)

Output (measured 2026-09-09):

With session: Cookie received by the server: session_id=abc123
Without session: Cookie received by the server: (no cookie)

How to read it: after planting a cookie through the session, asking again through the same session shows the cookie alive. The request sent without a session has an empty cookie. This is direct evidence that the session plays the role of an "ID wallet."

Why: when doing "consecutive actions that require authentication" in code — login → post list → write post — a session is essential. Carrying this flow in one session is the skeleton of web automation. Conversely, if an attacker steals someone’s cookie (via XSS, for example), they can impersonate that person without a password — that is the weight of calling a cookie an ID.

3-6. Predict — How Is a Redirect Recorded?

Our server’s /old answers only "moved away (301), the new address is /". If we request it with requests, where is the final destination, and where does the record remain?

Input

r = requests.get("http://127.0.0.1:8010/old")
print("Final address:", r.url)
print("Notices passed through:", [(x.status_code, x.url) for x in r.history])

Output (measured 2026-09-09):

Final address: http://127.0.0.1:8010/
Notices passed through: [(301, 'http://127.0.0.1:8010/old')]

How to read it: unlike curl, requests follows redirects by default. And r.history keeps the footprints it passed through (the 301 notice). "Where did it finally arrive" and "by what route did it come" show up separately.

Why: checking a shortened link’s real destination, tracing a phishing site’s detour route — all such analysis starts from this history attribute.

3-7. Code That Never Hangs — timeout and Exception Handling

Let’s attach the survival gear of automation tools.

Input

import requests

try:
    r = requests.get("http://127.0.0.1:59999/", timeout=3)
except requests.exceptions.ConnectionError:
    print("Request failed: cannot reach the server")

try:
    r = requests.get("http://127.0.0.1:8010/", timeout=3)
    print("Success:", r.status_code)
except requests.exceptions.ConnectionError:
    print("Request failed: cannot reach the server")

Output (measured 2026-09-09):

Request failed: cannot reach the server
Success: 200

How to read it: there is no server on port 59999, so a ConnectionError was raised, and we caught it and printed a notice. timeout=3 is a device meaning "if there’s no answer within 3 seconds, give up." Had we not caught the exception, the program would have died with a long error message — the real message looks like ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=59999): Max retries exceeded ... (measured 2026-09-09).

Why: without a timeout, requests waits forever if the server stays silent. This one line stops a tool scraping a hundred pages from freezing forever at the thirty-seventh.

4. Missions & Exercises

Mission — Build a Response Analyzer

Create analyze.py to perform the following (the lab.py server must be running).

Send a GET request to /headers with two params attached, and print the final URL and the status code
From the response body, pick out and print only the User-Agent line the server saw
Change the User-Agent to your own string, send the same request again, and confirm by output that the identity the server saw has changed
Send three name-value pairs by POST to /echo, and print the server’s unpacked result
Wrap every request in try/except with timeout=3, so that even with the server off it leaves a "request failed" notice and exits normally

Exercises

Q1. Explain the difference between params and data in terms of "where the data rides" and "which method each goes with."

Q2. You called r.json() and got JSONDecodeError: Expecting value: line 1 column 1 (char 0) (measured 2026-09-09). What situation is this, and what should you check before calling it?

Q3. If you do a login and a post in sequence using only requests.get(...) without a session, why does the login fall off? Explain using the concept of cookies.

Q4. Explain the difference between r.url and r.history in terms of "the final destination" and "the road traveled."

5. Model Answers & Completion Criteria

Mission Model Answer

import requests

BASE = "http://127.0.0.1:8010"

try:
    # 1. Attaching params
    r = requests.get(f"{BASE}/headers", params={"q": "security", "page": "2"}, timeout=3)
    print("Final URL:", r.url)
    print("Status code:", r.status_code)

    # 2. Picking out the User-Agent line the server saw
    for line in r.text.splitlines():
        if line.startswith("User-Agent"):
            print("UA seen by the server:", line)

    # 3. Re-checking after disguise
    r2 = requests.get(f"{BASE}/headers",
                      headers={"User-Agent": "MyAnalyzer/2.0"}, timeout=3)
    for line in r2.text.splitlines():
        if line.startswith("User-Agent"):
            print("UA seen by the server after disguise:", line)

    # 4. POST
    r3 = requests.post(f"{BASE}/echo",
                       data={"id": "admin", "pw": "1234", "memo": "first practice"},
                       timeout=3)
    print(r3.text)

except requests.exceptions.ConnectionError:
    print("Request failed: check that the lab.py server is running")
except requests.exceptions.Timeout:
    print("Request failed: the server did not answer within 3 seconds")

Execution result (measured 2026-09-09, partial):

Final URL: http://127.0.0.1:8010/headers?q=security&page=2
Status code: 200
UA seen by the server: User-Agent: python-requests/2.33.1
UA seen by the server after disguise: User-Agent: MyAnalyzer/2.0
POST body received by the server: id=admin&pw=1234&memo=first+practice
Unpacked result: {'id': 'admin', 'pw': '1234', 'memo': 'first practice'}

With no disguise at all, requests reports itself as python-requests/version — this too is one of today’s discoveries. And remember how non-ASCII text in a POST body would change into URL-encoded form like %EC%B2%AB.... Form data is transmitted URL-encoded (converted into characters that can ride on an address), and the server unpacks it again for use. The encoding sense from Step 50 is at work here too.

How to verify: ① When run with the server off, does the program leave a notice instead of dying? ② Are the User-Agent lines different before and after the disguise? ③ Do all three POST name-tags appear in the unpacked result? If all three are "yes," it’s complete.

Exercise Solutions

Q1 solution. params rides after the address (?name=value) and goes mostly with GET, while data rides in the request body and goes mostly with POST. Pass either as a dictionary and requests packs it into HTTP format.

Q2 solution. It’s a situation where the response body is not JSON — a case like our practice server’s /, where HTML came back (char 0 means it wasn’t JSON from the very first character). Before calling json(), you need the habit of looking at r.status_code and peeking at the front of the body with r.text[:200].

Q3 solution. HTTP has no memory between requests, so the proof of a successful login (the session cookie) must be presented again with the next request. Calling requests.get(...) fresh each time leaves no wallet to hold the cookie, so you become a stranger every time. The Session object is that wallet (3-5 measurement: the session kept session_id=abc123, while the session-less request had an empty cookie).

Q4 solution. r.url is the final address after all redirects have been followed, and r.history is the list of notices (301, etc.) passed through along the way. As in the 3-6 measurement, the final address is / while history retains (301, .../old).

Completion Criteria Checklist

[ ] I can build, start, and stop the lab.py practice server
[ ] I can send requests with requests.get/post and read status_code and headers
[ ] I can explain the difference between params and data (address vs body)
[ ] I confirmed by experiment that cookies persist in a Session
[ ] I habitually attach timeout and try/except
[ ] I can trace redirects with r.url and r.history
[ ] Mission: I completed the response analyzer

6. Common Pitfalls & Fixes

Wall 1. I ran the client without starting the server

Symptom (measured 2026-09-09):

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8010): Max retries exceeded ... (Caused by NewConnectionError(... [WinError 10061] No connection could be made because the target machine actively refused it))

Cause: the conversation partner (lab.py) is not running.
Fix: start the server first with python lab.py in terminal 1. The server window must stay open.

Wall 2. The port is blocked when starting the server

Symptom: OSError: [WinError 10048] ... — the specified port is already in use.
Cause: a previous server that never stopped, or another program, is occupying that number.
Fix: change lab.py’s 8010 to something like 8020, and change the address in the client code to match.

Wall 3. r.json() throws an error

Symptom (measured 2026-09-09): requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Cause: the response body is not JSON (an HTML page, etc.). It also happens often when the status code is not 200.
Fix: before json(), check that r.status_code == 200 and look at the front of the body first with print(r.text[:200]).

Wall 4. A request sits frozen for a long time

Symptom: get() hasn’t returned for several minutes.
Cause: the other side isn’t responding, and requests waits indefinitely by default.
Fix: make timeout=5 a habit. If there’s no answer, it gives up with an error. In an automation tool, waiting forever is the same as being stopped.

Wall 5. I definitely received a cookie, but it vanishes on the next request

Symptom: you planted a cookie via /set, but /who returns "(no cookie)".
Cause: you sent each request fresh with requests.get(...). There is no wallet for the cookie to be stored in.
Fix: create s = requests.Session() and send every request as s.get(...), s.post(...) (the 3-5 measurement is the evidence).

7. Summary

Today’s Concepts

Concept	One-line description
requests	Python’s representative HTTP client library
Response object	A gift box with pockets: status_code, headers, text, json(), and more
params / data	Riding on the address (GET) / riding in the body (POST)
Cookie	An ID slip issued by the server — planted via the response’s Set-Cookie, presented via the request’s Cookie header
Session	A wallet object that automatically stores and attaches cookies
timeout	"Give up if no answer within this time" — the survival gear of automation

Today’s Functions

Function/attribute	What it does
`requests.get(URL, params=, headers=, timeout=)`	GET request
`requests.post(URL, data=, timeout=)`	POST request
`r.status_code` / `r.headers` / `r.text`	Status code / response headers / body string
`r.json()`	JSON body → Python dictionary (only when it’s JSON!)
`r.url` / `r.history`	Final address / list of redirects passed through
`requests.Session()`	Create a cookie wallet
`python lab.py`	Start the practice server (stop: Ctrl+C)

The Instinct That Matters More Than Commands

Today’s core instinct is the habit of doubting responses. Beginners believe "a response came back, so it succeeded," but a practitioner’s order of checks is fixed — ① look at status_code ② print the first 200 characters of r.text ③ only then json() or parse. These three steps erase half of all "why isn’t this working?" debugging time. The security connection: what you learned today is dangerous if abused. Code that hammers the same request without rest becomes a service disruption by itself, so putting a comma like time.sleep(1) between repeated requests is an attitude that comes before technique. And remember the lab.py you built yourself today — a server, in the end, is "a set of branches that reads a request and puts out a fixed answer." With that picture in hand, no new web technology will ever frighten you.

Once every box is checked, Step 74 is complete. Click the checkbox in the sidebar to save your progress.