Step 74. The requests Library — Talking to the Web with Python

Level 1 — Programming and the Computer’s Insides | Difficulty ★★★☆☆ | Estimated time: 3 hours

Prerequisites: Step 73 complete; you know the HTTP request/response structure, GET and POST, headers, and status codes. You can write and run Python scripts and have used pip.

• What you need: Python, two terminals (one for the server + one for experiments). An internet connection is not needed — today again, our conversation partner is a localhost server we build ourselves.
• Caution: today’s practice is limited to an owned local practice environment. Every request travels only inside 127.0.0.1.

Last time we talked with a server using curl. One request per command line. Excellent, but limited — scraping a hundred pages means typing the command a hundred times, and pulling just the values you want out of a response means finding them by eye. Without automation we are not collectors but typing slaves. requests is the most famous library for sending HTTP requests from Python, and with this one library you handle GET/POST, parameters, headers, and cookies in a few lines of code. This library is the beating heart of web scanners, collectors, and testing tools.

1. Learning Objectives

By the end of this chapter, you will be able to:

• Build your own practice server (lab.py) in Python that understands POST and cookies
• Send requests with requests.get/post and read the response object’s status_code, headers, and text
• Distinguish and use params (address) and data (body) in code
• Experiment with the principle of “acting continuously as if logged in” by keeping cookies in a Session
• Handle connection/read timeouts and report failures explicitly

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. The Response Object, a Gift Box

What comes back when you call requests.get(...) is a response object. Think of it as “a gift box with many pockets (attributes) attached.”

What you read with your eyes as > and < in curl -v last time has now become attributes you pull out with a dot (.) in code.

2-2. params and data — A Difference of Riding Position

Method semantics and parameter placement are separate. Requests uses params for the URL query and data for request content; a POST can use both.

• params={“id”: “admin”} adds a URL query, including on a POST request.
• data={“pw”: “dummy”} form-encodes a dictionary in the body in this POST example. Use fictional inputs only.

Hand over a Python dictionary (a bundle of name-value pairs) as-is, and requests packs it into HTTP format by itself. There is no need to assemble ?id=admin&pw=1234 by hand.

2-3. Sessions — The Hand That Keeps Presenting the ID

HTTP treats requests independently. Applications may use cookies or other credentials to associate requests with state. requests.Session retains matching cookies and shared configuration across requests. A cookie is not necessarily an authentication credential: this lab only demonstrates persistence.

3. Follow Along

3-1. Installing and Building the Practice Server

Input

pip install requests

Create lab.py below. Unlike the preceding SimpleHTTPRequestHandler example, which supports GET and HEAD, this handler adds selected POST and cookie demonstrations. It is a local teaching fixture.

from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.parse import urlparse, parse_qs

QUOTES_PAGE = """<!DOCTYPE html>
<html>
<head><meta charset="utf-8"><title>Quote Practice Ground</title></head>
<body>
  <h1>Quotes of the Day</h1>
  <div class="quote">
    <span class="text">The greatest danger is a life without danger.</span>
    <small class="author">Elbert Hubbard</small>
    <div class="tags"><a class="tag" href="/tag/life">life</a></div>
  </div>
  <div class="quote">
    <span class="text">Seeing once is better than hearing a hundred times.</span>
    <small class="author">Proverb</small>
    <div class="tags"><a class="tag" href="/tag/wisdom">wisdom</a></div>
  </div>
  <div class="quote">
    <span class="text">Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.</span>
    <small class="author">Antoine de Saint-Exupéry</small>
    <div class="tags"><a class="tag" href="/tag/design">design</a> <a class="tag" href="/tag/wisdom">wisdom</a></div>
  </div>
</body>
</html>"""

class LabHandler(BaseHTTPRequestHandler):
    def _send(self, code, body, ctype="text/html; charset=utf-8", extra=None):
        data = body.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        for k, v in (extra or {}).items():
            self.send_header(k, v)
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        path = urlparse(self.path).path
        if path == "/":
            self._send(200, QUOTES_PAGE)
        elif path == "/headers":
            lines = [f"{k}: {v}" for k, v in self.headers.items()]
            self._send(200, "\n".join(lines), "text/plain; charset=utf-8")
        elif path == "/set":
            self._send(200, "A cookie has been planted.",
                       extra={"Set-Cookie": "session_id=abc123; Path=/"})
        elif path == "/who":
            cookie = self.headers.get("Cookie", "(no cookie)")
            self._send(200, f"Cookie received by the server: {cookie}",
                       "text/plain; charset=utf-8")
        elif path == "/old":
            self.send_response(301)
            self.send_header("Location", "/")
            self.end_headers()
        else:
            self._send(404, "<h1>404 — no such page</h1>")

    def do_POST(self):
        if urlparse(self.path).path != "/echo":
            self._send(404, "No such route", "text/plain")
            return
        try:
            length = int(self.headers.get("Content-Length", 0))
            self.connection.settimeout(3)
            if length < 0:
                raise ValueError("negative Content-Length")
            if length > 8192:
                # Drain only a bounded amount. A much larger rejected request
                # may end with a connection error instead of a readable 413.
                self.rfile.read(min(length, 8193))
                self._send(413, "Body too large", "text/plain")
                return
            data = self.rfile.read(length)
            if len(data) != length:
                raise ValueError("incomplete request body")
            raw = data.decode("utf-8")
        except (OSError, ValueError, UnicodeError):
            self._send(400, "Invalid request body", "text/plain")
            return
        fields = parse_qs(raw)
        flat = {k: v[0] for k, v in fields.items()}
        self._send(200, f"POST body received by the server: {raw}\nUnpacked result: {flat}",
                   "text/plain; charset=utf-8")

if __name__ == "__main__":
    print("Practice server started: http://127.0.0.1:8010  (stop: Ctrl+C)")
    HTTPServer(("127.0.0.1", 8010), LabHandler).serve_forever()

Input (terminal 1 — for the server)

python lab.py

Output (measured 2026-09-09):

Practice server started: http://127.0.0.1:8010  (stop: Ctrl+C)

How to read it: don’t be intimidated. It’s long, but what it does is simple — a set of branches that returns a fixed answer for each path. /headers is a mirror that returns the headers it received, /set is a counter that plants a cookie, /who is a counter that shows the cookie it received, and /echo is a counter that returns the POST body. We’ll keep using this server in this chapter and the crawling chapter.

Why: building a server yourself leaves in your body the fact that “a server is a program that reads the request line and headers and returns a fixed response.” To see a server through an attacker’s eyes, you must once have stood in the server’s position.

3-2. The First Request and Dissecting the Response

Open terminal 2 and create req1.py.

Input

import requests

r = requests.get("http://127.0.0.1:8010/", timeout=(2, 3))
print("Status code:", r.status_code)
print("Content-Type:", r.headers["Content-Type"])
print("First 60 chars of body:", repr(r.text[:60]))

Output (measured 2026-09-09):

Status code: 200
Content-Type: text/html; charset=utf-8
First 60 chars of body: '<!DOCTYPE html>\n<html>\n<head><meta charset="utf-8"><title>Qu'

How to read it: with three lines of code a GET request flew off, and from the returned box we pulled out and printed the status code, a header, and the body. The body is the very HTML we wrote into lab.py. Last chapter’s single curl line has merely become a few lines of code — but what differs is that this code can now go inside a loop.

3-3. Predict — Where on the Address Do the params Attach?

Time to predict. In the code below, what will the final address printed in r.url look like?

Input

r = requests.get("http://127.0.0.1:8010/headers", params={"q": "security", "page": "2"}, timeout=(2, 3))
print("Final address:", r.url)

Output (measured 2026-09-09):

Final address: http://127.0.0.1:8010/headers?q=security&page=2

How to read it: the two name-values passed as a dictionary attached after the ?, joined with &. requests built for us the GET address we used to assemble by hand. For the record, our server’s /headers ignores the tail of the address and returns only the headers — but as you can see, the address itself flew off fully assembled.

Why: this is the identity of the phenomenon where the address grows long when you type a word into a search box. And the first question of web testing — “if I change this address’s parameters, will the server respond differently?” — is born right here.

3-4. POST and Header Disguise

Input

r = requests.post("http://127.0.0.1:8010/echo", data={"id": "admin", "pw": "1234"}, timeout=(2, 3))
print(r.text)

r2 = requests.get("http://127.0.0.1:8010/headers",
                  headers={"User-Agent": "MySecurityStudyBot/1.0"}, timeout=(2, 3))
print(r2.text)

Output (measured 2026-09-09):

POST body received by the server: id=admin&pw=1234
Unpacked result: {'id': 'admin', 'pw': '1234'}
Host: 127.0.0.1:8010
User-Agent: MySecurityStudyBot/1.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

How to read it: the first part is evidence that the data riding in the POST body arrived at the server intact and was unpacked — you can also see that it traveled in name=value&name=value form. The second part is the list of headers the server saw. The User-Agent I changed shows up as-is, and the headers requests attaches by default (Accept-Encoding, Connection, etc.) appear together.

Why: this is the stage where you gain the conviction that “every part of a request can be assembled by me.” The essence of a web testing tool is exactly this freedom of assembly.

3-5. Sessions — An Experiment in Whether Cookies Persist

Input

s = requests.Session()
s.get("http://127.0.0.1:8010/set", timeout=(2, 3))          # the counter that plants a cookie
r = s.get("http://127.0.0.1:8010/who", timeout=(2, 3))      # check with the same session
print("With session:", r.text)

r2 = requests.get("http://127.0.0.1:8010/who", timeout=(2, 3))   # a fresh request without a session
print("Without session:", r2.text)

Output (measured 2026-09-09):

With session: Cookie received by the server: session_id=abc123
Without session: Cookie received by the server: (no cookie)

How to read it: after planting a cookie through the session, asking again through the same session shows the cookie alive. The request sent without a session has an empty cookie. This is direct evidence that the session plays the role of an “ID wallet.”

A Session is convenient for related requests; it is not the only way to send cookies. Some applications use bearer session cookies, whose theft can enable impersonation until controls or expiry intervene. Other cookies have no authentication role. Do not infer a successful login from this lab cookie.

3-6. Predict — How Is a Redirect Recorded?

Our server’s /old answers only “moved away (301), the new address is /”. If we request it with requests, where is the final destination, and where does the record remain?

Input

r = requests.get("http://127.0.0.1:8010/old", timeout=(2, 3))
print("Final address:", r.url)
print("Notices passed through:", [(x.status_code, x.url) for x in r.history])

Output (measured 2026-09-09):

Final address: http://127.0.0.1:8010/
Notices passed through: [(301, 'http://127.0.0.1:8010/old')]

How to read it: unlike curl, requests follows redirects by default. And r.history keeps the footprints it passed through (the 301 notice). “Where did it finally arrive” and “by what route did it come” show up separately.

Why: checking a shortened link’s real destination, tracing a phishing site’s detour route — all such analysis starts from this history attribute.

3-7. Bounding Connection and Read Waits

Let’s attach the survival gear of automation tools.

Input

import requests

for url in ["http://127.0.0.1:59999/", "http://127.0.0.1:8010/"]:
    try:
        r = requests.get(url, timeout=(2, 3))
        r.raise_for_status()
        print("Success:", r.status_code)
    except requests.Timeout:
        print("Connection/read timeout; not a total-duration deadline")
    except requests.RequestException as exc:
        print("Request failed:", type(exc).__name__)

Output (measured 2026-09-09):

Request failed: cannot reach the server
Success: 200

A connection refusal can raise ConnectionError; a slow or silent connection can raise Timeout. Port 59999 is only a candidate unused port: first ensure it is not serving anything on your machine. Requests timeout limits connection/read waits, not the total wall-clock duration of downloading an entire response.

Use explicit timeouts and catch RequestException. Also bound page count and response size. A peer sending data slowly can keep a read alive, so a strict end-to-end deadline needs separate control; timeout=3 is not a promise that the whole operation ends in three seconds.

4. Missions & Exercises

Mission — Build a Response Analyzer

Create analyze.py to perform the following (the lab.py server must be running).

1. Send a GET request to /headers with two params attached, and print the final URL and the status code
2. From the response body, pick out and print only the User-Agent line the server saw
3. Change the User-Agent to your own string, send the same request again, and confirm by output that the identity the server saw has changed
4. Send three name-value pairs by POST to /echo, and print the server’s unpacked result
5. Wrap every request in try/except with timeout=3, so that even with the server off it leaves a “request failed” notice and exits normally

Exercises

Q1. Where do params and a dictionary passed as data appear in these requests? Explain why a POST can contain both a query and a body.

Q2. You called r.json() and got JSONDecodeError: Expecting value: line 1 column 1 (char 0) (measured 2026-09-09). What situation is this, and what should you check before calling it?

Q3. Why does requests.Session retain the cookie set by this lab while separate calls do not automatically share it? Explain why persistence alone is not authentication.

Q4. Explain the difference between r.url and r.history in terms of “the final destination” and “the road traveled.”

import requests

BASE = "http://127.0.0.1:8010"

try:
    # 1. Attaching params
    r = requests.get(f"{BASE}/headers", params={"q": "security", "page": "2"}, timeout=3)
    print("Final URL:", r.url)
    print("Status code:", r.status_code)

    # 2. Picking out the User-Agent line the server saw
    for line in r.text.splitlines():
        if line.startswith("User-Agent"):
            print("UA seen by the server:", line)

    # 3. Re-checking after disguise
    r2 = requests.get(f"{BASE}/headers",
                      headers={"User-Agent": "MyAnalyzer/2.0"}, timeout=3)
    for line in r2.text.splitlines():
        if line.startswith("User-Agent"):
            print("UA seen by the server after disguise:", line)

    # 4. POST
    r3 = requests.post(f"{BASE}/echo",
                       data={"id": "admin", "pw": "1234", "memo": "first practice"},
                       timeout=3)
    print(r3.text)

except requests.exceptions.ConnectionError:
    print("Request failed: check that the lab.py server is running")
except requests.exceptions.Timeout:
    print("Request failed: the server did not answer within 3 seconds")

Final URL: http://127.0.0.1:8010/headers?q=security&page=2
Status code: 200
UA seen by the server: User-Agent: python-requests/2.33.1
UA seen by the server after disguise: User-Agent: MyAnalyzer/2.0
POST body received by the server: id=admin&pw=1234&memo=first+practice
Unpacked result: {'id': 'admin', 'pw': '1234', 'memo': 'first practice'}

6. Common Pitfalls & Fixes

Wall 1. I ran the client without starting the server

Symptom (measured 2026-09-09):

requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8010): Max retries exceeded ... (Caused by NewConnectionError(... [WinError 10061] No connection could be made because the target machine actively refused it))

Cause: the conversation partner (lab.py) is not running.
Fix: start the server first with python lab.py in terminal 1. The server window must stay open.

Wall 2. The port is blocked when starting the server

Symptom: OSError: [WinError 10048] ... — the specified port is already in use.
Cause: a previous server that never stopped, or another program, is occupying that number.
Fix: change lab.py’s 8010 to something like 8020, and change the address in the client code to match.

Wall 3. r.json() throws an error

Symptom (measured 2026-09-09): requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

An empty body, HTML, or malformed JSON can cause JSONDecodeError. A 200 status does not guarantee JSON, and an error response can contain valid JSON. Check status and expected Content-Type, handle an empty response such as 204, and catch JSONDecodeError.

Wall 4. A request sits frozen for a long time

Requests has no timeout by default. Specify a connect/read timeout, handle Timeout and other RequestException failures, and bound the work. A read timeout measures a wait for data, not the entire download duration. Diagnose the failure rather than reporting an empty successful result.

Wall 5. I definitely received a cookie, but it vanishes on the next request

Symptom: you planted a cookie via /set, but /who returns “(no cookie)”.
Cause: you sent each request fresh with requests.get(...). There is no wallet for the cookie to be stored in.
Fix: create s = requests.Session() and send every request as s.get(...), s.post(...) (the 3-5 measurement is the evidence).

7. Summary

Today’s Concepts

Today’s Functions

The Instinct That Matters More Than Commands

Today’s core instinct is the habit of doubting responses. Beginners believe “a response came back, so it succeeded,” but a practitioner’s order of checks is fixed — ① look at status_code ② print the first 200 characters of r.text ③ only then json() or parse. These three steps erase half of all “why isn’t this working?” debugging time. The security connection: what you learned today is dangerous if abused. Code that hammers the same request without rest becomes a service disruption by itself, so putting a comma like time.sleep(1) between repeated requests is an attitude that comes before technique. And remember the lab.py you built yourself today — a server, in the end, is “a set of branches that reads a request and puts out a fixed answer.” With that picture in hand, no new web technology will ever frighten you.

Once every box is checked, Step 74 is complete.