Step 342. Launching into a New Field — With the Posture of Level 0 and a Proven Methodology

Level 4 — Professional | Difficulty ★★★☆☆ | Estimated time: 2 days (confirming the entry path + building the environment + 1 unit of basic practice)

Prerequisites: you selected one next challenge field in Step 341 and wrote the selection document. This chapter’s hands-on example proceeds on the IoT/embedded axis (firmware carving) — if you chose a different axis, apply the same procedure to that axis’s taste-test.

What you need: Step 341’s selection document, a Linux environment (WSL is enough), Python 3, and a learning-notes repository. This chapter’s firmware-generation/carving scripts and outputs were measured in this book’s lab (WSL, Ubuntu 24.04, Python 3.12).
Caution: the core of today’s practice is "table-of-contents trust" in a standard entry path — even when confusion comes from the mix of what you know and what’s new, follow the table of contents in order for two weeks.
⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. Firmware, vulnerable apps, and cloud labs are all handled only as official distributions and in your own account.

The first step into a new field must always be humble. Even after completing this curriculum, you’re a beginner in a new field. But this time is different — you’re someone who knows in their body "how to go from beginner to expert," and you enter with the posture of the day you first opened PowerShell, plus a proven learning methodology (fundamentals → practice → organizing → sharing).

Today is that first practice day. You confirm the selected area’s entry path, build the environment, and complete one unit of basic practice. The example covered — IoT firmware carving — is an area where this book’s file-format knowledge transfers as-is, so you can confirm the "connecting link" with your hands from day one.

1. Learning Objectives

By the end of this chapter, you will be able to:

Find the selected area’s standard entry path (official docs, beginner labs, community curricula) and fix its table of contents
Build an area-specific practice environment under the "my lab / practice assets" principle
Carve embedded files from a firmware blob with a magic-byte scan
Distinguish a carving tool’s false positives (fake signatures)
Start the new field’s learning notes with the same record habits as Levels 0–2

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	Python 3 (standard library only: `zlib`, `zipfile`, `io`), a Linux shell (WSL)
Today’s command	`python3 make_fw.py` · `python3 carve.py router_fw.bin` · `ls -la` · `cat carved_*`
Concepts needed	Firmware blobs, magic bytes (file signatures), carving, false positives
Today’s deliverable	An entry-path table of contents + a practice environment + a carving-practice record + the learning notes’ first document

2-1. The Structure of Entering a New Field — Trust the Table of Contents

The first confusion in a new field comes at the point where "what you know and what’s new mix." You open firmware analysis and half is file-format talk you already know, half is bootloader talk you’ve never seen — you lose your sense of how far you need to study.

The prescription is clear-cut — trust the entry path’s table of contents. A standard entry path’s (THM/HTB tracks, official guides, community curricula) table of contents is an order polished by countless beginners’ trial and error. Follow the order for just two weeks and a map of the field forms in your head. Exploration that skips the table of contents comes after the map exists.

2-2. Environment-Building Principles — New Fields Also Only in "My Lab / Practice Assets"

The shape of environment building differs per area, but the principle is one — in a new field too, practice happens only in my lab and on practice assets.

Area	Environment’s shape	Cautions
Cloud	AWS Free Tier + an isolated practice account	Billing alerts are mandatory, lock the root account, a practice-only IAM user
Mobile	Android emulator + vulnerable practice apps (DIVA, etc.)	Emulator instead of a real device; analyze nothing but practice apps
IoT	Public firmware images + analysis tools	Only firmware officially distributed by the manufacturer
AI	Local playgrounds + OWASP materials	Don’t send attack prompts at external LLM services

Cloud’s "billing alert" is not a security issue but a wallet issue — the pitfall beginners hit most. The moment you create the account, turn on payment notifications first.

2-3. Firmware and Carving — Today’s Practice Concepts

Firmware is the software package baked into a device. Unpack one router firmware and out come a bootloader, a kernel, and a root filesystem — and finding config files, hardcoded credentials, and private keys inside is the classic of IoT security.

But firmware is often a blob — several pieces concatenated into a single file. A header followed by a compressed kernel followed by a filesystem image — often with no boundary markers. The technique used here is carving — find each file format’s magic bytes (signature) across the whole binary, and pull the file out from that point.

Format	Magic bytes	Clue
gzip	`1f 8b 08`	common for kernel/config compression
zip	`50 4b 03 04` ("PK")	APKs, some update packages
ELF	`7f 45 4c 46`	executables
SquashFS	`68 73 71 73` ("hsqs")	the representative router root filesystem

Among professional tools, binwalk is the standard. Today, though, you implement carving yourself with only the Python standard library — doing once by hand what the tool does is day-one’s share of entry, and it’s what lets you read binwalk’s output later.

2-4. False Positives — Signatures Appear by Coincidence Too

Carving’s first trap is the false positive. Magic bytes are patterns of only a few bytes, so the same byte sequence can appear by coincidence inside compressed data. Zip’s PK appearing inside the archive itself is the representative case.

So a carving tool’s output is not "discoveries" but "candidates" — only after confirming that a file actually opens (decompresses) at that offset is it real. In 3-3’s measurement this false positive actually occurs, so read the output carefully.

3. Follow Along

3-1. Confirming the Entry Path — The Table of Contents Goes on the Notes’ First Page

Find the selected area’s standard entry path and copy the whole table of contents into your learning notes. Here’s the IoT-axis example.

Entry-path table of contents (screen example — IoT/firmware axis):
1. Firmware structure — bootloader, kernel, root filesystem
2. Firmware acquisition & extraction — carving, the binwalk family of tools   <- today, here
3. Root-filesystem analysis — hunting configs, credentials, keys
4. Static analysis — intro to reversing the extracted binaries
5. Emulation & dynamic analysis — running firmware under QEMU
6. (Optional) Hardware interfaces — UART/JTAG

As you copy the table of contents, attach one line of "connection to my existing knowledge" to each item — item 4 connects to this book’s reversing chapters, item 5 to your WSL experience. Those one-liners determine your comprehension two weeks from now.

3-2. Making Practice Firmware — Measured

Instead of real router firmware, you build a practice blob with the same structure yourself — header + gzip piece + zip piece + random tail. Save as make_fw.py.

# make_fw.py — generate a fake practice firmware blob
import gzip, zipfile, io, os, random

random.seed(42)
hdr = b"LABFW" + bytes(random.randrange(256) for _ in range(59))  # 64-byte header
payload = gzip.compress(b"labhostnssh-rsa AAAA... lab-keyn")
zipbuf = io.BytesIO()
with zipfile.ZipFile(zipbuf, "w") as z:
    z.writestr("init/start.sh", "#!/bin/shn# lab init scriptn")
    z.writestr("etc/passwd", "root:x:0:0:root:/root:/bin/shn")
tail = bytes(random.randrange(256) for _ in range(128))
open("router_fw.bin","wb").write(hdr + payload + zipbuf.getvalue() + tail)
print("created: router_fw.bin", os.path.getsize("router_fw.bin"), "bytes")

Here’s the measured output from running it:

created: router_fw.bin 520 bytes

A 520-byte blob is born. Real firmware runs MBs to hundreds of MBs, but the structure is the same — pieces concatenated without boundaries. Because you made it yourself, you know the answers: the gzip piece holds a hostname and a key fragment; the zip piece holds init/start.sh and etc/passwd. Now pretend you don’t know the answers and recover them by carving.

3-3. Carving — Pulling Files Out with Magic Bytes

Implement the carving tool yourself. Save as carve.py.

# carve.py — pull files out of firmware via a magic-byte scan
import sys, zlib, zipfile, io

MAGICS = {
    b"x1fx8bx08": "gzip",
    b"PKx03x04": "zip",
    b"x7fELF": "elf",
    b"hsqs": "squashfs",
    b"ustar": "tar(approx)",
}
data = open(sys.argv[1], "rb").read()
print(f"target: {sys.argv[1]} ({len(data)} bytes)")
found = []
for magic, name in MAGICS.items():
    off = 0
    while True:
        i = data.find(magic, off)
        if i < 0:
            break
        found.append((i, name))
        off = i + 1
for off, name in sorted(found):
    print(f"  offset {off:>6}: {name} signature")

# extraction attempt — a signature is only a candidate; it must open to be real
for off, name in sorted(found):
    if name == "gzip":
        try:
            d = zlib.decompressobj(16 + zlib.MAX_WBITS)
            out = d.decompress(data[off:])
            open(f"carved_{off}.bin", "wb").write(out)
            print(f"[+] gzip @ {off} extracted -> carved_{off}.bin ({len(out)} bytes)")
        except Exception as e:
            print(f"[-] gzip @ {off} failed: {e}")
    if name == "zip":
        try:
            z = zipfile.ZipFile(io.BytesIO(data[off:]))
            for n in z.namelist():
                open("carved_" + n.replace("/", "_"), "wb").write(z.read(n))
            print(f"[+] zip @ {off} extracted -> {z.namelist()}")
        except Exception as e:
            print(f"[-] zip @ {off} failed: {e}")

Here’s the measured output from running it:

target: router_fw.bin (520 bytes)
  offset     64: gzip signature
  offset    114: zip signature
  offset    185: zip signature
[+] gzip @ 64 extracted -> carved_64.bin (32 bytes)
[+] zip @ 114 extracted -> ['init/start.sh', 'etc/passwd']
[-] zip @ 185 failed: negative seek value -71

How to read it: look at three things. ① The gzip at offset 64 — the header was exactly 64 bytes, so it matches the answer. ② The second zip signature at offset 185 and its failure — this is 2-4’s false positive. The zip file’s internal central directory also contains the PK pattern, so the scanner reported it as a "candidate," but actually opening it fails with the error negative seek value -71. A signature scan gives candidates; opening is the confirmation. ③ Extraction success is [+], failure [-] — this convention in tool output is the same in binwalk.

3-4. Checking the Extracts — Firmware Analysis’s First Harvest

Check the pulled-out files.

ls -la
cat carved_64.bin
cat carved_etc_passwd

Here’s the measured output:

-rw-r--r--  1 student student 1406 Sep  9 22:06 carve.py
-rw-r--r--  1 student student   32 Sep  9 22:06 carved_64.bin
-rw-r--r--  1 student student   30 Sep  9 22:06 carved_etc_passwd
-rw-r--r--  1 student student  688 Sep  9 22:06 make_fw.py
-rw-r--r--  1 student student  520 Sep  9 22:06 router_fw.bin

labhost
ssh-rsa AAAA... lab-key

root:x:0:0:root:/root:/bin/sh

(The ls output above is trimmed to the relevant rows from the measurement; a real lab may hold more artifacts from the extraction process.)

How to read it: you’ve just put "firmware analysis’s classic harvest" into your hands. The gzip piece yielded a hostname and an SSH key fragment; the zip piece yielded a passwd file. Real router-firmware analysis is exactly this picture — carve open the root filesystem, then hunt credentials, keys, and configs inside. Only the scale changes from today’s 520 bytes to real firmware of several MB; the procedure is identical.

3-5. Starting the Learning Notes — With Level 0’s Habits As-Is

Record right after the practice ends. The same template you used in Levels 0–2.

Learning notes' first document (screen example — IoT axis, Day 1):
- Concepts learned: firmware blobs, magic bytes, carving, false positives
- What I did by hand: built a blob with make_fw.py, extracted 3 files with carve.py
- Where I got stuck: the zip false positive at offset 185 — didn't know what the negative seek error was at first
- How I solved it: re-read the zip internal structure (central directory) and confirmed the false positive's principle
- Next to do: TOC item 3 — apply the same procedure to one real public firmware

Finally, set your own completion criteria — the consistent rule of this entire book. Example: "Reproduce TOC item 2 on a real firmware, and record at least 3 entries in the notes." In a new field too, only learning with completion criteria is learning that ends.

4. Missions & Exercises

Mission — Completing Basic Practice in the Selected Area

Find the standard entry path for the area selected in Step 341, copy its table of contents onto the learning notes’ first page, and attach one line of "connection to existing knowledge" to each item.
Build the practice environment per 2-2’s principle table (cloud: isolated account + billing alerts; mobile: emulator + vulnerable practice apps; IoT: public firmware and an analysis directory).
Perform one unit of entry practice — on the IoT axis, reproduce 3-2~3-4’s carving practice, then apply the same procedure to one real public firmware. On other axes: for cloud, IAM user & policy practice + one CloudGoat scenario; for mobile, analyzing three DIVA vulnerabilities.
Start the learning notes in 3-5’s template, recording every stuck point and its resolution without omission.
Set your own completion criteria for basic practice, document them, and confirm achievement.

Exercises

Exercise 1. Explain why the prescription "trust the table of contents" is needed when entering a new field, through the structure of the confusion of "what you know mixing with what’s new."

Exercise 2. Explain why a magic-byte scan in carving is a "candidate" rather than a "discovery," using 3-3’s measured case of offset 185.

Exercise 3. Explain why "setting up billing alerts" is the first item of entry practice rather than a security practice when building a cloud environment.

Exercise 4. Explain why you implement the first carving yourself in Python even though a tool (binwalk) exists, from the perspective of "the ability to read tool output."

5. Model Answers & Completion Criteria

Mission Model Answer

Check against these verification criteria.

Fidelity of the TOC: is the entry path’s table of contents copied in full, with a concrete connection to existing knowledge attached to each item?
Legality of the environment: is the practice environment confined to my lab / practice assets per 2-2’s table — for cloud, is there evidence billing alerts are set?
Reproducibility of the practice: for the carving practice, do the scripts and outputs survive in the notes so someone else (or future me) can reproduce them?
Understanding of false positives: does the practice record describe a false-positive case and the process of distinguishing it?
Self-defined completion criteria: are the completion criteria fixed in a measurable sentence, with achievement checked?

Exercise Answers

Answer 1. A new field’s materials mix existing knowledge with new concepts — the file-format parts of firmware analysis you already know; the bootloader parts are new. At this point the learner tilts toward two extremes — skipping ahead at the known parts with "I know this" and then hitting frustration at the new parts, or re-studying everything including the known parts and stalling progress. A standard entry path’s table of contents is an order polished by countless beginners’ trial and error, so it’s designed assuming this mix. Trusting the table of contents means delegating the judgment of order to experts’ accumulation — and thanks to that, the learner can focus only on "this item now" without the judgment cost of "what should I study." Once two weeks pass and the field’s map forms in your head, you may reorganize the table of contents by your own judgment — delegation is a strategy for the entry stretch, not a permanent rule.

Answer 2. Magic bytes are patterns of only a few bytes, so they appear by coincidence in places that aren’t the start of a real file. In the measurement, offset 185 caught a PK signature but extraction failed with negative seek value -71 — the scanner reported it as a "candidate" because the zip file’s internal central-directory region also contains records starting with PK. A signature is a hint that "something might be here," and only after confirming a file actually opens at that offset (decompression/parsing succeeds) does it become a "discovery." This is why the professional tool binwalk’s output also lists candidates alongside an entropy graph — the analyst’s job is picking the real ones from the candidates.

Answer 3. In the cloud, usage is billing, so leaving a practice environment running becomes an invoice immediately. The beginner’s typical accident goes like this — create a practice instance or storage, forget to delete it, and meet an unexpected bill weeks later. This accident is far more common than a security incident, and its effect on entry motivation is large. So a cloud entry’s first practice is setting up payment notifications and budget alerts before any security technique. Moreover, this habit continues into security — a cost anomaly is also the simplest sensor catching the behavior of assets you don’t know about (a signal of compromise).

Answer 4. If you use the tool right away, you build a "habit of believing" without the "ability to read" the output. binwalk’s output is a candidate list, and picking the real from the false positives is the user’s share. Implementing it yourself teaches you everything the tool does — scans magic bytes, attempts parsing at each offset, and marks success and failure. So you can judge why the tool’s [+]/[-] marks exist, why false-positive rows appear, and whether the tool silently missed anything. "Doing once by hand what the tool does" is this whole book’s repeated principle (like crafting packets by hand and writing exploits yourself), and in a new field the principle holds as-is.

Completion Criteria Checklist

[ ] I copied the selected area’s standard entry-path TOC into my notes and attached connecting links
[ ] I built the practice environment per 2-2’s principle table (billing alerts included for cloud)
[ ] I reproduced the carving practice (or the chosen axis’s entry practice) and left the outputs in my notes
[ ] I met a false positive firsthand and recorded the process of distinguishing it
[ ] My stuck points and their resolutions are in the learning notes
[ ] I set my own completion criteria for basic practice and confirmed achievement

6. Common Pitfalls & Fixes

Wall 1. python3 carve.py can’t find the file

Symptom: this error appears.

FileNotFoundError: [Errno 2] No such file or directory: 'router_fw.bin'

Cause: the directory where you ran carve.py differs from the directory where you made router_fw.bin — the script looks for the argument path relative to the current directory.

Fix: keep both files in the same directory and run from there — cd ~/lab342 then python3 make_fw.py && python3 carve.py router_fw.bin. A good habit to build — always practice in a dedicated directory. It also keeps the artifacts (carved_*) from scattering.

Wall 2. Extraction succeeded but the contents look corrupted

Symptom: you opened carved_64.bin in an editor and the beginning is strange characters.

Cause: distinguish two things. ① Real corruption — the extraction start point is wrong. ② Normal-but-binary — most of what you pull from firmware is not text but binaries like kernel images and filesystems.

Fix: first check the format with file carved_64.bin, and look at the hex with xxd carved_64.bin | head. If a text-expected piece (config-file family) is corrupted, the offset is wrong — try the scan results’ other candidates. As in 3-4’s measurement, only text pieces read directly in cat — "an extract that won’t read" is not failure; it’s the input to the next analysis step (file, strings, reversing).

Wall 3. The entry path I found is a really old resource

Symptom: the community-recommended curriculum is 3 years old, and the tool versions differ from today.

Cause: resource lifespans differ per field — the same problem as AI security’s short resource lifespan in Step 341.

Fix: verify in three layers. ① Concept validity — concepts like carving and magic bytes stay valid for decades. ② Tool currency — follow "what the tool does," not the tool’s name. ③ Official docs first — use community resources as the map, but always confirm commands and options in official documentation. What’s dangerous about old resources is not concepts but command lines.

Wall 4. I obtained real firmware, but it’s encrypted

Symptom: carving the firmware downloaded from the vendor catches no signatures at all.

Cause: recent firmware is increasingly distributed encrypted or obfuscated — it’s normal for a signature scan to fail.

Fix: at the entry stage, bypassing is not the goal — switch to public firmware that carving works on. Old router firmware and open-source firmware (the OpenWrt family) images are good practice material. Handling encrypted firmware (dumping from the device, intercepting the update process, etc.) is a later topic in the table of contents — trust the TOC; it’s not today’s item.

Wall 5. Every time "something I know" comes up, I want to skip it

Symptom: half the entry material is review, so you skip along — then at some point comprehension suddenly cuts off.

Cause: new terminology hides among the known content — you know "filesystem," but the field’s vocabulary (rootfs, SquashFS image) is new.

Fix: change the skipping rule — don’t skip whole sections; speed-read, but make a fresh terminology table. Time spent reading known content is not waste; it’s time mapping the field’s vocabulary onto your knowledge. The process of 2-1’s map forming is exactly this. And the record habit you learned in this book is the answer here too — one line in the notes like "review section: O, new term: rootfs (= root filesystem image)" is enough.

7. Summary

Today’s Concepts

Concept	One-line explanation
TOC trust	Delegate order judgment to the standard path during entry — reorganize once the map forms
My-lab principle	New fields too: practice only in my lab & on practice assets — for cloud, billing alerts first
Firmware blob	A package of header + compressed pieces + filesystem concatenated without boundaries
Magic bytes	Signatures announcing a format — gzip `1f 8b 08`, zip `PK`, ELF, squashfs
Carving	The technique of pulling files out from signature offsets
False positives	Signatures appear by coincidence too — confirmed only by opening
Learning notes	With Level 0’s habits as-is — concepts, stuck points, resolutions, next to-do

Today’s Tools & Commands

Tool/command	What it does
`python3 make_fw.py`	Generates the practice firmware blob (header + gzip + zip + tail)
`python3 carve.py <file>`	Magic-byte scan + extraction attempts + success/failure marks
`ls -la` / `cat carved_*`	Listing extracts and checking text pieces
`file` / `xxd`	Format judgment of extracts (the next step for ones that won’t read)
Learning-notes template	Five lines: concepts · practice · stuck · resolution · next to-do

The Core Instinct

Six hundred days ago, even a PowerShell prompt felt foreign to you; today, you pull files out of a blob in an unfamiliar field. The awkwardness in a new field is the same, but the procedure for handling awkwardness is already in your hands — check the fundamentals, try by hand, record the stuck points, organize.

One carving doesn’t make you a firmware analyst. But the difference between "a person who pulled a key and a passwd out of a 520-byte blob" and "a person who has only read about firmware analysis" is the difference this entire book has repeated. Today, having completed the first unit by hand, the new field is no longer foreign land — it’s a map in progress.

Once every box is checked, Step 342 is complete.