Step 239. Hex and File Signatures: file, binwalk, Manual Carving — An Identity Check on Six Bytes

Level 3 — Forensics Track | Difficulty ★★☆☆☆ | Estimated time: 3 hours

Prerequisites: Step 180 (A Taste of CTF Forensics) completed. You know the basics of file, xxd, strings, and the PNG chunk structure.

⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime.

What you need: WSL Ubuntu (file 5.45, xxd, strings — confirmed by measurement), Python 3 (measured: 3.12.14).
Caution: binwalk isn’t available in this environment, so it’s introduced as a "screen example" only. Everything else — building the identifier, repairing a broken header, manual carving — was measured locally on 2026-09-09.

This is the Forensics track’s first chapter. The first thing a forensic analyst does with an evidence file is ask, "Are you really who you say you are?" What proves a file’s identity isn’t its extension but its first few bytes — the magic bytes. Today you’ll build an identifier on these signatures yourself, repair a broken header by hand, and go as far as carving — extracting a file from a pile of garbage bytes. These three are the basic stamina of file forensics.

1. Learning Objectives

By the end of this chapter, you will be able to:

Name the file type from the magic bytes of 10 major formats
Write a file-signature identifier yourself in Python
Manually repair the header of a file whose signature was destroyed
Cut the signature-to-footer region out of a raw byte blob and bring a file back to life (carving)
Explain what binwalk does by contrasting it with manual carving

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category	Details
Language/environment	WSL Ubuntu bash + Python 3 (measured: file 5.45, xxd / Python 3.12.14)
Today’s commands	`file FILE`, `xxd FILE`, `xxd -s OFFSET -l LENGTH FILE`, Python `bytes.find()`
Concepts needed	Magic bytes, file headers and footers, the ZIP family’s sibling problem, data carving
Today’s deliverable	A signature table + a Python identifier + 2 recovered PNGs (header repair, carving)

2-1. Magic Bytes — A File’s Front Yard

Most file formats announce "I am this kind of file" with an agreed-upon starting byte sequence. These are the magic bytes, or the file signature. The file command reads these bytes and the structure behind them to determine the type.

Two core principles. First, the extension is decoration. It’s a tag the OS uses to pick an icon — unrelated to the file’s contents. Second, signatures can be destroyed. Break the first few bytes and tools classify the file as "data" (unidentified) — and repair those broken bytes by hand, and the file comes back to life.

2-2. Headers and Footers — A File’s Start and End Markers

If the signature is the "start marker," many formats also have an "end marker." PNG ends with an IEND chunk, JPEG with FF D9 (EOI). ZIP places a central directory near the file’s end (starting 50 4B 05 06).

When both markers exist, cutting between them yields one whole file. This is the principle of carving — extracting a file from a byte blob like chiseling a statue out of stone. It’s exactly what forensic tools (PhotoRec, foremost) and binwalk’s extraction feature do.

2-3. The ZIP Family’s Sibling Problem

Seeing 50 4B 03 04 (PK..) makes you want to answer ZIP, but docx, xlsx, jar, apk, and odt are all ZIP containers. The signature alone can’t tell these siblings apart. You have to look at what’s inside (a [Content_Types].xml means docx; an AndroidManifest.xml means apk). We confirm this live in today’s measurement.

2-4. binwalk — Carving, Automated (Screen Example)

binwalk sweeps the entire file and reports every position where a known signature appears. It isn’t installed in this environment, so we look at a screen example only (not actually run):

# Screen example — detecting embedded files with binwalk
$ binwalk firmware.bin
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             PNG image
41472         0xA200          gzip compressed data
102400        0x19000         Zip archive data, at least v2.0 to extract

# Screen example — extraction
$ binwalk -e firmware.bin        # unpacks the found files into _firmware.bin.extracted/
$ binwalk --dd='.*' firmware.bin # cuts everything by signature and saves to files

The manual carving we do in Python today is the very principle of that binwalk -e. Someone who knows the principle can do it by hand even without the tool.

3. Follow Along

3-1. Building the Signature Lab

The script below creates samples in 9 formats plus disguised, damaged, and buried files (this chapter’s local output was measured 2026-09-09).

Input (sig_lab.py)

import struct, zlib, zipfile, gzip, os, sqlite3

os.makedirs("siglab", exist_ok=True); os.chdir("siglab")

def chunk(typ, data):
    c = struct.pack(">I", len(data)) + typ + data
    return c + struct.pack(">I", zlib.crc32(typ + data) & 0xFFFFFFFF)

def make_png(path):
    sig  = b"x89PNGrnx1an"
    ihdr = chunk(b"IHDR", struct.pack(">IIBBBBB", 2, 2, 8, 2, 0, 0, 0))
    raw  = b"x00" + b"xffx00x00" * 2 + b"x00" + b"x00xffx00" * 2
    idat = chunk(b"IDAT", zlib.compress(raw))
    open(path, "wb").write(sig + ihdr + idat + chunk(b"IEND", b""))

make_png("sample.png")
open("sample.jpg", "wb").write(bytes.fromhex("FFD8FFE000104A46494600010100000100010000") + b"x00"*32 + b"xffxd9")
with zipfile.ZipFile("sample.zip", "w") as z: z.writestr("hello.txt", "PKn")
open("sample.gif", "wb").write(b"GIF89a" + b"x00" * 40)
open("sample.pdf", "wb").write(b"%PDF-1.4n%%EOFn")
with gzip.open("sample.gz", "wb") as f: f.write(b"gzip samplen")
open("sample.bmp", "wb").write(b"BM" + b"x00" * 60)
con = sqlite3.connect("sample.db"); con.execute("create table t(a)"); con.close()
with open("sample.docx", "wb") as f:
    with zipfile.ZipFile(f, "w") as z: z.writestr("[Content_Types].xml", "<Types/>")

make_png("disguise.txt")           # disguise: PNG content, txt extension
make_png("broken.png")
d = bytearray(open("broken.png", "rb").read())
d[1:4] = b"x00x00x00"           # damage: the three letters 'PNG' destroyed
open("broken.png", "wb").write(bytes(d))

inner = open("sample.png", "rb").read()   # burial: a PNG among garbage
open("fragment.bin", "wb").write(os.urandom(300) + inner + os.urandom(180))

Output

(12 files created — 9 samples, disguise.txt, broken.png, fragment.bin)

3-2. Completing the 10-Signature Table Yourself

Tear open the first 8 bytes of each file:

cd siglab
for f in sample.*; do printf '%-12s ' "$f"; xxd -l 8 -p "$f" | tr -d 'n'; echo; done

sample.bmp   424d000000000000
sample.db    53514c6974652066
sample.docx  504b030414000000
sample.gif   4749463839610100
sample.gz    1f8b08080c34a16a
sample.jpg   ffd8ffe000104a46
sample.pdf   255044462d312e34
sample.png   89504e470d0a1a0a
sample.zip   504b030414000000

(Measured 2026-09-09. ELF was confirmed as 7f 45 4c 46(.ELF) from /bin/ls, and PE as 4d 5a(MZ) from a Windows DLL.)

Organized into a table, that’s today’s first deliverable:

Format	Leading bytes (hex)	Visible shape	Footer
PNG	`89 50 4E 47 0D 0A 1A 0A`	`.PNG....`	`IEND` chunk + CRC
JPEG	`FF D8 FF`	—	`FF D9`
ZIP (zip/docx/jar/apk)	`50 4B 03 04`	`PK..`	central directory `50 4B 05 06`
GIF	`47 49 46 38`	`GIF8`	`3B`
PDF	`25 50 44 46 2D`	`%PDF-`	`%%EOF`
gzip	`1F 8B 08`	—	original CRC32 + size
BMP	`42 4D`	`BM`	none
SQLite	`53 51 4C 69 74 65 20 66`	`SQLite f`	—
ELF	`7F 45 4C 46`	`.ELF`	—
PE (exe/dll)	`4D 5A`	`MZ`	—

How to read it: in 89 50 4E 47, the 50 4E 47 is ASCII PNG. Signatures are often designed with recognizable letters mixed in — there’s even an anecdote that MZ is the initials of an MS-DOS-era developer (Mark Zbikowski).

3-3. Checking Against `file` — and One Twist

Let’s have file identify the same files:

file sample.png sample.jpg sample.zip sample.gif sample.pdf sample.gz sample.bmp sample.db sample.docx

sample.png:  PNG image data, 2 x 2, 8-bit/color RGB, non-interlaced
sample.jpg:  JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16
sample.zip:  Zip archive data, at least v2.0 to extract, compression method=store
sample.gif:  GIF image data, version 89a, 1 x 1
sample.pdf:  PDF document, version 1.4
sample.gz:   gzip compressed data, was "sample", ..., original size modulo 2^32 17
sample.bmp:  ASCII text, with no line terminators
sample.db:   SQLite 3.x database, ...
sample.docx: Zip archive data, at least v2.0 to extract, compression method=store

(Measured 2026-09-09.)

Two twists here. ① sample.bmp was judged "ASCII text," not "BMP." The BMP signature is just two bytes, BM — too short — so file also checks the structure after the header (size fields and such); our minimal sample has that structure empty and failed the check. The shorter the signature, the more room for false positives and misses. ② sample.docx was judged "Zip archive." That’s 2-3’s sibling problem confirmed in the flesh — a docx’s true identity is ZIP, and the distinction comes from the internal file list.

3-4. Writing a Python Identifier — a Miniature `file`

Move the signature table into code:

SIGS = {
    b"x89PNGrnx1an": "PNG image",
    b"xffxd8xff":      "JPEG image",
    b"PKx03x04":        "ZIP family (zip/docx/jar/apk...)",
    b"GIF8":              "GIF image",
    b"%PDF":              "PDF document",
    b"x1fx8b":          "gzip compressed",
    b"BM":                "BMP image",
    b"SQLite format 3":   "SQLite database",
    b"x7fELF":           "ELF executable (Linux)",
    b"MZ":                "PE executable (Windows)",
}

def identify(path):
    head = open(path, "rb").read(16)
    for sig, name in sorted(SIGS.items(), key=lambda kv: -len(kv[0])):
        if head.startswith(sig):
            return name
    return "unidentifiable (data)"

for f in ["sample.png", "sample.docx", "disguise.txt", "broken.png"]:
    print(f"{f:15s} -> {identify(f)}")

sample.png      -> PNG image
sample.docx     -> ZIP family (zip/docx/jar/apk...)
disguise.txt    -> PNG image
broken.png      -> unidentifiable (data)

(Measured 2026-09-09.)

How to read the output: three things are proven. ① disguise.txt, with its .txt extension, was identified as PNG — identity is spoken by bytes. ② broken.png came out "unidentifiable" — three destroyed signature letters erased its identity. ③ Why we check longest signature first with sorted(..., key=-len) — to prevent the accident of a shorter pattern matching before SQLite format 3 (16 bytes). The real file also checks thousands of magic patterns sorted by priority. We built a 10-line miniature of that ledger.

Also confirm file‘s own verdicts — file disguise.txt says PNG image data, 2 x 2, 8-bit/color RGB, and file broken.png says data (measured 2026-09-09).

3-5. Manually Repairing a Broken Header

Look at the front of broken.png:

xxd -l 16 broken.png

00000000: 8900 0000 0d0a 1a0a 0000 000d 4948 4452  ............IHDR

(Measured 2026-09-09.)

How to read it: the 50 4E 47(PNG) that should follow 89 was erased to 00 00 00. But from the 8th byte, IHDR sits intact — the body is alive and only the ID card’s front was torn off. The repair is simple. Overwrite with the correct signature:

data = bytearray(open("broken.png", "rb").read())
print("before repair:", data[:8].hex(" "))   # 89 00 00 00 0d 0a 1a 0a
data[1:4] = b"PNG"
open("fixed.png", "wb").write(bytes(data))
print("after repair:", bytes(data[:8]).hex(" "))  # 89 50 4e 47 0d 0a 1a 0a

file fixed.png

fixed.png:  PNG image data, 2 x 2, 8-bit/color RGB, non-interlaced

(Measured 2026-09-09. A three-byte repair resurrected the file.)

Why this works: a file format is a documented, agreed structure. Since "what must be at which byte" is fixed, write the correct answer into the broken spot and you’re done. Most CTF "the image won’t open" problems are this three-byte repair.

3-6. Manual Carving — Extracting a PNG from a Pile of Garbage

fragment.bin (554 bytes) is 300 bytes of random + a PNG + 180 bytes of random. It mimics a disk’s unallocated space. The carving procedure: find the signature, find the footer, cut between them:

blob = open("fragment.bin", "rb").read()
start = blob.find(b"x89PNGrnx1an")
end   = blob.find(b"IEND", start) + 8      # 'IEND' 4 bytes + CRC 4 bytes
print(f"blob size: {len(blob)}, PNG start: {start}, end: {end}")
open("carved.png", "wb").write(blob[start:end])

blob size: 554, PNG start: 300, end: 374

(Measured 2026-09-09. The start offset 300 matches exactly where we planted it.)

Recovery isn’t complete until you check whether the extraction is truly intact. Let’s recompute the PNG chunks’ CRCs ourselves:

import struct, zlib
carved = open("carved.png", "rb").read()
pos = 8
while pos < len(carved):
    length, = struct.unpack(">I", carved[pos:pos+4])
    typ = carved[pos+4:pos+8]
    crc_stored, = struct.unpack(">I", carved[pos+8+length:pos+12+length])
    crc_calc = zlib.crc32(typ + carved[pos+8:pos+8+length]) & 0xFFFFFFFF
    print(f"chunk {typ.decode():4s} length {length:2d}  CRC {'OK' if crc_stored==crc_calc else 'CORRUPT'}")
    pos += 12 + length

chunk IHDR length 13  CRC OK
chunk IDAT length 17  CRC OK
chunk IEND length  0  CRC OK

(Measured 2026-09-09. file carved.png also judged it a valid PNG.)

How to read the output: all three chunks’ CRCs match — mathematical proof that the 74 bytes torn from the garbage are a byte-perfect PNG. This is what happens behind the "recovered" lists that binwalk -e or PhotoRec show you. For formats without a footer (e.g., BMP, some streams), you must estimate the end from the next signature’s position or a length field in the header, which makes carving one step harder.

4. Missions & Exercises

Mission — Completing the Signature Ledger and a Carving Tool

Add 3 formats to 3-2’s table — confirm the leading bytes of RAR (Rar!), 7z, and ELF by direct experiment or official documentation, and write them into the table
Extend identify() into deep_scan(path) that also finds whether a second signature exists inside the file (hint: repeat find for every signature, every position)
Build your own blob with a JPEG (FF D8 FF ~ FF D9) buried inside, extract it with a carving script, and confirm the file verdict

Exercises

Exercise 1. A file’s first 4 bytes are 50 4B 03 04 and its extension is .apk. What will file‘s verdict be? And what more must you look at to be sure it’s really an Android app?

Exercise 2. In 3-3, file judged the minimal BMP sample as "ASCII text." Explain what this case says about the relationship between "signature length and identification reliability."

Exercise 3. In carving, why is the + 8 in end = blob.find(b"IEND", start) + 8 needed? What happens with only + 4?

Exercise 4. When repairing a broken PNG, distinguish the cases where fixing just the signature works from those where it doesn’t. What damage would keep the file from opening even after a signature repair?

5. Model Answers & Completion Criteria

Mission Model Answer

① Additions to the signature table (per official docs; 7z is 37 7A BC AF 27 1C):

Format	Leading bytes (hex)	Shape
RAR 4.x	`52 61 72 21 1A 07 00`	`Rar!...`
7z	`37 7A BC AF 27 1C`	`7z..'`

② deep_scan example:

def deep_scan(path):
    blob = open(path, "rb").read()
    for sig, name in SIGS.items():
        pos = blob.find(sig)
        while pos != -1:
            print(f"  offset {pos:6d}: {name}")
            pos = blob.find(sig, pos + 1)

Applied to fragment.bin, it prints offset 300: PNG image — a signature at a position other than 0 is evidence of "a file inside a file."

③ JPEG carving: cut with find(b"xffxd8xff") and find(b"xffxd9", start) + 2. JPEG’s footer is 2 bytes, hence the + 2. Success is the extraction being judged "JPEG image data" by file. Note that FF D9 can appear by chance inside JPEG data, so in the field you sometimes carve multiple candidates.

How to verify: ① do the table’s bytes match real files? ② does deep_scan report non-zero offsets too? ③ is the JPEG extraction’s file verdict JPEG?

Exercise Answers

Answer 1. file will judge it "Zip archive data" (just as docx was judged in 3-3). To know it’s really an apk, you must look inside — check with unzip -l for AndroidManifest.xml and classes.dex. The signature only tells you the container type.

Answer 2. The shorter the signature, the higher the chance of an accidental match, and the more tools rely on secondary verification (checking subsequent structure fields). Two bytes BM alone can’t establish BMP, so file looked at the size fields too — and our sample had them empty, causing the miss. Conversely, PNG’s 8-byte signature makes an accidental match practically impossible, so the signature alone is strong evidence.

Answer 3. find returns the position where the pattern starts. An IEND chunk is structured length 4 + type 4 ('IEND') + data 0 + CRC 4, so 4 more CRC bytes follow the IEND letters, and since the find position is itself the start of the type field, the 4 letters must be included too. With only + 4, the cut ends at the IEND letters and the last 4 bytes (CRC) are lost — strict viewers and our CRC verification fail.

Answer 4. Signature repair works: when only a few leading bytes are broken and the body (IHDR, IDAT) is intact. Doesn’t work: when IDAT data is damaged or a CRC is wrong (revealed as CORRUPT in 3-6’s verification), or when IHDR’s width/height fields are destroyed. Those need a second-round repair — estimating correct values field by field against the format documentation — and that’s the real reason hex editors exist.

Completion Criteria Checklist

[ ] I can name the file type instantly from the 10-signature table
[ ] I wrote the Python identify() identifier and saw through the disguised file
[ ] I know that distinguishing the ZIP siblings (docx/jar/apk) requires checking the internal file list
[ ] I repaired a broken PNG’s signature and restored its file verdict
[ ] I extracted a PNG via signature-to-footer carving and verified it with CRC
[ ] I can explain that binwalk’s -e/--dd is manual carving automated
[ ] Mission: 3 signatures added + deep_scan + JPEG carving complete

6. Common Pitfalls & Fixes

Wall 1. binwalk: command not found

Symptom: bash: binwalk: command not found (measured 2026-09-09 — not present in this environment).
Cause: the tool isn’t installed.
Fix: today’s exercises all work with Python find — binwalk’s core action is itself a signature scan. When you genuinely need embedded-file extraction, consider sudo apt install binwalk then.

Wall 2. find returns -1

Symptom: you searched for a signature and got position -1.
Cause: one of three — ① you opened the file in text mode ("r") and the bytes got mangled, ② you wrote the signature value wrong (e.g., wrote 89 50 as the string "8950" instead of x89x50), ③ it really isn’t that format.
Fix: check open(path, "rb") first, then check that the signature is a byte literal like b"x89PNGrnx1an". If it’s still -1, look at the actual leading bytes with xxd FILE | head.

Wall 3. The carved file won’t open

Symptom: extraction succeeded but no viewer opens it.
Cause: mostly a wrongly cut end — you dropped the + 8, or the footer pattern appeared earlier inside the data.
Fix: pinpoint what’s broken with 3-6’s CRC verification routine. For PNG, chunk-level verification points exactly at the damage.

Wall 4. file only says "data"

Symptom: broken.png: data (measured 2026-09-09).
Cause: the signature is destroyed, so it matched nothing in the magic database. "Unidentified," not "useless."
Fix: look at the front with xxd | head, estimate which format’s remains these are, and repair the header as in 3-5. Structural markers like IHDR mid-data are your clues.

Wall 5. I assumed zip and the unzip broke

Symptom: unzip errors out.
Cause: the ZIP signature (PK..) is present but the central directory at the file’s end is broken or truncated. A ZIP’s key directory is at the end, not the start.
Fix: check with xxd FILE | tail whether 50 4B 05 06 (end of central directory) is there. If it’s truncated, the file up to that point likely isn’t the original.

7. Summary

Today’s Concepts

Concept	One-line explanation
Magic bytes	The identity marker at a file’s front — this, not the extension, is the evidence
Header & footer	Start and end markers — cut between them for one file
Carving	A recovery technique that tears the signature-to-footer region out of a byte blob
ZIP siblings	docx·jar·apk are all ZIP — distinguish by the internal file list
Signature length & reliability	Shorter means more false positives/misses — check longest signature first
CRC verification	The procedure that mathematically confirms an extraction is intact

Today’s Commands & Code

Command/code	What it does
`file FILE`	Identify the type via the magic-byte DB
`xxd -l 8 -p FILE`	Print the first 8 bytes as hex
`xxd -s OFFSET -l LENGTH FILE`	View a desired region mid-file
`blob.find(SIGNATURE)`	Find carving’s starting point
`zlib.crc32(type+data)`	Verify PNG chunk integrity
`binwalk -e FILE` (screen example)	Automatically extract embedded files

An Instinct More Important Than Commands

The 40-line identifier and carving script you built today run on the same logic as the hearts of file, binwalk, and PhotoRec. "Installing and using a tool" and "knowing the principle and doing it by hand" are different levels of skill — the difference shows when a tool goes silent or misjudges (like today’s BMP).

And one more thing. Today you broke files and fixed them. Because you personally tried the ways an attacker hides evidence (header destruction, extension disguise, file burial), you can now find those traces as a defender. Every technique in forensics stands on this symmetry.

Once every box is checked, Step 239 is complete.

Step 239. Hex and File Signatures: file, binwalk, Manual Carving — An Identity Check on Six Bytes

Step 239. Hex and File Signatures: file, binwalk, Manual Carving — An Identity Check on Six Bytes

1. Learning Objectives

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. Magic Bytes — A File’s Front Yard

2-2. Headers and Footers — A File’s Start and End Markers

2-3. The ZIP Family’s Sibling Problem

2-4. binwalk — Carving, Automated (Screen Example)

3. Follow Along

3-1. Building the Signature Lab

3-2. Completing the 10-Signature Table Yourself

3-3. Checking Against file — and One Twist

3-4. Writing a Python Identifier — a Miniature file

3-5. Manually Repairing a Broken Header

3-6. Manual Carving — Extracting a PNG from a Pile of Garbage

4. Missions & Exercises

Mission — Completing the Signature Ledger and a Carving Tool

Exercises

5. Model Answers & Completion Criteria

Mission Model Answer

Exercise Answers

Completion Criteria Checklist

6. Common Pitfalls & Fixes

7. Summary

Today’s Concepts

Today’s Commands & Code

An Instinct More Important Than Commands

3-3. Checking Against `file` — and One Twist

3-4. Writing a Python Identifier — a Miniature `file`