Step 220. Packing and Unpacking — UPX and Finding the OEP
Level 3 — Advanced Reversing | Difficulty ★★★★☆ | Estimated time: 4 hours
Prerequisites: Step 219 (Anti-Debugging Techniques and Bypasses), Step 178 (Intro to Reversing). You know the basic use of
strings,objdump, and gdb.
⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. Packing and unpacking are basic skills of malware analysis, but unpacking someone else’s software and distributing it raises legal issues.
- What you need: WSL Ubuntu (measured: Ubuntu 24.04). UPX is not installed in this environment, so the UPX commands themselves are shown as output examples, while packing’s core principles (compression = string concealment, stub = runtime restoration device) are measured hands-on with gzip.
- Caution: don’t download arbitrary packed files from the internet and run them. Packing is also malware’s basic disguise.
Step 219’s sample showed its messages plainly in strings. But real-world binaries — especially malware — often catch nothing in strings. That’s because the original code has been wrapped wholesale in compression and encryption. This is packing, and peeling the wrapping back off is unpacking. Today you confirm hands-on why packing hides strings, and learn the unpacking procedure of the industry-standard packer UPX along with the OEP concept.
1. Learning Objectives
By the end of this chapter, you will be able to:
- Explain the structure of a packer — a compressed payload plus an unpacking stub
- Demonstrate with hands-on measurement that compression neutralizes string analysis
- Explain what the OEP (Original Entry Point) is and why it’s the goal of unpacking
- Know the difference between UPX automatic unpacking (
upx -d) and manual unpacking (memory dump) - Explain conceptually the "dumped it but it won’t run" situation — the IAT rebuild problem
2. Background Knowledge — Today’s Tools and Concepts
Today’s Tools at a Glance
| Category | Details |
|---|---|
| Language/environment | WSL Ubuntu bash (measured: Ubuntu 24.04, gzip 1.12, binutils strings/file) |
| Today’s commands | gzip -c, strings, file, upx (not installed → output examples) |
| Concepts needed | Packers and stubs, OEP, tail jump, memory dumping, IAT rebuild |
| Today’s deliverables | A before/after-compression strings comparison record + OEP concept notes |
2-1. The Packer’s Structure — A Compressed Lump and Its Wrapper
A packer is a tool that wraps an executable’s original code and data in compression (or encryption). The resulting structure has two pieces.
- Payload: the compressed original. In this state it can neither run nor have strings extracted.
- Stub: a small restoration code attached at the front of the file. When executed, it unpacks the payload in memory and jumps to the restored original code.
From the outside, the executable is only the stub. What static analysis tools (strings, disassemblers) see is the stub and an inscrutable compressed lump — the original code exists nowhere in the file as plaintext. Think of it as Step 178’s XOR hiding extended to the whole program.
2-2. UPX — The Most Famous "Honest" Packer
UPX (Ultimate Packer for eXecutables) is an open-source compression packer whose original purpose is saving file size. Since it isn’t malicious, it leaves its name and version in the file and restores perfectly with one line of upx -d.
Field sense: a UPX-packed file retains section names like UPX0, UPX1 and an "UPX!" magic. But precisely because of that "honesty," malware authors use modified UPX — modified UPX with the magic erased or the compression scheme changed gets refused by upx -d. "Situations where the automatic route doesn’t work" are the default in the field, and that’s when you need manual unpacking.
2-3. OEP — The Target Point of Unpacking
A packed program’s execution order goes like this: the OS starts the stub → the stub restores the original in memory → it jumps to the original’s starting point. The destination of that final jump is the OEP (Original Entry Point).
The definition of manual unpacking is one sentence: stop execution at the exact moment the stub finishes restoring and jumps to the OEP, and dump the unpacked original from memory. Because the original code, absent from the file, exists in plaintext in memory while running — "whatever you want to hide must be revealed at runtime" (the same law continuing from Steps 178 and 219).
The stub’s final jump is usually one big jmp to a very distant address, so it’s called a tail jump. Manual unpacking practice is, in effect, "finding the tail jump."
2-4. Homework After the Dump — IAT Rebuild
Dumping at the OEP isn’t the end. The address table a program goes through when calling external library functions (like printf) is called the IAT (Import Address Table), and the addresses at dump time are based on that run’s memory layout, so saving them as-is breaks the file in other environments.
That’s why dump tools (like Scylla on Windows) rebuild the address table in the order "IAT Autosearch → Get Imports → Fix Dump." "I dumped it but it won’t run" — 90% of unpacking failures are a missing IAT rebuild. Today we only grasp the concept; hands-on tool practice remains homework for when you have a Windows analysis environment.
3. Follow Along
3-1. Measuring the Principle — Compress, and strings Goes Silent
Even in this UPX-less environment, packing’s first effect (string concealment) can be measured. gzip doesn’t produce an executable, but the principle "compression = information concealment" is identical. We’ll use Step 219’s guard219 as material.
cd ~/lab219_223
strings ./guard219 | wc -l
gzip -c -9 ./guard219 > guard219.gz
stat -c "%s %n" guard219 guard219.gz
strings ./guard219.gz | wc -l
101
16312 guard219
2931 guard219.gz
34
(Measured 2026-09-09.)
How to read the output: the original had 101 strings; the compressed copy has only 34, and most of those are noise. Size shrank from 16312 → 2931 bytes. Decisively, check whether the secret phrase is gone:
strings ./guard219.gz | grep FLAG
strings -e S ./guard219.gz | grep "감지"
(neither command prints anything)
(Measured 2026-09-09.)
How to read it: the FLAG and the Korean detection messages alike evaporated wholesale. Check the file format too and the picture completes:
file guard219 guard219.gz
guard219: ELF 64-bit LSB pie executable, x86-64, ... not stripped
guard219.gz: gzip compressed data, was "guard219", ... original size modulo 2^32 16312
(Measured 2026-09-09.)
How to read it: the compressed copy is no longer an ELF. The executable’s identity — format, strings, structure — is all hidden behind compression. This is half of what a packer does — the other half (the stub that makes it runnable) is the only difference.
3-2. No Stub Means No Execution — Measured
Try running the compressed copy directly:
chmod +x guard219.gz
./guard219.gz
bash: line 1: ./guard219.gz: cannot execute binary file: Exec format error
(Measured 2026-09-09.)
How to read it: an obvious result, but an important one. Compression alone doesn’t make an executable. To run, "code that unpacks it" (a stub) must be attached at the front of the file; the OS runs the stub, and the stub restores the original in memory. This is the difference between gzip and a packer, and the reason a packed file is "executable on the outside, compressed lump on the inside."
3-3. UPX Automatic Unpacking — Output Examples
Since this environment has no UPX (and we won’t install it), the standard procedure is shown as output examples. If you want to try it yourself, install it via the package manager (sudo apt install upx-ucl).
# Output example — based on an environment with UPX installed
upx -9 -o guard219_packed ./guard219
# Output example
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2024
UPX 4.2.4 Markus Oberhumer, Laszlo Molnar & John Reiser May 9th 2024
File size Ratio Format Name
-------------------- ------ ----------- -----------
16312 -> 6420 39.36% linux/amd64 guard219_packed
(Output example — not measured. Actual figures vary by version and file.)
The packed file looks different on the outside:
# Output example
strings ./guard219_packed | head
# Output example
UPX!
UPX!
$Info: This file is packed with the UPX executable packer http://upx.sf.net $
...
(Output example — not measured.)
How to read it: instead of the original strings, UPX’s self-introduction appears. An honest packer wearing a name tag. Automatic unpacking is one line:
# Output example
upx -d guard219_packed
# Output example
16312 <- 6420 39.36% linux/amd64 guard219_packed
(Output example — not measured.)
How to read it: the arrow direction is the reverse of packing (small → large), and it’s back to the original size. UPX knows its own compression scheme, so it restores perfectly. Automatic unpacking is possible "because the packer is known" — when an unknown or modified packer arrives, this button doesn’t exist.
3-4. A Map of Manual Unpacking — Finding the OEP (Concept Procedure)
Here’s the standard procedure for when you meet a binary that upx -d refuses. It’s a screen example based on x64dbg/Scylla (Windows), but the concepts are identical in Linux gdb.
# Screen example — 5 steps of manual unpacking
1. Load the packed binary in a debugger → the starting point is the "stub"
(clue: pushad / pusha — saves all registers at once, to restore them later)
2. Follow the stub code, passing through the restoration loop
(a long repeating loop, then suddenly one jmp to a very distant address)
3. Set a breakpoint on that tail jump → the destination is the OEP
4. Stop execution at the OEP and dump memory with Scylla (Dump)
5. Rebuild the address table with IAT Autosearch → Get Imports → Fix Dump
→ now the original code reads in a disassembler/decompiler
The sense for judging an OEP: when you reach the OEP, the code’s "expression" changes. The stub’s machine-like restoration loop ends, and program-like startup code appears (a standard prologue — setting up a stack frame and calling library initialization). It’s a sense built with experience, but the first clue is always "the distant jump right after the restoration loop."
3-5. The Boundary Between What We Measured Today and What We Didn’t
| Item | Method | Status |
|---|---|---|
| Compression = string/structure concealment | gzip + strings comparison | Measured 2026-09-09 |
| Compression alone can’t execute | Ran the compressed copy directly → Exec format error | Measured 2026-09-09 |
| UPX packing / automatic unpacking | upx commands | Output examples (UPX not installed) |
| 5 steps of manual unpacking | x64dbg + Scylla | Screen example (Windows tools) |
| IAT rebuild | Scylla Fix Dump | Concept explanation |
Knowing the honest boundary is also analysis skill. You felt the principle (concealment and restoration) hands-on, and you have a map of the tool procedure you can reproduce as-is in a real environment.
4. Missions & Exercises
Mission — Write a "Before/After Packing Report"
- Pick Step 219’s guard219 (or any practice binary) and record its pre-packing state —
fileresult,stringscount, FLAG search result - Compress it with gzip and record the same items again
- Record the result of trying to run the compressed copy (
Exec format error), and write in two sentences "why a packer needs a stub" - Final paragraph: organize "if I met a modified packer that required manual unpacking, in what order (5 steps) would I approach it"
Exercises
Exercise 1. Explain why strings goes silent on a packed file, citing the measured figures from 3-1 (101 → 34).
Exercise 2. In 3-2 the compressed copy wouldn’t run, but a UPX-packed file runs. What part creates this difference, and what is that part’s mission?
Exercise 3. Explain why the OEP is called "the target point of unpacking," from the perspective of "the place where the unpacked original code exists."
Exercise 4. What is the most common cause of "dumped it but it won’t run," and why does it happen?
5. Model Answers & Completion Criteria
Mission Model Answer
[Before packing] file: ELF 64-bit LSB pie executable, not stripped
strings: 101 / FLAG search: FLAG{ant1_d3bug_byp4ss3d} found
[After packing] file: gzip compressed data (not an ELF)
strings: 34 / FLAG search: none
[Run attempt] ./guard219.gz → cannot execute binary file: Exec format error
Why a stub is needed: compressed data cannot be executed directly by the CPU.
Someone must unpack it in memory first and jump to the unpacked code,
and that "someone" is the stub attached at the front of the file.
[Manual unpacking approach order] ① enter the stub in a debugger → ② identify
the pushad and restoration loop → ③ break on the tail jump to capture the OEP
→ ④ dump at the OEP → ⑤ static analysis after IAT rebuild
(Autosearch → Get Imports → Fix Dump).
How to verify: do the before/after figures match actual command output (re-run to compare)? Does the stub explanation include both "restore in memory" and "jump"? Are all 5 steps present in order? Especially, step ⑤ must not be missing — it’s the star of Wall 2.
Exercise Answers
Answer 1. The original’s strings are stored as plaintext inside the ELF file, so strings caught 101. Compression rewrites the data into different bit patterns, so the same byte sequences no longer exist in the compressed copy, leaving only 34 pieces of noise. The evidence is that the FLAG and Korean messages both escaped the search. It’s because static analysis’s first tool only reads the file’s "surface."
Answer 2. The stub. The stub is restoration code attached at the front of a packed file, and it’s the actual entry point the OS executes. Its mission is twofold — unpack the compressed payload into the original code in memory, and jump to the unpacked original’s entry point (OEP). The gzip copy lacks this part, which is why the OS refused to run it.
Answer 3. The original code doesn’t exist as plaintext in the file, but at the moment the stub finishes restoring, the plaintext original is in memory. The OEP is the end of that "restoration complete" and the starting point of the original’s execution. Stop there and dump memory, and you obtain the original code that couldn’t be read from the file. That’s why every step of manual unpacking aims at "reaching the OEP."
Answer 4. A missing IAT (Import Address Table) rebuild. When a program calls external library functions it goes through an address table called the IAT, and the table at dump time holds memory addresses valid only in that run. In another run/environment, libraries load at different addresses, so the table becomes invalid, and the moment a wrong address is called, the program breaks. That’s why dump tools perform an IAT rebuild, refilling the address table based on "function names."
Completion Criteria Checklist
- [ ] I can explain a packer’s two parts (compressed payload + stub)
- [ ] I measured and compared strings counts and FLAG search results before and after compression
- [ ] I confirmed the compressed copy doesn’t run and know why
- [ ] I can explain what the OEP and the tail jump are
- [ ] I know when
upx -dworks (when the packer is known) and when it doesn’t - [ ] I can recite the 5 steps of manual unpacking in order
- [ ] I can state in one sentence why an IAT rebuild is needed
6. Common Pitfalls & Fixes
Wall 1. upx -d refuses
Symptom (output example): NotPackedException: not packed by UPX or CantUnpackException.
Cause: the file was wrapped by a packer other than UPX, or it’s likely modified UPX with the magic/section names erased. Automatic unpacking only works "when the packer is known."
Fix: switch to manual unpacking (the 5 steps in 3-4). The refusal itself is information — it means "this file intends to hide."
Wall 2. I dumped it, but it won’t run or dies immediately
Symptom: running the dumped exe terminates immediately or throws an address error.
Cause: missing IAT rebuild — addresses valid only in that run are baked into the dump (2-4).
Fix: don’t skip the Scylla order: "IAT Autosearch" → "Get Imports" → "Fix Dump." If red (failed) entries remain in the Get Imports list, manually assign the correct library functions.
Wall 3. I can’t tell whether it’s the OEP or not
Symptom: you followed the jump and only more machine-like code appeared.
Cause: you picked the wrong tail jump (a jump inside the restoration loop), or the packer restores in multiple layers.
Fix: the real OEP’s expression is "a program-like beginning" — a stack-frame prologue and library initialization calls. If there are multiple layers, repeat the same procedure at each layer. Patience is the tool.
Wall 4. I ran the compressed copy and panicked at Exec format error
Symptom: ./guard219.gz: cannot execute binary file: Exec format error (measured 2026-09-09).
Cause: not an error — normal. A gzip copy is a data file, not an executable, so the OS refuses it.
Fix: this result is exactly today’s measured conclusion — "compression alone can’t run; a stub is needed." Don’t panic; record it.
Wall 5. I’m afraid to run a packed file in case it’s malware
Symptom: manual unpacking is "work done while running it," so you hesitate.
Cause: a rational fear. Packing is malware’s formal dress.
Fix: always do manual unpacking in an isolated analysis VM (a virtual machine with snapshots, network cut). Practice with files you made yourself as you did today, and judge files of unknown origin with static analysis only from the start.
7. Summary
Today’s Concepts
| Concept | One-line explanation |
|---|---|
| Packer | A tool that wraps original code in compression/encryption — turns an executable into data |
| Stub | Restoration code at the file’s front — unpacks in memory and jumps to the OEP |
| UPX | The most famous open-source packer — honest, so automatic unpacking (upx -d) works |
| Modified packer | A packer with magic erased or scheme changed — no automatic unpacking; manual unpacking needed |
| OEP | Original Entry Point — the original’s start address the stub jumps to when restoration ends |
| Tail jump | The one distant jmp from the stub’s end to the OEP — manual unpacking’s target |
| IAT rebuild | Reconstructing the dump’s address table by function name — skip it and the dump won’t run |
Today’s Commands
| Command | What it does |
|---|---|
gzip -c -9 ./binary > out.gz |
Compression for principle practice — confirm string concealment |
strings ./file | wc -l |
Compare string counts before and after packing |
file ./file |
Tell whether it’s an ELF or compressed data |
upx -9 -o output input |
UPX packing (on environments that have it; output example today) |
upx -d packed-file |
UPX automatic unpacking — possible only when the packer is known |
| (x64dbg+Scylla) BP on tail jump → Dump → Fix Dump | The standard manual unpacking procedure |
An Instinct More Important Than Commands
Packing isn’t magic; it’s wrapping. As today’s measurement showed, one compression cuts 101 strings to 34 and evaporates the FLAG — but to run, it must be unpacked sometime, and the moment it’s unpacked, it’s in memory. The great reversing principle "if the file hides it, ask the running program" holds exactly in unpacking too.
And knowing a tool’s boundary is skill. If upx -d works, press the button; if not, unfold the 5-step map. What matters is being able to say for yourself "which stage am I at right now," whichever side you’re on — with that sense, there’s no reason to fear a packer you’ve never seen.
Once every box is checked, Step 220 is complete. Click the checkbox in the sidebar to save your progress.