Step 219. Anti-Debugging Techniques and Bypasses — Programs That Dodge Debuggers, and How to Break Through

Level 3 — Advanced Reversing | Difficulty ★★★★☆ | Estimated time: 4 hours

Prerequisites: Step 178 (CTF First Taste 3: Reversing). You can use gcc and gdb on WSL Ubuntu, and you’re comfortable with basic gdb commands like break/x/s.

⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. The samples you’ll analyze today are practice binaries you compile yourself.

• What you need: WSL Ubuntu (measured: Ubuntu 24.04, gcc 13.3.0, gdb 15.1, x86-64). A working folder of ~/lab219_223 is recommended.
• Caution: anti-debugging bypass techniques are essential for malware analysis, but applying them to someone else’s commercial software (games, DRM) raises license and legal issues. Today’s only target is "samples I made myself."

In Step 178 you met a well-behaved crackme — it asked for a password, obediently compared it, and told you the result. Real-world binaries are different. Malware and commercial protection schemes check for themselves "am I under a debugger right now?" and, if detected, terminate or behave deceptively. This is anti-debugging. Today you build a watching program yourself, then break through its surveillance net with gdb. You have to know the building side to see the breaking side.

1. Learning Objectives

By the end of this chapter, you will be able to:

• Explain what anti-debugging is and why it’s used
• Explain the principle of the ptrace self-check (PTRACE_TRACEME) and confirm it getting detected in gdb
• Demonstrate with hands-on measurement how a timing check (rdtsc) catches a debugger
• Bypass checks by manipulating a check function’s return value with gdb’s return (int)0
• Organize a catalog of Windows-family detection techniques (IsDebuggerPresent, etc.)

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. Anti-Debugging — A Device for Chasing Analysts Away

Anti-debugging is the collective term for techniques where a program checks for itself "am I being analyzed right now?" and, on detection, responds with termination, deception, or destruction. Malware uses it to dodge analysts’ debuggers; games and DRM use it to block cheats and cracks.

There are dozens of check methods, but the gist is three. ① Ask the OS or CPU directly (is a debugger attached?), ② measure time (execution is slow under a debugger), ③ lay traps (deliberately raise an exception and see if a debugger catches it). Today you implement ① and ② yourself.

2-2. The ptrace Self-Check — "Someone Is Already Watching"

On Linux, a debugger attaches to a target process via a system call called ptrace. And there’s one rule — only one tracer can attach to a process.

The ptrace(PTRACE_TRACEME, ...) self-check exploits this rule in reverse. It’s a request saying "please trace me" — and if gdb is already attached, the request fails (returns -1). In other words, failure itself is detection. Succeeds in a normal run, fails under a debugger — nothing is simpler as a debugger detector.

Windows has an API that does the same job, IsDebuggerPresent() (it reads a flag in the Process Environment Block, PEB). Different operating system, same mindset: ask the OS, "am I being watched?"

2-3. Timing Checks — If It’s Slow, It’s a Debugger

The second technique is measuring time. rdtsc is an instruction that reads the CPU’s Time Stamp Counter, returning clock ticks elapsed since boot. Read it twice, before and after a short code fragment, and the difference gives you the execution time.

Key fact: when you stop at a breakpoint in a debugger or single-step instruction by instruction, all of that human time in between lands in the counter. A stretch that normally takes thousands of clocks balloons to billions with a single breakpoint. "This stretch took abnormally long = someone stopped and peered in" — that’s a timing check.

2-4. The Basic Bypass — Flip the Check Function’s Return Value

If the check lives in a function, the bypass is simple. When a check function returns "detected=1, safe=0," you set a breakpoint at the function’s entry and use gdb’s return (int)0 to make it return 0 without executing the function body. The checking code ends up seeing "safe" forever.

A more permanent method is patching — overwriting the check code in the binary itself with NOPs (do-nothing instructions) or flipping a conditional branch. It’s a common approach in x64dbg and Ghidra; today we focus on the gdb bypass. In the field, "debugger-hiding plugins" like ScyllaHide do this automatically — a screen example comes in 3-7.

3. Follow Along

3-1. The Lab — Building a Program That Watches

Become the author and build a sample armed with anti-debugging. Work in the ~/lab219_223 folder.

Input (guard219.c)

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/ptrace.h>
#include <x86intrin.h>

__attribute__((noinline))
static int check_ptrace(void){
    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) return 1;   /* failure = already being traced */
    return 0;
}

__attribute__((noinline))
static void measured_work(void){
    volatile int s = 0;
    for (int i = 0; i < 1000; i++) s += i;                  /* the measured target: a short task */
}

__attribute__((noinline))
static int check_timing(void){
    unsigned long long a = __rdtsc();
    measured_work();
    unsigned long long b = __rdtsc();
    printf("[timing] elapsed clocks: %llun", b - a);
    return (b - a) > 100000000ULL;                          /* over 100M clocks = suspicious */
}

int main(void){
    char pw[64];
    if (check_ptrace()) { puts("[!] 디버거 감지: ptrace"); return 1; }   /* "[!] debugger detected: ptrace" */
    if (check_timing()) { puts("[!] 디버거 감지: 타이밍"); return 1; }   /* "[!] debugger detected: timing" */
    printf("password: ");
    if (scanf("%63s", pw) != 1) return 1;
    if (strcmp(pw, "n0-debug-z0ne") == 0)
        puts("correct! FLAG{ant1_d3bug_byp4ss3d}");
    else
        puts("wrong.");
    return 0;
}

__attribute__((noinline)) is a gcc directive meaning "don’t inline this function." Without it you can’t set breakpoints by function name in gdb.

Compile and run normally

mkdir -p ~/lab219_223 && cd ~/lab219_223
gcc -O1 -o guard219 guard219.c
echo n0-debug-z0ne | ./guard219

[timing] elapsed clocks: 5466
password: correct! FLAG{ant1_d3bug_byp4ss3d}

(Measured 2026-09-09. The elapsed clock count varies with CPU and load — anywhere in the thousands to tens of thousands is normal.)

How to read the output: run without a debugger, it quietly passes both checks, and with the right password you get the FLAG. The measured stretch was 5466 clocks — one twenty-thousandth of the 100-million threshold. Remember this number. It’s about to balloon enormously.

3-2. Static Analysis — Finding the Checkpoints in the File

Now forget the source and become the analyst. The first move is always strings (the rhythm from Step 178).

strings ./guard219 | grep -iE "timing|password|correct|wrong"

[timing] 
password: 
wrong.
correct! FLAG{ant1_d3bug_byp4ss3d}
check_timing

(Measured 2026-09-09.)

A discovery to note: the Korean detection message [!] 디버거 감지: ptrace ("[!] debugger detected: ptrace") is not visible. strings by default only catches 7-bit ASCII, so UTF-8 Korean text gets cut. To see 8-bit characters too, give it the -e S option:

strings -e S ./guard219 | grep "감지"

[!] 디버거 감지: ptrace
[!] 디버거 감지: 타이밍

(Measured 2026-09-09.)

How to read it: the mere fact that "there are two kinds of detection messages" is itself a clue — it means there are at least two checks. Confirm the ptrace call in the disassembly:

objdump -d ./guard219 | grep -B1 -A1 "ptrace@plt" | head -8

00000000000010d0 <ptrace@plt>:
    10d0:	f3 0f 1e fa          	endbr64
--
    1228:	e8 a3 fe ff ff       	call   10d0 <ptrace@plt>
    122d:	48 83 f8 ff          	cmp    $0xffffffffffffffff,%rax

(Measured 2026-09-09.)

How to read it: right after call ptrace comes cmp $0xffffffffffffffff — 0xFFFF…FF is -1. Code checking "is ptrace’s return value -1?" — the trace of a self-check. Let’s also see main’s call order:

gdb -batch -ex "disassemble main" ./guard219 | grep call | head -4

   0x00000000000012a9 <+26>:	call   0x120b <check_ptrace>
   0x00000000000012b2 <+35>:	call   0x123c <check_timing>
   0x00000000000012ce <+63>:	call   0x10e0 <__printf_chk@plt>
   0x00000000000012e2 <+83>:	call   0x10f0 <__isoc99_scanf@plt>

(Measured 2026-09-09.)

How to read it: main immediately calls check_ptrace → check_timing in that order. The structure forces you through two checkpoints before the password input (scanf).

3-3. Measuring Detection — Just Running It Under gdb

Run it under gdb with no countermeasures:

echo AAAA > in.txt
gdb -batch -ex "run < in.txt" ./guard219

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[!] 디버거 감지: ptrace
[Inferior 1 (process 866) exited with code 01]

(Measured 2026-09-09.)

How to read it: it exited before even taking a password. With gdb attached via ptrace, the program requests TRACEME again → fails (-1) → "detected" verdict. The principle from 2-2 was measured exactly as stated. This is the analyst’s first barrier.

3-4. Bypass 1 — Neutralizing the ptrace Check

Set a breakpoint at check_ptrace‘s entry and make it return 0 (safe) without executing the function body. Create a gdb command file (bypass1.gdb):

break check_ptrace
run < in.txt
return (int)0
continue

gdb -batch -x bypass1.gdb ./guard219

Breakpoint 1 at 0x120b

Breakpoint 1, 0x000055555555520b in check_ptrace ()
[timing] elapsed clocks: 19578
password: wrong.
[Inferior 1 (process 973) exited normally]

(Measured 2026-09-09.)

How to read it: the ptrace detection message is gone — because the check function was fooled into returning 0. This time the timing check passed too — 19578 clocks. After return (int)0 it ran straight through with no breakpoints, so the measured stretch was fast. The password AAAA was wrong, hence wrong. — but the checkpoint was passed. Half a success.

Note: in gdb 15, writing return 0 without a cast gives the error Return value type not available for selected stack frame. Please use an explicit cast of the value to return. (measured 2026-09-09, on a binary without debug info). Specify the type as in return (int)0. Covered again in Wall 1.

3-5. Measuring the Timing Check Firing — Caught the Moment You Stop

Let’s see whether the timing check really detects "stopping." Set a breakpoint in the middle of the measured stretch (measured_work), stall there for 2 seconds, then continue. shell sleep 2 stands in for "the time a human spends reading code while stopped" (timing_demo2.gdb):

break check_ptrace
break measured_work
run < in.txt
return (int)0
continue
shell sleep 2
continue

Breakpoint 2 at 0x11e9

Breakpoint 1, 0x000055555555520b in check_ptrace ()

Breakpoint 2, 0x00005555555551e9 in measured_work ()
[timing] elapsed clocks: 7397204434
[!] 디버거 감지: 타이밍
[Inferior 1 (process 1209) exited with code 01]

(Measured 2026-09-09.)

How to read it: the decisive scene. A stretch that was normally 5466 clocks became 7.3 billion clocks from a 2-second stall, exceeding the threshold (100 million) and firing "debugger detected: timing." In real debugging, even without sleep, the few seconds a human spends stopped at a breakpoint reading the screen get captured as-is. The analyst’s thinking time itself becomes the detection signal — that’s the stubborn part of timing checks.

3-6. Bypass 2 — Breaking Both Checks with One Script

Apply the same technique to check_timing. Feed the correct password and go all the way (bypass2.gdb):

break check_ptrace
break check_timing
run < in.txt
return (int)0
continue
return (int)0
continue

echo n0-debug-z0ne > in.txt
gdb -batch -x bypass2.gdb ./guard219

Breakpoint 1 at 0x120b
Breakpoint 2 at 0x123c

Breakpoint 1, 0x000055555555520b in check_ptrace ()

Breakpoint 2, 0x000055555555523c in check_timing ()
password: correct! FLAG{ant1_d3bug_byp4ss3d}
[Inferior 1 (process 1063) exited normally]

(Measured 2026-09-09.)

How to read it: two checkpoints opened in turn, but both were neutralized by return (int)0, and the program ran to the end and printed the FLAG. The timing measurement wasn’t even executed — because the function body was skipped wholesale. When checks are cleanly separated into function units, the bypass is just as cleanly function-level.

3-7. Doing the Same Automatically — ScyllaHide (Screen Example)

In real analysis you don’t manually return 0 at every check. Tools like the ScyllaHide plugin for x64dbg automate "debugger hiding." Since this environment doesn’t have x64dbg (a Windows tool), we show it as a screen example:

# Screen example — ScyllaHide usage flow (x64dbg, Windows)
1. Load the target exe in x64dbg → Plugins > ScyllaHide
2. Check the techniques to bypass in the profile:
   [x] IsDebuggerPresent      [x] CheckRemoteDebuggerPresent
   [x] NtQueryInformationProcess  [x] Timing (rdtsc/GetTickCount family)
3. Apply and run → detection APIs return "none" even with the debugger attached

How to read it: what it does is the same as what you did by hand today — making every detection point answer "safe." The difference is that it handles dozens of detection kinds in bulk and hides the debugger itself. Even when a tool automates it, "what is being hidden" is exactly the principle you learned today.

4. Missions & Exercises

Mission — Add a Third Checkpoint and Break Through It

1. Add a third check check_length to guard219.c — called before check_timing, takes no arguments, returns an integer, and its body treats a simple computation (e.g., a product of two numbers) as "detected" if it’s nonzero (the check logic is up to you — what matters is the structure where main branches on this function’s return)
2. After compiling, confirm the new check’s call site with disassemble main
3. Write bypass3.gdb that neutralizes all three checks with return (int)0 and reaches the FLAG
4. Leave a record: "how many checks were there, and in what order did you neutralize them"

Exercises

Exercise 1. Explain why the ptrace(PTRACE_TRACEME, ...) self-check means "failure = debugger," from the perspective of the tracer rule.

Exercise 2. In 3-5, a 2-second stall at a breakpoint inflated the elapsed clocks from 5466 to 7.3 billion. Explain why this difference arises, based on what rdtsc measures.

Exercise 3. The return (int)0 bypass doesn’t execute the function body. Yet in 3-4 the timing check "passed" — why? (Hint: what state was the timing check in at that point?)

Exercise 4. If today’s sample’s checks were mixed inline into main instead of being a separated function like check_ptrace(), how would the bypass have differed? Answer with which gdb techniques would have been needed.

__attribute__((noinline))
static int check_length(void){
    volatile int x = 7, y = 6;
    return (x * y) != 42;   /* true → 1 = "detected" — in fact always 0 (safe) */
}

break check_ptrace
break check_length
break check_timing
run < in.txt
return (int)0
continue
return (int)0
continue
return (int)0
continue

6. Common Pitfalls & Fixes

Wall 1. I typed return 0 and got a cast error

Symptom: Return value type not available for selected stack frame. Please use an explicit cast of the value to return.
Cause: on a binary compiled without debug info, gdb doesn’t know the function’s return type. An error actually encountered during this chapter’s measurements (gdb 15.1).
Fix: specify the return type as in return (int)0. For a function returning a pointer, use return (void*)0.

Wall 2. Korean messages don’t show up in strings

Symptom: detection messages clearly exist, but strings search finds nothing (measured 2026-09-09: 0 hits for "감지" with default strings).
Cause: strings by default only catches runs of 7-bit ASCII. UTF-8 Korean has every byte at 0x80 or above, so it gets cut.
Fix: catch 8-bit characters with strings -e S ./binary. English messages and flags appear with the default option too, so make a habit of running both.

Wall 3. print/x $rax in a gdb script errors with The history is empty.

Symptom: inside a script (-x file), print-family commands after finish produce an error. A problem actually encountered during measurement.
Cause: an entanglement between batch-script execution order and gdb’s value-history handling — the same command works when passed as an -ex argument.
Fix: when viewing registers, use info registers rax (short: i r rax) instead of print. This command doesn’t use value history, so it’s stable in scripts.

Wall 4. I stopped, but the timing detection didn’t fire

Symptom: you paused briefly at a breakpoint and it passed with no detection message.
Cause: batch-mode breakpoint handling takes only a few milliseconds and may not cross the threshold (measured 2026-09-09: batch-mode stalls alone reached 2.26 million clocks, below the 100-million threshold). Or the stall location may be outside the measured stretch.
Fix: a human stopping for a few seconds in interactive gdb always gets caught. For demos or automation, put shell sleep 2 in the script to make the stall explicit.

Wall 5. I bypassed one and got detected again — there are multiple checks

Symptom: you neutralized one spot and a different detection message appeared.
Cause: real-world binaries plant the same check in multiple places or layer different techniques. Today’s sample had two as well.
Fix: as in 3-2, collect all detection-message strings with strings -e S first, then backtrack the references to each message in the disassembly and make a checkpoint list first. Bypassing one at a time without a list has no end. Analysis starts with drawing the map.

7. Summary

Today’s Concepts

Today’s Commands

An Instinct More Important Than Commands

Today’s core is the symmetry of attack and defense. The checking side looks for "traces of surveillance"; the bypassing side looks for "the check’s location." The detection-message strings become the very map of checkpoints, and when a check is separated into a function, the bypass was a single line of return (int)0.

Looked at in reverse from the defender’s side, you now understand why real-world protection schemes mix checks inline, layer multiple techniques, and quietly misbehave instead of dying on detection — because fixing today’s sample’s weaknesses points exactly in the real world’s direction. Only those who have built can break, and only those who have broken can build properly.

Once every box is checked, Step 219 is complete.