Step 169. Encryption and Detection Evasion Concepts — The Arms Race Between the Hiders and the Seekers

Level 2 — Introduction to Security and Attack Skill Basics | Difficulty ★★☆☆☆ | Estimated time: 2 hours

Prerequisites: the XOR concept from Step 90 (the XOR file encryption tool), and Step 168’s malware classification and life cycle.

• What you need: Python 3, a Linux lab (WSL or Kali, with gcc). You can follow along entirely without internet access.
• ⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime.
• Character note: today is a [concept] chapter. Instead of building a "technique" that slips past detection, you learn the principle of what an antivirus looks at when it catches something.

Antivirus software and attackers have been playing an old game of hide-and-seek. When the antivirus stores the "fingerprints of known malware," the attacker wraps the same code in a different shape to change the fingerprint. When the antivirus starts watching "behavior," the attacker tries to disguise even the behavior as normal. Today we learn the rules of this hide-and-seek — obfuscation (tangling code), packing (wrapping executables), and the two axes of detection: signature-based and behavior-based detection. We will not build malware. With harmless strings and a 10-line C program you write yourself, you’ll verify by measurement "what changes when you hide, and what can’t be changed."

---

1. Learning Objectives

By the end of this chapter, you will be able to:

• Explain the difference between obfuscation and encryption from the perspective of "the key that reverses it"
• Compare the strengths and weaknesses of signature-based and behavior-based detection
• Hide and restore a string with base64, hex, and XOR
• Demonstrate "the same program with a changed face" using strings and sha256sum
• Explain with a case study the principle that "behavior is hard to hide"

---

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. Antivirus Catches Things Two Ways

Antivirus detection comes in two broad kinds.

First, signature-based detection. It stores fingerprints of known malware — hashes, specific byte patterns, specific strings — in a database, and catches a file when that fingerprint is found in it. The fingerprint database is updated several times a day.

Second, behavior-based detection. Instead of how a file looks, it watches what it does while running. "A document program suddenly launched PowerShell," "right after execution it opened hundreds of files and rewrote their contents" — behavior that normal programs rarely perform is the suspect.

2-2. Obfuscation ≠ Encryption — Whether There Is a Key

Obfuscation is "tangling something to make it hard to read." Converting to base64, splitting a string into small pieces, renaming variables to meaningless letters — things like that. The key point is this: there is no key. Anyone who knows the procedure can reverse it.

Encryption, on the other hand, must be impossible to reverse without a key. As you confirmed in Step 90, a cipher’s strength comes from the key space. That’s why "I encoded it in base64" is never "I encrypted it" — it’s also the reason accidents of storing passwords in base64 still happen today.

2-3. Packing — Wrapping an Executable

Packing is a technique that compresses and transforms an executable to hide its original byte pattern. The representative tool is UPX — and UPX itself is a legitimate compression tool. The problem is that a packed file has a completely different signature. That’s why malware authors love it too, and why antiviruses use the very fact that "it’s packed" as a suspicious signal.

2-4. The Structure of the Arms Race — Why It Never Ends

This relationship sums up in one line:

Antivirus: stores fingerprints → Attacker: changes the fingerprint (obfuscation/packing) → Antivirus: watches behavior
→ Attacker: disguises behavior (imitates normal programs) → Antivirus: behavior combinations, reputation, AI ...

This is not a game where one side wins and it ends. That’s why defenders layer signatures and behavior — signatures are fast and accurate but easy to evade; behavior is hard to evade but has false positives (mistaking something innocent for malicious). Layering two methods with different weaknesses is the industry standard.

---

3. Follow Along

3-1. Hiding One String Three Ways

Let’s hide the harmless string http://files.local/readme.txt in three ways. This isn’t malware — it’s an experiment to see the principle of "changing the face."

Input: obfuscation.py:

import base64

original = "http://files.local/readme.txt"
print("original:", original)

b64 = base64.b64encode(original.encode()).decode()
print("base64:", b64)
print("base64 restored:", base64.b64decode(b64).decode())

hexed = original.encode().hex()
print("hex:", hexed)
print("hex restored:", bytes.fromhex(hexed).decode())

key = 42
xored = bytes([b ^ key for b in original.encode()])
print("xor bytes:", xored)
print("xor restored:", bytes([b ^ key for b in xored]).decode())

Output (measured 2026-09-09):

original: http://files.local/readme.txt
base64: aHR0cDovL2ZpbGVzLmxvY2FsL3JlYWRtZS50eHQ=
base64 restored: http://files.local/readme.txt
hex: 687474703a2f2f66696c65732e6c6f63616c2f726561646d652e747874
hex restored: http://files.local/readme.txt
xor bytes: b'B^^Zx10x05x05LCFOYx04FEIKFx05XOKNGOx04^R^'
xor restored: http://files.local/readme.txt

How to read it: all three faces are different. To a human eye — and to a signature engine that searches strings literally — the original is invisible. Yet all three restore to exactly the original. The execution result is the same means the program’s "behavior" hasn’t changed one bit. That is today’s first conclusion — obfuscation changes the fingerprint but cannot change the behavior.

3-2. Same Program, Two Faces — The strings Experiment

This time let’s confirm it in real executables. We build two programs: a C program with the string embedded plainly, and a program that hides the same string with XOR and restores it at runtime. Their behavior is completely identical.

Input (in your Linux lab):

// plain.c — version where the string is visible as-is
#include <stdio.h>
int main(void){ printf("http://files.local/readme.txtn"); return 0; }

// hidden.c — version with the string hidden by XOR
#include <stdio.h>
#include <string.h>
int main(void){
  unsigned char enc[] = {0x42,0x5e,0x5e,0x5a,0x10,0x05,0x05,0x4c,0x43,
    0x46,0x4f,0x59,0x04,0x46,0x45,0x49,0x4b,0x46,0x05,0x58,0x4f,0x4b,
    0x4e,0x47,0x4f,0x04,0x5e,0x52,0x5e};
  char buf[sizeof(enc)+1]; size_t i;
  for(i=0;i<sizeof(enc);i++) buf[i]=enc[i]^42;
  buf[i]=0;
  printf("%sn", buf);
  return 0;
}

Input (compile and run):

gcc -o plain plain.c
gcc -o hidden hidden.c
./plain
./hidden
strings plain | grep "files.local"
strings hidden | grep "files.local" || echo "(hidden: string not visible)"

Output (measured 2026-09-09):

http://files.local/readme.txt
http://files.local/readme.txt
http://files.local/readme.txt
(hidden: string not visible)

How to read it: both runs print the same single line. But viewed with strings (a tool that extracts readable strings from an executable), the string appears plainly in plain and is absent in hidden. It’s a program doing the same thing, with only its face different — and that face is exactly what signature detection looks at.

3-3. The Fingerprint Changed — Comparing Hashes

The most-used file fingerprint is the hash. Let’s compare the two files’ fingerprints.

Input:

sha256sum plain hidden
ls -l plain hidden

Output (measured 2026-09-09):

72f24c30479f9a800d3071b4d8f26c2dd4c1a598a72eebc5dfc8adb4e2f6c34d  plain
f0061c5b7129bdb6ed6889d4acbc736a07ec650e5cc1593704733c54720792e1  hidden
-rwxr-xr-x 1 root root 16016 ... hidden
-rwxr-xr-x 1 root root 15960 ... plain

How to read it: the hashes are completely different. A database saying "the file with this hash is malicious" is powerless against such a transformation — change the hash and it’s a different file. This is why real malware spreads with small mutations. That’s why modern antiviruses don’t rely on a single hash but look at byte patterns, structural features, and behavior together.

Why do this: with two files made by your own hands, you’ve proven that "change the surface and fingerprint detection slips by." Now the opposite question remains — what else does the antivirus look at?

3-4. EICAR — The ‘Fake Virus’ That Antiviruses React To

There is a standard, safe way to watch an antivirus catch something by signature: the EICAR test file — a 68-character string with no functionality whatsoever, created by the European Institute for Computer Anti-Virus Research. Nearly every antivirus in the world is agreed to detect this string as "test malware."

Since this exercise requires external lookups, we replace it here with a screen example. On a computer with antivirus running, save the EICAR string as a file or upload it to a scanning service like VirusTotal, and you’ll see a screen where most engines detect it as "EICAR-Test-File."

Screen example (what uploading the EICAR file to VirusTotal looks like):
Engine A: EICAR-Test-File (not a virus)
Engine B: Test.EICAR
Engine C: Eicar-Test-Signature
... (dozens of engines, all detecting it)

How to read it: this string can do no harm, yet everything catches it — purely because "the fingerprint is in the DB." It’s a scene where the power and the limit of signature detection show at the same time. A fingerprint is certain, but it can’t touch something whose fingerprint changed (like hidden in 3-2).

Caution: an antivirus quarantining or deleting the EICAR file is normal behavior. When the test is done, just clean it up from quarantine.

3-5. Behavior Is Hard to Hide — Reading a Case

A representative combination that behavior-based detection catches is "a document spawning a shell." A normal Word document has no reason to run PowerShell. So when a process parent-child relationship like winword.exe → powershell.exe appears, an EDR (Endpoint Detection and Response — endpoint behavior monitoring) raises an alert no matter how cleverly the file is obfuscated.

Screen example (typical wording of an EDR alert):
Suspicious process tree detected:
  winword.exe (PID 5120)
    └─ powershell.exe -enc aHR0cDov... (PID 6012)
Rule: Office application spawned script interpreter

How to read it: the base64 string after -enc is an obfuscated command — the very technique we did in 3-1. The obfuscation hid the content, but it could not hide the behavior that "Word launched PowerShell." This is exactly where the attacker’s wrapping technique and the defender’s detection principle face each other.

Why do this: today’s second conclusion — you must know evasion techniques to design defenses. Whichever side you go to, this structure is non-negotiable.

---

4. Missions & Exercises

Mission — Completing the "Two-Axis Detection Comparison Table + Arms-Race Diagram"

1. Build your own table comparing signature-based and behavior-based detection — use at least 4 comparison items (what it looks at, speed, evasion difficulty, false positives)
2. Redraw the 2-4 arms-race cycle in your own words — attach to each stage, in parentheses, one piece of evidence from today’s labs (one of 3-1~3-5)
3. Explain the difference between obfuscation and encryption in two sentences or fewer using "whether there is a key"
4. Save it as detection-principles.md in your Step 89 wiki — a one-line summary / the comparison table / the diagram / today’s 3 measured commands

Exercises

Exercise 1. A colleague says they’ve "encrypted" a password by converting it to base64. Explain to them what’s wrong.

Exercise 2. In 3-2, hidden‘s string wasn’t caught by strings. Can you judge whether this program is "safe"? Answer from the signature perspective and the behavior perspective separately.

Exercise 3. Explain why hash-based detection (3-3) is powerless against obfuscation, in terms of "the properties of hash functions."

Exercise 4. Explain why signatures and behavior detection are layered, using each method’s weakness.

---

Answers & completion criteria · expand/collapse

5. Model Answers & Completion Criteria

Mission Model Answer

Here’s an example of the comparison table — put the wording in your own words.

Item	Signature-based	Behavior-based
What it looks at	The file’s fingerprint (hash, pattern, strings)	Actions while running (processes, files, network)
Speed	Fast (can pre-scan)	Slow (must watch it run)
Evasion	Easy — just change the surface (measured 3-2, 3-3)	Hard — must change the behavior too (3-5)
False positives	Few (fingerprints are exact)	Present — normal programs can act suspiciously too

Example of the arms-race diagram: "fingerprint DB (confirmed with EICAR, 3-4) → fingerprint changed by obfuscation (measured 3-1~3-3) → behavior monitoring appears → normal-program mimicry (the 3-5 process-tree deception attempt) → behavior combination analysis…". If each arrow carries one piece of evidence you saw today, it’s complete.

How to verify: ① does the table have the four items? ② does each stage of the diagram have today’s lab evidence attached in parentheses? ③ does the word "key" appear in your "obfuscation ≠ encryption" explanation? All three being "yes" means complete.

Exercise Answers

Answer 1. base64 is a keyless transformation — that is, obfuscation, not encryption. The whole world knows the transformation procedure, so anyone can restore it instantly (in the 3-1 measurement it came back in one line). To be called encryption, it must be impossible to reverse without a key, and its strength must come from the key space.

Answer 2. From the signature perspective, the string isn’t visible, so "this file" alone can’t be matched against known fingerprints — it looks clean on the surface. But from the behavior perspective, everything it does when executed is exposed. Today’s example does nothing but print one line on screen, so it’s harmless — but if it had been deleting files or reaching out to the network, behavior detection would catch it. "Not caught by fingerprints" and "safe" are different sentences.

Answer 3. A hash function produces a completely different output when even one byte of the input changes (3-3 measurement: the two hashes are entirely different). Since obfuscation is work that changes bytes, the hash necessarily changes and no longer matches the fingerprint stored in the DB. A hash is a tool that finds only "the exact same file."

Answer 4. Signatures are fast with few false positives but powerless against surface transformations (3-2); behavior sees through transformations but has false positives and must wait for execution. Use only one side and its weakness becomes a hole as-is. When you layer methods whose weaknesses differ, the attacker must climb both walls at once, so the cost rises sharply — that’s why the field is always layered.

Completion Criteria Checklist

• [ ] I can explain the difference between obfuscation and encryption with "whether there is a key"
• [ ] I hid and restored a string with base64/hex/XOR
• [ ] I measured the difference between the plain and hidden versions with strings
• [ ] I confirmed the fingerprint changes with sha256sum
• [ ] I built a comparison table of the pros and cons of signature/behavior detection
• [ ] I can explain "behavior is hard to hide" with a case study
• [ ] I can say what EICAR is and why it’s a safe test

---

6. Common Pitfalls & Fixes

Wall 1. I get binascii.Error: Incorrect padding

Symptom (measured 2026-09-09, after stripping the = from a base64 string and restoring):

binascii.Error: Incorrect padding

Cause: base64 appends = at the end to make the length a multiple of 4. Strip that, or cut the middle, and restoration is refused.
Fix: when copying a string, grab the whole thing including the trailing =. "Even an invisible single character is data" is the first rule of handling encodings.

Wall 2. I get ValueError: non-hexadecimal number found in fromhex()

Symptom (measured 2026-09-09, after clipping one character off a hex string):

ValueError: non-hexadecimal number found in fromhex() arg at position 7

Cause: in hex, two characters make one byte. An odd length means the pairing breaks and it fails.
Fix: check that nothing was missed when copying, and when generating it yourself, use the output of string.encode().hex() as-is.

Wall 3. I get TypeError: unsupported operand type(s) for ^: 'str' and 'int'

Symptom (measured 2026-09-09, after XORing a string directly):

TypeError: unsupported operand type(s) for ^: 'str' and 'int'

Cause: XOR is an operation between numbers. A string’s (str) characters aren’t numbers as they are.
Fix: as in 3-1, convert to bytes with .encode() and then apply ^ key to each byte. Step 90’s "a file is a list of numbers" is the same rule here.

Wall 4. The hidden string shows up in strings

Symptom: you made hidden, yet the original text appears in strings.
Cause: most often you scanned the source code (the .c file), or scanned plain, which has the string embedded plainly before compilation. Or you left the string in a comment without restore code.
Fix: confirm that the scan target is the compiled binary. And in the XOR-hidden version, the original string must exist nowhere in the source — only the byte array (0x42,0x5e,...) should be there.

Wall 5. Jumping to "I obfuscated it, so it’s safe / it’s cracked"

Symptom: seeing that the string is invisible, you write "detection completely evaded."
Cause: you skipped 3-5 — the fingerprint is hidden but the behavior is unchanged.
Fix: write today’s conclusion sentence as-is — "Obfuscation changes the fingerprint. It cannot change the behavior." Defenders look at both.

---

7. Summary

Today’s Concepts

Today’s Commands & Code

An Instinct More Important Than Commands

Today you made two files that behave identically, and confirmed by hand what collapses (hashes, string fingerprints) and what doesn’t (behavior) when only the surface is changed. This pair of experiments is a miniature of the attack-defense arms race. The names of evasion techniques will keep changing, but the composition — "the one who changes fingerprints and the one who watches behavior" — will not. Don’t memorize techniques; remember the composition — someone who knows the composition knows exactly where a new technique fits when they see it.

---

Once every box is checked, Step 169 is complete.