Step 179. CTF Taste Test 4: Three Crypto Intros — Cryptography as Math Puzzles
Level 3 — Real-World CTF and Advanced Attack Skills | Difficulty ★★★☆☆ | Estimated time: 5 hours
Prerequisites: Step 90 (encoding and XOR), Steps 176–178 (CTF formats and solving habits). You can write basic Python.
⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. Dreamhack (dreamhack.io) is a legal learning platform built to be solved.
- What you need: Python 3 (measured: 3.12.14), a notepad. Everything runs locally, no external platform connection.
- Caution: platform challenge screens are shown only as "Screen example," and the three mini-challenges forming the backbone are built by you and solved hands-on with Python.
CTF’s Crypto category is close to math puzzles. From a given ciphertext and hints (part of the key, parameters, source code), you find the author’s mistake or weak numbers and recover the plaintext. Unlike categories that hunt for intrusion traces, here your weapons are a single line of Python and an eye for "something’s off about this string." Today you build — and break — the three classics of introductory Crypto: encoding detection, XOR, and toy RSA.
1. Learning Objectives
By the end of this chapter, you will be able to:
- Explain the difference between encoding and encryption
- Guess the encoding type from a string’s shape (trailing
=, only 0-9a-f, etc.) - Back-calculate a one-byte XOR key using known plaintext (the flag format
DH{) - Factor a small-n RSA key, recover the private key d, and decrypt with
pow(c, d, n) - Summarize "why it breaks" in one sentence, no formulas
2. Background Knowledge — Today’s Tools and Concepts
Today’s Tools at a Glance
| Category | Details |
|---|---|
| Language/environment | Python 3 (measured: 3.12.14) — interactive shell or scripts |
| Today’s commands | bytes.fromhex(), base64.b64decode(), codecs.decode(s, "rot13"), pow(c, d, n) |
| Concepts needed | Encoding vs encryption, shape clues of Base64/hex/rot13, XOR’s self-inversion, RSA’s public/private keys |
| Today’s artifact | Solutions for three mini-challenges + one line each on "why it breaks" |
2-1. Encoding Is Not Encryption
Encoding is "changing notation." There’s no key, and anyone who knows the rule can reverse it. Base64, hex, and rot13 all belong here. Encryption is what you can’t open without a key.
Half of introductory CTF challenges are encodings. They look like ciphers but are really just strings with a changed appearance — so the first question is always "is this encryption, or encoding?"
2-2. What a String’s Shape Tells You
| Clue | Guess |
|---|---|
Ends in = or == |
Base64 (padding characters) |
| Only 0-9 and a-f, even length | hex (hexadecimal notation) |
| Word-shaped but spelling is odd | rot13 or Caesar (alphabet shift) |
| An unknown blob of bytes | XOR or real encryption |
Not a perfect rule, but in introductory challenges this table alone decides your first attempt. If wrong, move to the next candidate — the loop of guessing and verifying is daily life in Crypto solving.
2-3. XOR — Do It Twice and You’re Back
XOR (exclusive OR) returns the original value when applied twice with the same key. plaintext XOR key = ciphertext, ciphertext XOR key = plaintext. So if you know part of the plaintext, you can back-calculate the key:
ciphertext[0] XOR plaintext[0] = key
CTF flags start with a fixed header like DH{, so XORing the ciphertext’s first byte with D gives you the key. This is the smallest form of a known-plaintext attack.
2-4. One Lap of RSA — Learning Airplane Principles on a Bicycle
RSA is a public-key cryptosystem built on the idea of publishing n, the product of two primes p and q, and keeping the secret key d unknown as long as n can’t be split back apart. The flow:
n = p × q, public key (n, e), private key d = inverse of e (mod (p-1)(q-1))
Encryption: c = m^e mod n Decryption: m = c^d mod n
Real-world n has hundreds of digits and can’t be split. But when a challenge’s n is small — you can split it by trying divisions. Today you’ll verify that Python’s pow(c, d, n) is the whole of decryption. In the field, the standard move is searching n on factordb.com to check whether it’s already been factored (Screen example only).
3. Follow Along
3-1. Preparing the Lab — The Challenge Generator
Let’s build three mini-challenges that copy the skeleton of Dreamhack Crypto difficulty-1 problems verbatim. Save the script below as crypto_lab.py and run it (every output in this chapter was measured 2026-09-09 on Python 3.12.14).
Input (crypto_lab.py — challenge generation part)
import base64
# Challenge 1: Base64 inside hex
flag1 = "DH{hex_4nd_b4se64_l4y3rs}"
b64 = base64.b64encode(flag1.encode()).decode()
hexed = b64.encode().hex()
print("Challenge 1 ciphertext:", hexed)
# Challenge 2: one-byte XOR
flag2 = "DH{x0r_k3y_1s_0n3_byt3}"
key = 0x5A
cipher = bytes(b ^ key for b in flag2.encode())
print("Challenge 2 ciphertext (hex):", cipher.hex())
# Challenge 3: toy RSA
p, q, e = 61, 53, 17
n = p * q
d = pow(e, -1, (p - 1) * (q - 1))
c = pow(ord("A"), e, n)
print(f"Challenge 3: n={n}, e={e}, c={c}")
Output
Challenge 1 ciphertext: 5245683761475634587a52755a4639694e484e6c4e6a5266624452354d334a7a66513d3d
Challenge 2 ciphertext (hex): 1e1221226a2805316923056b29056a34690538232e6927
Challenge 3: n=3233, e=17, c=2790
From now on, you’re a solver who received only these three ciphertexts. Forget the author’s mind (the code above) and solve them by looking only at the shape of the ciphertexts.
3-2. Challenge 1: Encoding Detection — hex or Base64?
Look at the ciphertext: 5245683761475634...66513d3d. Made only of 0-9 and a-f, even length — line two of the 2-2 table, hex is the first candidate.
s = "5245683761475634587a52755a4639694e484e6c4e6a5266624452354d334a7a66513d3d"
step1 = bytes.fromhex(s).decode()
print(step1)
REh7aGV4XzRuZF9iNHNlNjRfbDR5M3JzfQ==
How to read the output: you unwrapped the hex and got not a flag but another string. But look at the end — ==, line one of the table: Base64. The encoding was two layers deep.
import base64
print(base64.b64decode(step1).decode())
DH{hex_4nd_b4se64_l4y3rs}
Why it breaks, in one line: "Encoding is a keyless transform — no matter how many layers, knowing the rules unwinds it all."
3-3. Challenge 2: One-Byte XOR — Back-Calculating the Key from the Flag Header
Unwrapping the second ciphertext 1e1221226a28... from hex doesn’t give readable characters either — the bytes.fromhex result is broken bytes. This time there’s a real transform: XOR. You don’t know the key, but you do know the flag starts with DH{.
cipher = bytes.fromhex("1e1221226a2805316923056b29056a34690538232e6927")
key = cipher[0] ^ ord("D") # ciphertext[0] XOR plaintext[0] = key
print(hex(key))
print(bytes(c ^ key for c in cipher).decode())
0x5a
DH{x0r_k3y_1s_0n3_byt3}
How to read the output: XORing the first byte 0x1e with D (0x44) gave 0x5a, and that single key unlocked everything. As a check, verify the second byte too — 0x12 ^ ord("H") should also be 0x5a.
Why it breaks, in one line: "XOR unwinds with the same key applied twice, and knowing one plaintext character back-calculates the key — fatal for flags with fixed headers."
Prediction: what if the key weren’t one byte but two alternating bytes,
0x5A 0x3C? From the first character you could only recover half the key. The clue then is "the same byte pattern repeats regularly" — estimating key length is Crypto’s next-level assignment.
3-4. Challenge 3: Toy RSA — Small n Means Game Over
The third challenge hands you the parameters whole: n=3233, e=17, c=2790. RSA’s public key and a ciphertext. Find the "odd thing" here — n is way too small. Four digits can be split by hand.
n, e, c = 3233, 17, 2790
# factor n — small, so just try dividing everything
for i in range(2, n):
if n % i == 0:
p = i
break
q = n // p
print(f"{n} = {p} x {q}")
# recover the private key d, then decrypt
phi = (p - 1) * (q - 1)
d = pow(e, -1, phi) # modular inverse of e
m = pow(c, d, n) # decryption = this one line
print(f"d = {d}, m = {m} → '{chr(m)}'")
3233 = 53 x 61
d = 2753, m = 65 → 'A'
(Measured 2026-09-09. 53 and 61 may come out in swapped order, but the product is the same.)
How to read the output: the moment you split n, you can compute (p-1)(q-1), get e’s inverse d, and decrypt with pow(c, d, n). The plaintext was the single letter A (65). Real challenges use slightly larger n, but the principle is the same — and the cheat code then is factordb.com: if someone already factored your n, it’s an instant answer (Screen example):
# Screen example — the shape of a factordb.com search result for 3233
3233 = 53 * 61 (FF, fully factored)
Why it breaks, in one line: "RSA’s safety stands on ‘n can’t be split’ — if n is small or already factored, the private key is recovered for free."
3-5. What It Looks Like on the Real Platform — Screen Example
Applying the flow you practiced locally to Dreamhack’s Crypto category looks like this (Screen example — this environment did not connect):
# Screen example — the typical layout of a platform challenge page
[Crypto] baby-rsa Difficulty: 1
Attachments: chal.py, output.txt ← they give you source and parameters
Flag submission: DH{ ... }
Platform challenges, too, are ultimately extensions of today’s three types. Read the source → find the "odd thing" (small n, a reused key, encoding only) → solve with Python. You’re not learning something new — you’re taking today’s eye with you as-is.
4. Missions & Exercises
Mission — Build and Solve Your Own Crypto Challenge
Transform each of today’s three types once, and experience both "author → solver":
- Encoding challenge: build a challenge where your own flag (
DH{...}format) is Base64-encoded, then hex-encoded on top — two layers - XOR challenge: build a challenge encrypted with a different key byte (e.g.,
0x42), then solve it by back-calculating the key from theDH{header using only the ciphertext - RSA challenge: pick different primes p, q (each under 100), make n, e, c, then split n on sight and decrypt
- Attach one line of "why it breaks" to each challenge and organize them in your notes
Exercises
Exercise 1. Look at the string aGVsbG8=. What encoding would you guess, and on what grounds? What’s the decoding result?
Exercise 2. An XOR ciphertext’s first byte is 0x37 and the plaintext’s first character is D (0x44). What’s the key? And decrypt 0x37 0x2d 0x22 with that key.
Exercise 3. Here’s a toy RSA with n = 143, e = 7, c = 48. Factor n, compute d, and decrypt.
Exercise 4. rot13 is an encoding that "shifts the alphabet by 13." Decode uryyb frphevgl. And why does applying rot13 twice return the original string?
5. Model Answers & Completion Criteria
Mission Model Answer
An example of the transformed challenges (measured 2026-09-09 — the case with the XOR key changed to 0x42):
flag = "DH{m1ss10n_cl34r}"
key = 0x42
cipher = bytes(b ^ key for b in flag.encode())
print(cipher.hex())
# solution: cipher[0] ^ ord('D') = 0x42 → full decryption succeeds
For the RSA variant, just pick new primes like p=67, q=71. Keep both primes under 100 so an easy full division splits n. And keep the plaintext m as a number smaller than n (a single character) — if m exceeds n, information gets cut by the modular operation.
How to verify: ① did you solve all three challenges "looking only at the ciphertext (parameters)" — without looking at the generation code? ② did the XOR key back-calculation pass not just the first character but the second-character check too? ③ does each "why it breaks" describe the content of the hole (no key, key back-calculable, small n) rather than a technique name?
Exercise Answers
Answer 1. Guessed as Base64. The clue is the trailing = (padding). base64.b64decode("aGVsbG8=") → hello. Base64’s look — only letters, digits, +//, in blocks of 4 — is also a hint.
Answer 2. The key is 0x37 ^ 0x44 = 0x73. Decryption: 0x37^0x73=0x44(D), 0x2d^0x73=0x5e(^), 0x22^0x73=0x51(Q) → D^Q. Even when the plaintext isn’t a flag, the back-calculation procedure is the same.
Answer 3. 143 = 11 × 13. phi = 10 × 12 = 120, d = pow(7, -1, 120) = 103. Decryption: pow(48, 103, 143) = 80 → the character P. (Check it yourself in Python.)
Answer 4. codecs.decode("uryyb frphevgl", "rot13") → hello security. The alphabet has 26 letters, so shifting 13 twice moves 26 — right back to the start. That’s why rot13 is "its own inverse," with encryption and decryption being the same operation.
Completion Criteria Checklist
- [ ] I can explain the difference between encoding and encryption (presence of a key)
- [ ] I guess encodings from shape clues like
=padding and hex character sets - [ ] I solved the hex + Base64 two-layer challenge with
bytes.fromhexandbase64.b64decode - [ ] I decrypted by back-calculating the XOR key from the
DH{header - [ ] I factored a small-n RSA and decrypted with
pow(c, d, n) - [ ] I wrote down one line of "why it breaks" for each of the three challenges
- [ ] Mission: I built and solved my own challenges transforming all three types
6. Common Pitfalls & Fixes
Wall 1. bytes.fromhex() throws a ValueError
Symptom: ValueError: non-hexadecimal number found in fromhex() arg at position 4 (measured 2026-09-09).
Cause: the string has non-0-9a-f characters mixed in (spaces, a 0x prefix, typos). The position tells you the culprit.
Fix: strip spaces (s.replace(" ", "")), remove the 0x prefix, and check the length is even. Still failing? Then that string isn’t hex — move to the next candidate in the 2-2 table.
Wall 2. Base64 decoding throws Incorrect padding
Symptom: binascii.Error: Incorrect padding (measured 2026-09-09).
Cause: a Base64 string’s length must be a multiple of 4, but the trailing = got cut during copying.
Fix: copy the original again and preserve the =. In a pinch, append as many = as needed — s + "=" * (-len(s) % 4).
Wall 3. pow(e, -1, phi) throws ValueError: base is not invertible
Symptom: ValueError: base is not invertible for the given modulus (measured 2026-09-09).
Cause: e and phi are not coprime. An inverse exists only when they’re coprime. You hit this when picking e arbitrarily in the mission.
Fix: pick e as a prime (3, 5, 7, 17, etc.) and check phi isn’t a multiple of it. This is also why real RSA uses primes like e=65537.
Wall 4. Only part of the XOR decryption is garbled
Symptom: the front reads fine but the back is broken.
Cause: it’s a challenge where the key isn’t one byte but several (key length > 1). The key derived from the first character is only part of the key.
Fix: if the known plaintext is the three characters DH{, you can back-calculate up to three bytes. If that’s still not enough, the next step is estimating the key’s repetition period — beyond introductory scope, so for today, confirm you’re dealing with "one-byte-key challenges" only.
Wall 5. Using ^ for exponentiation in Python
Symptom: 2 ^ 3 comes out as 1, not 8.
Cause: in Python, ^ is XOR; exponentiation is ** or pow().
Fix: use ^ only when you need XOR. RSA math goes through pow(m, e, n) — the three-argument pow is modular exponentiation.
7. Summary
Today’s Concepts
| Concept | One-line explanation |
|---|---|
| Encoding | A keyless notation change — Base64, hex, rot13. Anyone who knows the rule can invert it |
| Known-plaintext attack | The technique of back-calculating a key from part of the plaintext (the flag header) |
| XOR’s self-inversion | XOR twice with the same key and you’re back — that’s what makes key back-calculation possible |
| RSA | A public-key cipher safe as long as n=p×q can’t be split — once split, it’s over |
| Modular inverse | pow(e, -1, phi) — the one line of Python that computes the private key d |
| Shape clue | = means Base64, 0-9a-f means hex — a string’s appearance is your first hypothesis |
Today’s Commands
| Command | What it does |
|---|---|
bytes.fromhex(s) |
Hex string → bytes |
base64.b64decode(s) |
Base64 decode |
codecs.decode(s, "rot13") |
Undo rot13 |
bytes(b ^ key for b in cipher) |
One-byte XOR decryption |
pow(e, -1, phi) |
Compute the modular inverse (private key d) |
pow(c, d, n) |
RSA decryption |
An Instinct More Important Than Commands
A Crypto challenge’s starting point is not calculation but observation. "What does this string’s shape tell me?" "Which of these parameters is odd?" Today’s three challenges were all games of finding the author’s compromise — patching with encoding, a one-byte key, a small n.
Real-world cryptography is a different beast from today’s toys. But "the eye that finds weak spots in implementations and numbers" stays the same. When you pick your main field in Step 181, if you felt today that "the math puzzles were the most fun," that’s an important signal.
Once every box is checked, Step 179 is complete. Click the checkbox in the sidebar to save your progress.