InfoSec 3001 — Information Security from the Offensive Perspective

Step 50. Exploring Encoding — The Bridge Between Letters and Numbers

Step 50. Exploring Encoding — The Bridge Between Letters and Numbers

Level 1 — Programming and the Computer’s Insides | Difficulty ★★☆☆☆ | Estimated time: 3 hours

Prerequisites: Steps 41–49 complete. You know basic Python syntax and how to handle strings.

  • What you need: a computer with Python installed. Today is a day with more experiments than code — open interactive mode (the >>> you get after typing python) before you start.
  • Caution: today’s practice is 100% safe. There are no experiments that change or delete files — everything is a conversion experiment inside memory.

Computers don’t know letters. The only thing they know is numbers. So the ‘A’ and the ‘가’ you see on screen are all numbers inside. Then what number is ‘A’? Who decided? That question is today’s starting point. Once you know that letters are numbers, every conversion that turns numbers back into letters starts to become visible — strings twisted into Base64, dumps written in hexadecimal, Korean broken by a strange code page. All of them are "conversion-table problems between numbers and letters."

1. Learning Objectives

By the end of this chapter, you will be able to:

  • Verify with ord/chr experiments the principle that characters are stored as numbers inside a computer
  • Explain the relationship between ASCII, Unicode, UTF-8, Base64, and hexadecimal (hex)
  • Move between str and bytes with encode/decode
  • Convert "Hello" into a number list, hex, and Base64, and convert it back
  • Diagnose broken Korean as a "code page mismatch" and fix it by matching the rule

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category Details
Language/environment Python 3 interactive mode (>>>) + short scripts
Today’s functions ord() (letter→number), chr() (number→letter), encode()/decode(), bytes.hex()/bytes.fromhex(), base64.b64encode/b64decode
Concepts needed ASCII, Unicode, UTF-8, Base64, hexadecimal, the str vs bytes distinction
Today’s artifact viewer.py — a converter that shows one string in four different outfits

2-1. ASCII — A Table of 128 Numbers

In the early days of computers, the table America decided on — "let’s number the English letters" — is ASCII. Uppercase A is number 65, lowercase a is 97, digit 0 is 48. It’s a small table with 128 slots in total, and it’s still embedded in every corner of the computer world.

2-2. Unicode and UTF-8 — The Resident Registration of the World’s Characters

With 128 slots you can’t fit Korean or Chinese characters. So the giant table that numbers every character in the world is Unicode. ‘가’ is number 44032, for example.

And the rule that moves that number into actual bytes (the storage unit) is UTF-8. If Unicode is the resident registration number, UTF-8 is the format for writing that number on the envelope. This is why we write encoding="utf-8" every time we open a file.

2-3. Base64 — Translation into 64 Letters

Email and the web are "channels through which only letters pass safely." Binary data like photos can’t pass through this channel as-is. So the method that translates all data into 64 safe letters (A~Z, a~z, 0~9, +, /) is Base64.

Here’s today’s most important sentence: Base64 is not encryption. It’s just a "conversion" whose translation table is public and that anyone can reverse. If you see SGVsbG8= written somewhere, no secret is hidden — it’s just "Hello" written in different letters.

2-4. Hexadecimal (hex) — Shorthand Writing for Bytes

One byte is a number from 0 to 255. The convention is to write it as two hexadecimal digits (00~ff) — like "48 65 6c 6c 6f". Hexadecimal is a number system that attaches a~f (10~15) after 0~9, and memory dumps, hash values, and color codes (#ff5500) all use this notation.

3. Follow Along

3-1. ord and chr — Peeking at the Number Table

Input (interactive mode, one at a time at >>>)

ord('A')
ord('a')
ord('0')
chr(65)
chr(97)

Output (measured 2026-09-09):

65
97
48
'A'
'a'

How to read it: ord(letter) asks "what’s this letter’s number?", and chr(number) asks "what’s this number’s letter?" They’re each other’s reverse. Also note that the uppercase and lowercase numbers differ by 32 (65 vs 97).

3-2. Korean Numbers — Checking Unicode

Input

ord('가')
ord('힣')
hex(ord('가'))

Output (measured 2026-09-09):

44032
55203
'0xac00'

How to read it: ‘가’ is Unicode number 44032. hex() converts a decimal number into hexadecimal notation, and 0x is the signpost saying "what follows is hexadecimal." You may have seen notation like "U+AC00" in documents — that’s this number.

Predict: what letter is chr(44033)? It’s the number right after ‘가’… (measured answer: ‘각’) And chr(ord('나') + 1)? (measured answer: ‘낙’) Predict, then check.

3-3. encode and decode — Letters to Bytes, Bytes to Letters

Input

"Hello".encode("utf-8")
"가".encode("utf-8")
b"Hello".decode("utf-8")

Output (measured 2026-09-09):

b'Hello'
b'\xea\xb0\x80'
'Hello'

How to read it: encode() turns letters (str) into bytes (bytes), and decode() turns bytes back into letters. The b in front of the result is a marker saying "this is bytes." Look at ‘가’ becoming three bytes (ea b0 80) — UTF-8 is a rule that writes English in 1 byte and Korean in 3 bytes. \xea is the notation for "one byte with the hexadecimal value ea."

Why: the str vs bytes distinction is a wall you’ll keep hitting later in network communication, file handling, and encryption. Meet it head-on today, once.

3-4. Base64 — Translate and Reverse

Input (encode_decode.py)

import base64

original = "Hello"
encoded = base64.b64encode(original.encode("utf-8"))
print(encoded)

decoded = base64.b64decode(encoded).decode("utf-8")
print(decoded)

Output (measured 2026-09-09):

b'SGVsbG8='
Hello

How to read it: the flow is "letters → bytes (encode) → Base64 letters (b64encode)," and reversing goes in the opposite order. That strange string SGVsbG8= was actually "Hello." The = at the end is Base64’s characteristic tail (for padding), and seeing this tail is a clue for suspecting Base64.

Predict: what is aGVsbG8gd29ybGQ=? Run b64decode yourself and check. (measured answer 2026-09-09: hello world) "Reading" Base64 means having the machine reverse it like this.

3-5. hex Conversion — Practicing Hexadecimal Writing

Input

data = "Hello".encode("utf-8")
print(data.hex())
print(bytes.fromhex("48656c6c6f").decode("utf-8"))

Output (measured 2026-09-09):

48656c6c6f
Hello

How to read it: .hex() turns bytes into a hexadecimal string, and bytes.fromhex() is the reverse. In "48 65 6c 6c 6f", 48 is ‘H’, 65 is ‘e’… it’s the ASCII numbers you saw in 3-1, written in hexadecimal. The two notations for the same numbers — decimal (72, 101) and hex (48, 65) — are now connected.

3-6. The All-in-One Converter — Seeing Four Forms

Input (viewer.py)

import base64

text = input("Enter a string: ")
data = text.encode("utf-8")

print("Numbers by character:", [ord(c) for c in text])
print("hex:", data.hex())
print("base64:", base64.b64encode(data).decode())

Run (measured 2026-09-09, Hello entered):

Enter a string: Hello
Numbers by character: [72, 101, 108, 108, 111]
hex: 48656c6c6f
base64: SGVsbG8=

How to read it: four faces of one string. Letters, a number list, hex writing, Base64 — all different notations of the same content.

3-7. When Code Pages Mismatch — The cp949 Experiment

Korean Windows has one more old code page living in it besides UTF-8: cp949. When these two mix, characters break. Let’s break them ourselves, then fix them.

Input (interactive mode)

data = "안녕".encode("cp949")
print(data)
print(data.decode("cp949"))

Output (measured 2026-09-09):

b'\xbe\xc8\xb3\xe7'
안녕

How to read it: translated with cp949, ‘안녕’ becomes four bytes (be c8 b3 e7). For reference, in UTF-8 it’s six bytes — the same letters take a different number of bytes depending on the code page (measured 2026-09-09). Reversing with the same code page reads fine.

Now force-read these bytes as UTF-8: data.decode("utf-8") — measured (2026-09-09):

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte

Let’s also test the opposite direction: reading Korean written in UTF-8 with cp949. Some letters error out (e.g., "안녕" gives UnicodeDecodeError: 'cp949' codec can't decode byte 0xec ... — measured 2026-09-09), and some letters read as completely wrong characters without an error:

u = "보안".encode("utf-8")
print(u.decode("cp949"))

蹂댁븞

(Measured 2026-09-09. "보안" became "蹂댁븞" — you’ve probably seen broken characters like this.)

"Broken = the code page used when writing differs from the one used when reading" — this is today’s formula. When code pages mismatch, either an error occurs or, with bad luck, it silently breaks. One of the two.

Why: when you open files scraped from the web or made by old programs, you’ll meet broken characters. This is a vaccine that lets you think "I just need to match the code page" instead of panicking.

4. Missions & Exercises

Mission — Complete the Conversion Playground and Decode Three Strings

  1. Extend viewer.py so you can choose the conversion direction. Menu: 1. letters→numbers/hex/Base64, 2. Base64→letters, 3. hex→letters
  2. Add Step 46’s exception handling so menus 2 and 3 don’t die on bad input (strange Base64, odd-length hex, etc.)
  3. Decode the three strings below yourself and write the results in your notes
    • Base64: c2VjdXJpdHk=
    • hex: 637466
    • number list: [83, 116, 117, 100, 121]
  4. Write the commands or code used for each decoding in your README

Exercises

Q1. Explain the relationship between ASCII, Unicode, and UTF-8 using the distinction of "number table vs envelope format."

Q2. Why is Base64 not encryption? State the criterion that separates conversion from encryption in one sentence.

Q3. data.decode("utf-8") raised UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe .... What’s the situation, and how do you solve it?

Q4. Why does "48656c6c6f".decode() error, and what’s the correct two-leg order?

5. Model Answers & Completion Criteria

Mission Model Answer

The skeleton of the conversion playground:

import base64

while True:
    print("1. Text->numbers/hex/Base64  2. Base64->text  3. hex->text  0. Quit")
    menu = input("Choice: ").strip()
    if menu == "0":
        break
    elif menu == "1":
        text = input("String: ")
        data = text.encode("utf-8")
        print("Numbers:", [ord(c) for c in text])
        print("hex:", data.hex())
        print("base64:", base64.b64encode(data).decode())
    elif menu == "2":
        s = input("Base64 string: ").strip()
        try:
            print("Decoded:", base64.b64decode(s).decode("utf-8"))
        except Exception:
            print("Not Base64-shaped, or data that isn't UTF-8 text.")
    elif menu == "3":
        s = input("hex string: ").strip()
        try:
            print("Decoded:", bytes.fromhex(s).decode("utf-8"))
        except ValueError:
            print("Not hex-shaped (0~9, a~f, even length).")
        except UnicodeDecodeError:
            print("The bytes are fine, but they're not UTF-8 text.")
    else:
        print("Please choose 0~3.")

Decoding the three strings (measured 2026-09-09):

c2VjdXJpdHk=  -> security
637466        -> ctf
[83, 116, 117, 100, 121] -> Study

Commands used for decoding: base64.b64decode("c2VjdXJpdHk=").decode("utf-8"), bytes.fromhex("637466").decode("utf-8"), "".join(chr(n) for n in [83, 116, 117, 100, 121]).

How to verify: ① Does entering something like "!!!" into menu 2 show guidance instead of dying? ② Does entering an odd length ("486") into menu 3 show the ValueError guidance? ③ Are the three answers (security / ctf / Study) written in your notes? If all three are "yes," it’s complete.

Exercise Solutions

Q1 solution. ASCII is a 128-slot number table centered on English; Unicode is a giant number table numbering every character in the world; UTF-8 is the rule (envelope format) for writing those Unicode numbers as actual bytes. If Unicode is the number, UTF-8 is the storage format of that number.

Q2 solution. Because the translation table is public and anyone can reverse it without a key. The criterion: "if it can be reversed without a key, it’s a conversion; if it can only be reversed with a key, it’s encryption." A password written in Base64 is not a password — it’s just a password written strangely.

Q3 solution. It’s a situation where those bytes were not written under UTF-8 rules — they were written with a different code page (cp949, etc.), or you skipped the Base64 decoding. The fix is to trace back "what rule made these bytes" and match the rule. Output from Korean-language Windows programs is often cp949 (as in the 3-7 measurement, the cp949 byte 0xbe raises exactly this error when read as UTF-8).

Q4 solution. Because "48656c6c6f" is text (str), not bytes — decode is something you use on bytes (measured 2026-09-09: AttributeError: 'str' object has no attribute 'decode'). The correct order: hex string → bytes.fromhex() → bytes → .decode() → text. You cross in two legs.

Completion Criteria Checklist

  • [ ] I can move between letters and numbers with ord and chr
  • [ ] I can move between str and bytes with encode/decode
  • [ ] I can explain that Base64 is a conversion (not encryption)
  • [ ] I can read hex notation and reverse it with bytes.fromhex
  • [ ] I verified by experiment that a cp949 vs UTF-8 mismatch is the cause of broken text
  • [ ] When I meet a strange string, I can guess from its tail and character composition
  • [ ] Mission: I completed the conversion playground and decoded the three strings

6. Common Pitfalls & Fixes

Wall 1. UnicodeDecodeError occurs

Symptom (measured 2026-09-09):

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte

Cause: those bytes were not written under UTF-8 rules. They were written with a different code page (cp949, etc.), or you skipped the Base64 decoding.
Fix: trace back "what rule made these bytes." Output from Korean-language Windows programs is often cp949. Match the rule, and it opens.

Wall 2. Mixing str and bytes

Symptom (measured 2026-09-09):

TypeError: can only concatenate str (not "bytes") to str

Cause: letters (str) and bytes (bytes) are different species. Mixing them without encode/decode is an error.
Fix: make a habit of checking which species a variable currently is with type(). Think of it as "bytes at network/file boundaries, str inside my program," and encode/decode only at the boundaries.

Wall 3. Mistaking Base64 for encryption

Symptom: you find important information "encrypted" in Base64 and think you’ve found something big.
Cause: confusing conversion with encryption. Base64 is a public translation anyone can reverse without a key.
Fix: remember the criterion — "if it can be reversed without a key, it’s a conversion; if it can only be reversed with a key, it’s encryption."

Wall 4. Trying to decode a hex string

Symptom (measured 2026-09-09):

AttributeError: 'str' object has no attribute 'decode'

Cause: "48656c6c6f" is text (str), not bytes. decode is something you use on bytes.
Fix: remember the order. hex string → bytes.fromhex() → bytes → .decode() → text. You cross in two legs.

Wall 5. Panicking at broken characters

Symptom: you open a file and see broken characters like 蹂댁븞.
Cause: the code page used when writing differs from the one used when reading — reading "보안" written in UTF-8 with cp949 produces exactly this shape (section 3-7, measured 2026-09-09).
Fix: don’t panic; recall "I just need to match the code page." Guess the code page of the program (or system) that made the file and re-read with something like encoding="cp949".

7. Summary

Today’s Concepts

Concept One-line description
ASCII A 128-slot number table centered on English (A=65, a=97, 0=48)
Unicode The number table of every character in the world (‘가’=44032=U+AC00)
UTF-8 The rule for writing Unicode numbers as bytes (English 1 byte, Korean 3 bytes)
cp949 Korean Windows’ old Korean code page — breaks when mixed with UTF-8
Base64 Translates all data into 64 safe letters — a conversion, not encryption
hex Shorthand writing of a byte as two hexadecimal digits
str / bytes Letters and bytes — crossed only via encode/decode

Today’s Functions

Function What it does
ord(letter) / chr(number) letter↔number
hex(number) decimal → hexadecimal notation (0x marker)
string.encode("utf-8") letters → bytes
bytes.decode("utf-8") bytes → letters
base64.b64encode / b64decode Base64 translate / reverse
bytes.hex() / bytes.fromhex() hex writing / reversing

The Instinct That Matters More Than Commands

Let’s record the procedure for when you meet a strange string. (a) Guess from the tail and character composition — an = tail and a 64-letter composition means Base64; only 0~9 and a~f with even length means hex. (b) Try reversing with the guessed rule. (c) If that fails, try a different rule — it’s common to decode Base64 and find more Base64. (d) If bytes come out, match the code page and read them as letters. With these four steps, most converted strings open.

The connection to security: attackers hide commands in Base64 to evade detection, and defenders spot Base64 shapes in logs and reverse them. Seeing something like powershell -enc SGVsbG8= and recognizing "ah, what’s after -enc is Base64" is the eye today’s skill builds. Finally, remember the journey of the single letter ‘가’ — remembered as Unicode number 44032, becoming three bytes (ea b0 80) in UTF-8, becoming 6rCA in Base64 and eab080 in hex (all measured 2026-09-09). They’re all different outfits of the same ‘가’. The habit of asking "what outfit is this data wearing right now" — that solves half of encoding problems.

Once every box is checked, Step 50 is complete. Click the checkbox in the sidebar to save your progress.

← 이전 스텝
다음 스텝 →