InfoSec 3001 — Information Security from the Offensive Perspective

Step 35. TCP Deep Dive — The Art of the Handshake

Step 35. TCP Deep Dive — The Art of the Handshake

Level 0 — Understanding Computer Operation and Structure | Difficulty ★★★☆☆ | Estimated time: 3 hours

Prerequisites: Step 34 (Ports and Services) complete. We use Windows PowerShell plus paper and pen. The best practice tool for this chapter is paper and pen.

  • What you need: a Windows PC, PowerShell, paper and pen.
  • Caution: today’s practice is only lookups and drawing on paper, so it’s 100% safe.

Think about making a phone call. Nobody blurts out business the moment they pick up. "Hello?" — "Yes, hello." — "Ah, hi there." Only after this brief greeting does the conversation begin. And when it ends, "I’ll hang up now" — "Yes, take care." Nobody hangs up abruptly.

TCP (Transmission Control Protocol), the internet’s main delivery courier, is designed to follow this etiquette exactly. Before sending data, it always establishes a "connection," and when finishing, it performs a prescribed goodbye. Today you learn the exact order of those greetings.

It may look like boring memorization, but these three lines of greeting do enormous work in the security world. Port scans are made of this greeting, DDoS attacks abuse it, and firewalls read these greetings to make decisions. A quiet chapter — but one that takes you very far.

1. Learning Objectives

By the end of this chapter, you will be able to:

  • Explain what it means that TCP is "connection-oriented"
  • Draw the 3-way handshake (SYN → SYN-ACK → ACK) on blank paper
  • Explain the roles of the major flags (SYN, ACK, FIN, RST)
  • Read a table of connection states (Established, TimeWait, SynSent…) and interpret what’s happening
  • Explain how port scanning exploits this handshake

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

Category Details
Language/environment PowerShell 5.1 (lookups only) + paper and pen
Today’s commands netstat -an | findstr ESTABLISHED (live connections), Get-NetTCPConnection | Group-Object State (distribution by state), Test-NetConnection (checking handshake success)
Concepts needed Connection-oriented, 3-way handshake, flags (SYN/ACK/FIN/RST), sequence numbers, the 4-way connection teardown

2-1. Connection-Oriented — "Lay the Line First, Then Talk"

Communication methods divide broadly into two. TCP, the connection-oriented method, lays the line before talking (establishes a connection), keeps checking during the conversation whether things arrived, and reels the line back in when done. On the other side is the connectionless method (UDP), which just throws without any such procedure.

The reward of connection orientation is reliability. If a letter goes missing, TCP resends it; if the order gets scrambled, it fixes it. Web, SSH, mail — every communication where "not a single character may be lost" chose TCP.

2-2. The 3-Way Handshake — Three Greetings

This is the exact order for establishing a connection. Let’s look at it as a picture of the client (the caller) and the server (the receiver).

Client                        Server
    |------ SYN -------->|   "I'd like to connect" (first greeting)
    |<---- SYN-ACK ------|   "Sure, I'm ready too" (reply + confirmation)
    |------ ACK -------->|   "Confirmed. Let's begin" (closing greeting)
    |                     |
    |=== now data ======>|   the real conversation begins

Why three times instead of two? For each side to confirm that "my words reach the other side, and the other side’s words reach me," you need at least three. SYN-ACK is merged into one, but in terms of roles it carries both the server’s ACK ("I heard you") and the server’s SYN ("let’s start on my end too").

2-3. Flags — Indicator Lights Attached to Letters

Every TCP letter (segment) carries small flags that announce its purpose — flags. You only need to memorize four.

Flag Meaning Phone analogy
SYN Request to start a connection "Hello?" (dialing)
ACK Confirmation of receipt "Yes, I’m listening"
FIN Normal termination request "I’ll hang up now"
RST Forced reset Slamming the phone down

ACK is not a special event. After a connection is established, nearly every data letter carries an ACK as a confirmation marker — "I got your letter." TCP communication after the handshake is a continuous stream of "data + confirmation."

2-4. Termination Takes Four Greetings

When finishing, the two sides exchange FIN and wrap up with four greetings.

Client                        Server
    |------ FIN -------->|   "I'll hang up now"
    |<------ ACK --------|   "Got it"
    |<------ FIN --------|   "I'm done too, let's hang up"
    |------ ACK -------->|   "Yes, thanks for your time"

Why is termination (4) one more than initiation (3)? At initiation, the server’s "I heard you" and "let’s start too" are merged into one letter (SYN-ACK), but at termination they don’t merge. Even if one side says "let’s hang up" first, the other side may still have data left to send, so it sends ACK first and FIN later. Each side must finish saying everything it has to say before parting.

2-5. Sequence Numbers — The Serial Numbers on Letters

Every TCP letter is stamped with a sequence number. The receiver uses these numbers to assemble the order and can say, "After number 3 came number 5 — please send number 4 again." This numbering system is the foundation of ordering guarantees and retransmission.

One security story: early TCP start numbers were predictable. So an attacker who "guessed the next number" could slip into someone else’s connection (TCP session hijacking). Modern operating systems pick start numbers randomly to prevent this. It’s a prime example of "randomness" becoming a security ingredient.

2-6. Firewalls Read the Handshake

A quick note that the greetings you learned today are also a language for defensive equipment. Modern firewalls don’t judge by a single letter. They remember "which conversation this letter is part of" — this is called stateful inspection.

An ACK flying in from outside with no warning is discarded as "a strange letter sending a confirmation without any start," while an ACK belonging to a conversation that began with a SYN from inside is let through. In other words, the firewall knows the exact picture you drew today and suspects any letter that deviates from it. The contest between those who know the rules and those who aim for the exceptions plays out in this single greeting.

3. Follow Along

3-1. Drawing the Handshake

On paper, raise two pillars (client, server) and draw the handshake with three arrows. Label each arrow SYN, SYN-ACK, ACK.

Repeat three times until you can draw it without looking at the book. Knowing in your head and drawing with your hand are different things. When the picture comes out of your hand, you’ve understood it.

3-2. Seeing the Distribution of Connection States Right Now

The greetings you learned today are flying around your computer at this very moment. Let’s count the traces by state.

Get-NetTCPConnection | Group-Object State | Sort-Object Count -Descending

Name        Count
----        -----
Established    76
Bound          70
TimeWait       60
Listen         56
CloseWait       6
SynSent         1

(Measured 2026-09-09. The numbers change moment to moment.)

How to read the output — each state is a single frame of the greetings you learned today:

  • Listen 56 — 56 of the "doors waiting for guests" from Step 34.
  • Established 76 — 76 "on a call" connections that passed the three-handshake.
  • TimeWait 60 — connections keeping their seat briefly after finishing the goodbye (the 4-way FIN exchange). It’s a lingering grace period for handling late-arriving letters. A full 60 connections just said goodbye.
  • CloseWait 6 — the other side sent a FIN ("I’ll hang up"), but my program hasn’t yet made its closing greeting.
  • SynSent 1 — a connection caught right at this moment waiting for the other side’s SYN-ACK after sending its SYN. A single frame mid-handshake.

Even at the instant you run this command, the numbers in this table are alive and moving. Run it a few times and watch the numbers rise and fall — the handshakes and goodbyes you couldn’t see start to appear as numbers.

3-3. Seeing Live Connections

netstat -an | findstr ESTABLISHED

  TCP    127.0.0.1:3452        127.0.0.1:64743        ESTABLISHED
  TCP    127.0.0.1:3452        127.0.0.1:64744        ESTABLISHED
  TCP    127.0.0.1:10086       127.0.0.1:64728        ESTABLISHED

(Measured 2026-09-09. Your list will likely show many 443 (HTTPS) connections opened by your browser.)

How to read the output: the left side is my address+port, the right side is the other’s address+port. The measured lines are conversations among programs inside my computer (127.0.0.1 to 127.0.0.1). Run it with a website open in your browser and you’ll see lines ESTABLISHED with …:443 — all connections that safely completed the three greetings.

Making it onto this list means "SYN → SYN-ACK → ACK succeeded." From the result alone, you can reconstruct the handshake behind it.

3-4. Tracing Handshake Success and Failure from Results

Let’s look at last chapter’s Test-NetConnection again with today’s eyes.

Test-NetConnection 172.30.1.254 -Port 80

TcpTestSucceeded : True

(Measured 2026-09-09, output excerpt.)

What happened behind that single True line: my computer sent a SYN → the gateway replied with SYN-ACK → my computer finished with ACK. True because the three lines crossed safely.

The closed door (127.0.0.1:65000, measured 2026-09-09) was False. What happened behind it: a SYN was sent, but → an RST came back saying "no such room." Even among "responses," SYN-ACK (welcome) and RST (rejection) are exact opposites.

3-5. The Principle of Scanning — Three Kinds of Answers

Now let’s organize how this handshake becomes the raw material of scanning tools (like nmap). When a scanner sends a SYN to a port:

Answer that comes back Meaning Scanner’s verdict
SYN-ACK "Sure" — a service is waiting open
RST "No such room" — slammed shut closed
(silence) A firewall quietly discarded the letter filtered

How to read it: a scanning tool is not magic. It’s just a machine that uses TCP’s etiquette rules to "knock on doors and classify the answers." And in section 3-4, you performed this classification by hand — both kinds.

Predict: what state does a server end up in if an attacker sends only SYNs and never sends the final ACK? (Answer: the server sends SYN-ACK and is left holding "half-open" connections, endlessly waiting for the final ACK. When tens of thousands of such half connections pile up, the server’s memory is paralyzed — the seed of a SYN flood attack. For now, just take away the intuition that "forcing a greeting to break off puts the other side in trouble.")

3-6. Compressing It into an Analogy

Compress today’s lesson into one paragraph of phone analogy and say it out loud.

"TCP is like a telephone. When calling, it starts with hello (SYN), then yes hello (SYN-ACK), then ah yes (ACK). During the call, it confirms it’s listening with uh-huh (ACK) and stamps numbers so the order doesn’t tangle (sequence numbers). When ending, it finishes politely with I’ll hang up (FIN), yes (ACK), me too (FIN), yes (ACK). If the number doesn’t exist, a slammed receiver (RST) comes back."

If this paragraph flows out smoothly, the core of this chapter is done.

4. Missions & Exercises

Mission — A Handshake Observation Journal of My Computer

  1. Visit two websites in a row in your browser
  2. Right after visiting, run netstat -an | findstr ESTABLISHED and find the port-443 connection lines
  3. Record the state distribution with Get-NetTCPConnection | Group-Object State
  4. Run the same command again 2–3 minutes later, and write down what grew and what shrank (especially the change in TimeWait)
  5. Organize your observations into tcp-diary.txt, and on the last line fill in "Today on my computer, the number of connections that had sent a SYN and were waiting for an answer (SynSent) was __"

Exercises

Q1. Close the book and draw the 3-way handshake on blank paper. Arrow directions and flag names must all be correct.

Q2. Write the three possible answers that can come back when a port scanner sends a SYN, and the scanner’s verdict for each.

Q3. Why does connection termination take four exchanges, one more than initiation (three)? Explain via why SYN-ACK merges but the terminating ACK and FIN don’t.

Q4. You received a report: "I can’t access the website." Depending on which line of the handshake it stopped at, how do the candidate causes differ? Answer in three branches.

5. Model Answers & Completion Criteria

Mission Model Answer

Example observation journal from the measured computer (2026-09-09):

[ESTABLISHED right after visiting]
127.0.0.1:3452 ↔ 127.0.0.1:64743   ESTABLISHED
(if the browser was open)  myaddress:5xxxx ↔ externalserver:443  ESTABLISHED

[State distribution — 1st run]
Established 76 / Bound 70 / TimeWait 60 / Listen 56 / CloseWait 6 / SynSent 1

[2 minutes later — 2nd run]
TimeWait shrank and Established grew, or vice versa — as you browse, new connections appear
and finished connections fade away through TimeWait

[Sample last line]
"Today on my computer, the number of connections waiting for an answer after sending a SYN (SynSent) was 1"

How to verify: ① Did you find a port-443 ESTABLISHED line (best seen right after a browser visit — a little later it says goodbye and disappears)? ② Was the state distribution recorded at two points in time? ③ Is there an interpretation attached — "TimeWait is the lingering grace of a connection that has finished closing"? The numbers themselves differ every time — you’re done when you can interpret the state names, not the numbers.

Exercise Solutions

Q1 solution. Client → server: SYN; server → client: SYN-ACK; client → server: ACK. Common errors: the direction of the second arrow (the server answers), writing SYN-ACK as "ACK-SYN" (there’s no odd order where the answer comes first), and omitting the third one (the connection is complete only when the third ACK arrives).

Q2 solution. ① SYN-ACK → open — the service replied according to etiquette. ② RST → closed — a rejection meaning "no such room." ③ Silence → filtered — suspected that a firewall quietly discarded the letter.

Q3 solution. At initiation, the server’s "I heard you (ACK)" and "let’s start too (SYN)" are decided at the same moment, so they merge into one letter (SYN-ACK). But at termination, "I heard you (ACK)" can be sent immediately, while "I’m done too (FIN)" is possible only after all remaining data has been sent. Because of that time gap, termination splits into two letters, and since both sides each finish their part, it totals four.

Q4 solution. ① The SYN never left → a problem on my side (connection/address/gateway layer). ② The SYN went out but no answer came back → a mid-path or far-side problem (including firewall silence). ③ An RST came back → we reached the other server, but the service on that port is off. The same "doesn’t work" is three different illnesses, and the prescriptions differ for all three.

Completion Criteria Checklist

  • [ ] I can draw the 3-way handshake on blank paper
  • [ ] I can draw the termination procedure (four FIN exchanges)
  • [ ] I can explain the roles of SYN, ACK, FIN, RST
  • [ ] I can interpret the meaning of Established, TimeWait, and SynSent in a state table
  • [ ] I can explain that port scanning is the classification of handshake answers
  • [ ] I can state the role of sequence numbers (ordering and retransmission) in one sentence
  • [ ] Mission: I completed the tcp-diary.txt observation journal

6. Common Pitfalls & Fixes

Wall 1. "I keep mixing up the SYN, SYN-ACK, ACK order."

Symptom: you mix up what the second one was or what the server sends.

Cause: because you’re trying to memorize it as letters.

Fix: go with the phone analogy. "Hello (dialing) → yes hello (answering) → ah yes (confirming)." Analogies are unforgettable. Drawing the picture three times also works well.

Wall 2. "There are too many unfamiliar words in the state table."

Symptom: you get stuck on state names like Bound, CloseWait, TimeWait.

Cause: every state is just one frame of the greetings you learned today.

Fix: read just three with confidence — Listen (a waiting door), Established (on a call), TimeWait (the lingering grace after goodbye). Classify the rest as "a middle frame of the handshake or the goodbye." As the measurement in section 3-2 caught one SynSent, middle states are momentary, so small counts are normal.

Wall 3. "What on earth is a packet with only an ACK?"

Symptom: while reading explanations, you get stuck on the phrase "a packet with only the ACK flag set."

Cause: after a connection is established, nearly every data letter carries an ACK — because it always doubles as the confirmation marker "I got your letter."

Fix: just understand that TCP communication after the handshake is a continuous stream of "data + confirmation." ACK is not a special event but an everyday greeting.

Wall 4. "A red warning appeared in Test-NetConnection so I thought it failed."

Symptom: on a closed port, a warning line like WARNING: TCP connect to (… ) failed appears (measured 2026-09-09).

Cause: it’s not an error but a result report. "We tried to connect and it didn’t work" is part of this command’s normal output.

Fix: read the result table, not the warning line. TcpTestSucceeded : False means "closed," and behind it an RST (or silence) came back in response to the SYN.

Wall 5. "I can’t organize when RST shows up."

Symptom: RST appears in several situations, which is confusing.

Cause: RST is "the all-purpose reset for abnormal situations," so it has several stages to appear on.

Fix: remember just three. ① When a connection request arrives for a nonexistent room. ② When a letter arrives for an already-terminated connection. ③ When something doesn’t add up and the connection is forcibly cut. The common thread: a declaration that "this conversation cannot stand."

7. Summary

Today’s Concepts

Concept One-line description
TCP The connection-oriented courier — lays the line, confirms, and ends politely
3-way handshake SYN → SYN-ACK → ACK, the three greetings of connection
Flags SYN (start) / ACK (confirm) / FIN (normal end) / RST (forced reset)
4-way teardown FIN→ACK→FIN→ACK — each side finishes its own goodbye
Sequence number The serial number on letters — the basis of ordering and retransmission
TimeWait The lingering grace period of a connection that finished saying goodbye
Stateful firewall Equipment that remembers conversation context and suspects letters outside the etiquette

Today’s Commands

Command What it does
netstat -an | findstr ESTABLISHED See connections that passed the handshake
Get-NetTCPConnection | Group-Object State See the distribution by connection state
Test-NetConnection address -Port number Check handshake success (your own equipment only)

The Instinct That Matters More Than Commands

Someone who knows the 3-way handshake and someone who doesn’t see the same failure differently. Faced with "it won’t connect," the one who doesn’t know ends at "the internet is broken," but the one who knows asks at which line of the handshake it stopped. No answer, RST, an unfinished handshake — the same "doesn’t work" is three different illnesses. The ability to ask where a problem sits is real skill.

Today’s chapter had few commands and many pictures. Chapters like this last the longest. Commands may be forgotten, but the picture remains, and that picture finds the way in unfamiliar situations. Only someone who knows proper etiquette recognizes strange etiquette — the half handshake of a scan, the flooding SYNs of a flood, all are variations visible against today’s bright picture. Three greetings and four goodbyes — now they’re yours.

Once every box is checked, Step 35 is complete. Click the checkbox in the sidebar to save your progress.

← 이전 스텝
다음 스텝 →