Step 35. TCP Deep Dive — The Art of the Handshake

Level 0 — Understanding Computer Operation and Structure | Difficulty ★★★☆☆ | Estimated time: 3 hours

Prerequisites: Step 34 (Ports and Services) complete. We use Windows PowerShell plus paper and pen. The best practice tool for this unit is paper and pen.

• What you need: a Windows PC, PowerShell, paper and pen.
• Caution: this section’s practice is only lookups and drawing on paper, so it’s read-only in the stated lab.

Think about making a phone call. Nobody blurts out business the moment they pick up. “Hello?” — “Yes, hello.” — “Ah, hi there.” Only after this brief greeting does the conversation begin. And when it ends, “I’ll hang up now” — “Yes, take care.” Nobody hangs up abruptly.

TCP (Transmission Control Protocol), the internet’s main delivery courier, is designed to follow this etiquette exactly. Before sending data, it always establishes a “connection,” and when finishing, it performs a prescribed goodbye. Today you learn the exact order of those greetings.

It may look like boring memorization, but these three lines of greeting do enormous work in the security world. Port scans are made of this greeting, DDoS attacks abuse it, and firewalls read these greetings to make decisions. A quiet chapter — but one that takes you very far.

1. Learning Objectives

By the end of this unit, you will be able to:

• Explain what it means that TCP is “connection-oriented”
• Draw the 3-way handshake (SYN → SYN-ACK → ACK) on blank paper
• Explain the roles of the major flags (SYN, ACK, FIN, RST)
• Read a table of connection states (Established, TimeWait, SynSent…) and interpret what’s happening
• Explain how port scanning exploits this handshake

2. Background Knowledge — Today’s Tools and Concepts

This section’s Tools at a Glance

2-1. Connection-Oriented — “Lay the Line First, Then Talk”

Communication methods divide broadly into two. TCP, the connection-oriented method, lays the line before talking (establishes a connection), keeps checking during the conversation whether things arrived, and reels the line back in when done. On the other side is the connectionless method (UDP), which just throws without any such procedure.

The reward of connection orientation is reliability. If a letter goes missing, TCP resends it; if the order gets scrambled, it fixes it. Web, SSH, mail — every communication where “not a single character may be lost” chose TCP.

2-2. The 3-Way Handshake — Three Greetings

This is the exact order for establishing a connection. Let’s look at it as a picture of the client (the caller) and the server (the receiver).

Client                        Server
    |------ SYN -------->|   "I'd like to connect" (first greeting)
    |<---- SYN-ACK ------|   "Sure, I'm ready too" (reply + confirmation)
    |------ ACK -------->|   "Confirmed. Let's begin" (closing greeting)
    |                     |
    |=== now data ======>|   the real conversation begins

Why three times instead of two? For each side to confirm that “my words reach the other side, and the other side’s words reach me,” you need at least three. SYN-ACK is merged into one, but in terms of roles it carries both the server’s ACK (“I heard you”) and the server’s SYN (“let’s start on my end too”).

2-3. Flags — Indicator Lights Attached to Letters

Every TCP letter (segment) carries small flags that announce its purpose — flags. You only need to memorize four.

ACK is not a special event. After a connection is established, nearly every data letter carries an ACK as a confirmation marker — “I got your letter.” TCP communication after the handshake is a continuous stream of “data + confirmation.”

2-4. Termination Takes Four Greetings

During graceful closure, each sending direction is closed with FIN and acknowledged. Four separate segments are common, but FIN and ACK may be combined; resets and simultaneous closure differ.

Client                        Server
    |------ FIN -------->|   "I'll hang up now"
    |<------ ACK --------|   "Got it"
    |<------ FIN --------|   "I'm done too, let's hang up"
    |------ ACK -------->|   "Yes, thanks for your time"

Why is termination (4) one more than initiation (3)? At initiation, the server’s “I heard you” and “let’s start too” are merged into one letter (SYN-ACK), but at termination they don’t merge. Even if one side says “let’s hang up” first, the other side may still have data left to send, so it sends ACK first and FIN later. Each side must finish saying everything it has to say before parting.

2-5. Sequence Numbers — The Serial Numbers on Letters

TCP sequence numbers track byte positions in each direction, not a count of packets or letters. Acknowledgments identify the next expected byte, and retransmission repairs missing stream data. SYN and FIN each consume one sequence number.

One security story: early TCP start numbers were predictable. So an attacker who “guessed the next number” could slip into someone else’s connection (TCP session hijacking). Modern operating systems pick start numbers randomly to prevent this. It’s a prime example of “randomness” becoming a security ingredient.

2-6. Firewalls Read the Handshake

A quick note that the greetings you learned today are also a language for defensive equipment. Modern firewalls don’t judge by a single letter. They remember “which conversation this letter is part of” — this is called stateful inspection.

An ACK flying in from outside with no warning is discarded as “a strange letter sending a confirmation without any start,” while an ACK belonging to a conversation that began with a SYN from inside is let through. In other words, the firewall knows the exact picture you drew today and suspects any letter that deviates from it. The contest between those who know the rules and those who aim for the exceptions plays out in this single greeting.

3. Follow Along

Before you begin. TCP sequence numbers count bytes. SYN and FIN each consume one sequence number. Graceful close can use separate or combined FIN/ACK segments; a reset is another ending. A completed handshake establishes TCP reachability, not application health.

3-1. Drawing the Handshake

On paper, raise two pillars (client, server) and draw the handshake with three arrows. Label each arrow SYN, SYN-ACK, ACK.

Repeat three times until you can draw it without looking at the book. Knowing in your head and drawing with your hand are different things. When the picture comes out of your hand, you’ve understood it.

3-2. Seeing the Distribution of Connection States Right Now

The greetings you learned today are flying around your computer at this very moment. Let’s count the traces by state.

Get-NetTCPConnection | Group-Object State | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize

Name        Count
----        -----
Established    76
Bound          70
TimeWait       60
Listen         56
CloseWait       6
SynSent         1

Illustrative output. Values, wording, account names, and available fields vary by environment.

How to read the output — each state is a single frame of the greetings you learned today:

• Listen 56 — 56 of the “doors waiting for guests” from Step 34.
• Established 76 — 76 “on a call” connections that passed the three-handshake.
• TimeWait 60 counts entries in that state in this snapshot. TIME-WAIT supports handling delayed segments and retransmitted closing traffic; it does not specify that every connection followed exactly four separate segments.
• CloseWait 6 — the other side sent a FIN (“I’ll hang up”), but my program hasn’t yet made its closing greeting.
• SynSent 1 — a connection caught right at this moment waiting for the other side’s SYN-ACK after sending its SYN. A single frame mid-handshake.

Even at the instant you run this command, the numbers in this table are alive and moving. Run it a few times and watch the numbers rise and fall — the handshakes and goodbyes you couldn’t see start to appear as numbers.

3-3. Seeing Live Connections

netstat -an | findstr ESTABLISHED

  TCP    127.0.0.1:3452         127.0.0.1:64743        ESTABLISHED
  TCP    127.0.0.1:3452         127.0.0.1:64744        ESTABLISHED
  TCP    127.0.0.1:10086        127.0.0.1:64728        ESTABLISHED

Illustrative output. Values, wording, account names, and available fields vary by environment.

How to read the output: the left side is my address+port, the right side is the other’s address+port. The example lines are conversations among programs inside my computer (127.0.0.1 to 127.0.0.1). Run it with a website open in your browser and you’ll see lines ESTABLISHED with …:443 — all connections that safely completed the three greetings.

Making it onto this list means “SYN → SYN-ACK → ACK succeeded.” From the result alone, you can reconstruct the handshake behind it.

3-4. Tracing Handshake Success and Failure from Results

Let’s look at last chapter’s Test-NetConnection again with this section’s eyes.

Test-NetConnection 172.30.1.254 -Port 80

TcpTestSucceeded : True

Illustrative output. Values, wording, account names, and available fields vary by environment.

What happened behind that single True line: my computer sent a SYN → the gateway replied with SYN-ACK → my computer finished with ACK. True because the three lines crossed safely.

A failed test to 127.0.0.1:65000 establishes only that this TCP attempt failed. A reset from an unused local port is one explanation; filtering and other conditions are possible. The boolean result alone is not a packet capture proving which flags were exchanged.

3-5. The Principle of Scanning — Three Kinds of Answers

Now let’s organize how this handshake becomes the raw material of scanning tools (like nmap). When a scanner sends a SYN to a port:

How to read it: a scanning tool is not magic. It’s just a machine that uses TCP’s etiquette rules to “knock on doors and classify the answers.” And in subsection 3-4, you performed this classification by hand — both kinds.

Predict: what state does a server end up in if an attacker sends only SYNs and never sends the final ACK? (Answer: the server sends SYN-ACK and is left holding “half-open” connections, endlessly waiting for the final ACK. When tens of thousands of such half connections pile up, the server’s memory is paralyzed — the seed of a SYN flood attack. For now, just take away the intuition that “forcing a greeting to break off puts the other side in trouble.”)

3-6. Compressing It into an Analogy

Compress this section’s lesson into one paragraph of phone analogy and say it out loud.

“TCP is like a telephone. When calling, it starts with hello (SYN), then yes hello (SYN-ACK), then ah yes (ACK). During the call, it confirms it’s listening with uh-huh (ACK) and stamps numbers so the order doesn’t tangle (sequence numbers). When ending, it finishes politely with I’ll hang up (FIN), yes (ACK), me too (FIN), yes (ACK). If the number doesn’t exist, a slammed receiver (RST) comes back.”

If this paragraph flows out smoothly, the core of this unit is done.

4. Missions & Exercises

Mission — A Handshake Observation Journal of My Computer

1. Visit two websites in a row in your browser
2. Right after visiting, run netstat -an | findstr ESTABLISHED and find the port-443 connection lines
3. Record the state distribution with Get-NetTCPConnection | Group-Object State
4. Run the same command again 2–3 minutes later, and write down what grew and what shrank (especially the change in TimeWait)
5. Organize your observations into tcp-diary.txt, and on the last line fill in “Today on my computer, the number of connections that had sent a SYN and were waiting for an answer (SynSent) was __”

Exercises

Q1. Close the book and draw the 3-way handshake on blank paper. Arrow directions and flag names must all be correct.

Q2. Write the three possible answers that can come back when a port scanner sends a SYN, and the scanner’s verdict for each.

Q3. Explain why graceful closure handles each TCP direction separately. Why can a common four-segment close sometimes use fewer segments?

Q4. You received a report: “I can’t access the website.” Depending on which line of the handshake it stopped at, how do the candidate causes differ? Answer in three branches.

[ESTABLISHED right after visiting]
127.0.0.1:3452 ↔ 127.0.0.1:64743   ESTABLISHED
(if the browser was open)  myaddress:5xxxx ↔ externalserver:443  ESTABLISHED

[State distribution — 1st run]
Established 76 / Bound 70 / TimeWait 60 / Listen 56 / CloseWait 6 / SynSent 1

[2 minutes later — 2nd run]
TimeWait shrank and Established grew, or vice versa — as you browse, new connections appear
and finished connections fade away through TimeWait

[Sample last line]
"Today on my computer, the number of connections waiting for an answer after sending a SYN (SynSent) was 1"

6. Common Pitfalls & Fixes

Wall 1. “I keep mixing up the SYN, SYN-ACK, ACK order.”

Symptom: you mix up what the second one was or what the server sends.

Cause: because you’re trying to memorize it as letters.

Fix: go with the phone analogy. “Hello (dialing) → yes hello (answering) → ah yes (confirming).” Analogies are unforgettable. Drawing the picture three times also works well.

Wall 2. “There are too many unfamiliar words in the state table.”

Symptom: you get stuck on state names like Bound, CloseWait, TimeWait.

Cause: every state is just one frame of the greetings you learned today.

Fix: read just three with confidence — Listen (a waiting door), Established (on a call), TimeWait (the lingering grace after goodbye). Classify the rest as “a middle frame of the handshake or the goodbye.” As the measurement in subsection 3-2 caught one SynSent, middle states are momentary, so small counts are normal.

Wall 3. “What on earth is a packet with only an ACK?”

Symptom: while reading explanations, you get stuck on the phrase “a packet with only the ACK flag set.”

Cause: after a connection is established, nearly every data letter carries an ACK — because it always doubles as the confirmation marker “I got your letter.”

Fix: just understand that TCP communication after the handshake is a continuous stream of “data + confirmation.” ACK is not a special event but an everyday greeting.

Wall 4. “A red warning appeared in Test-NetConnection so I thought it failed.”

Symptom: on a closed port, a warning line like WARNING: TCP connect to (… ) failed appears (Illustrative source example).

Cause: it’s not an error but a result report. “We tried to connect and it didn’t work” is part of this command’s normal output.

Fix: False records a failed TCP connection attempt. It does not reveal whether a SYN was sent or whether an RST, timeout, or local failure occurred. Additional observation is needed.

Wall 5. “I can’t organize when RST shows up.”

Symptom: RST appears in several situations, which is confusing.

Cause: RST is “the all-purpose reset for abnormal situations,” so it has several stages to appear on.

Fix: remember just three. ① When a connection request arrives for a nonexistent room. ② When a letter arrives for an already-terminated connection. ③ When something doesn’t add up and the connection is forcibly cut. The common thread: a declaration that “this conversation cannot stand.”

7. Summary

This section’s Concepts

This section’s Commands

The Instinct That Matters More Than Commands

Someone who knows the 3-way handshake and someone who doesn’t see the same failure differently. Faced with “it won’t connect,” the one who doesn’t know ends at “the internet is broken,” but the one who knows asks at which line of the handshake it stopped. No answer, RST, an unfinished handshake — the same “doesn’t work” is three different illnesses. The ability to ask where a problem sits is real skill.

This section’s chapter had few commands and many pictures. Chapters like this last the longest. Commands may be forgotten, but the picture remains, and that picture finds the way in unfamiliar situations. Only someone who knows proper etiquette recognizes strange etiquette — the half handshake of a scan, the flooding SYNs of a flood, all are variations visible against this section’s bright picture. Three greetings and four goodbyes — now they’re yours.

Once every box is checked, Step 35 is complete.