Step 284. CTF Debrief Block A + Team Meeting: Weakness Analysis — Drawing the Team’s Map with Data, Not Gut

Level 3 — The CTF Competition Cycle | Difficulty ★★☆☆☆ | Estimated time: 2 days (0.5 day debrief + 1 day data cleanup & team meeting)

Prerequisites: three competition appearances (Steps 279–283) with each debrief block complete. Especially Step 280’s debrief block A procedure (digging unsolved problems to the bottom).

• What you need: the problem lists and team solve records (or scoreboard captures) of competitions #1–#3, a team meeting tool (video call or chat, either works), and the Python analysis script built below. The script’s run screen is measured; the competition results data is fictional example data.
• ⚠️ All exercises in this chapter are for your own lab and legal platforms only. Applying them to unauthorized systems is a crime. The scoreboards and solve records handled today are information the competition organizers made public — use only legitimately collected data.
• This chapter is a consolidation chapter — no new attack techniques. Instead, you learn how to turn the records accumulated over three competitions into "the team’s map."

After three competitions, every team has the same conversation. "I think we’re kind of good at Web." "Honestly, isn’t Pwn hopeless for us?" — all gut feeling. A training direction set by gut can only be evaluated by gut a month later.

Today we turn that conversation into numbers. Organize the results of competitions #1–#3 into a per-field table, distinguish whether "fields we can’t solve" are "fields nobody could solve" or "fields only we can’t solve," then fix the next competition’s training direction in a 60-minute team meeting. If the debrief is the device that fixes an individual’s skill, today’s meeting is the device that fixes the team’s skill.

---

1. Learning Objectives

By the end of this chapter, you will be able to:

• Organize three competitions’ results into a per-field table of attempts / successes / time taken
• Distinguish "fields we’re weak in" from "fields the competition made hard" using the overall solve rate
• Classify weaknesses into three kinds (skill gap / missing owner / time allocation) and set cause-specific prescriptions
• Run a team meeting with an agenda and time limits, and leave minutes
• Convert analysis results into "action plans with owners and deadlines through the next competition"

---

2. Background Knowledge — Today’s Tools and Concepts

Today’s Tools at a Glance

2-1. Why Data — You Can’t Fix a Team on Gut

Individual debriefs work tolerably on gut. "I couldn’t solve that problem because I didn’t know the heap" — memory is fairly accurate. But at team scale it’s a different story. Each person remembers a different competition, and the difficulty of one’s own field is remembered exaggerated.

The real reason data is needed is the language of the meeting. The claim "Pwn seems like our weakness" invites the rebuttal "no, that competition’s Pwn was weirdly hard," and that argument never ends. With a table saying "5 Pwn problems offered, our attempts 0, overall solve rate 25%," the argument disappears and prescription discussion begins.

2-2. The 3 Weakness Classes — Same Zero, Different Causes

Reading the analysis table, not all zeros are the same zero. The prescription differs completely by cause.

Why this distinction matters: prescribing headcount for a skill problem, or study for a headcount problem, collapses you identically at the next competition. Asking "why" over the table is the meeting’s core agenda.

2-3. The Overall Solve Rate — A Filter That Strains Out "The Competition Was Hard"

Most CTFs publish the number of solving teams per problem after the end. Solving teams ÷ participating teams is the overall solve rate. This number is needed for exactly one reason — to remove the illusion of self-reference.

Failing a problem with a 5% overall solve rate is not a weakness; it’s normal. Missing a 60% overall-solve-rate problem three competitions in a row is a real weakness. When the claim "our Web is weak" comes up in a meeting, pass it through this filter first — half of such claims resolve into the simple fact that the competition was hard.

2-4. Running a Team Meeting — A Meeting Without an Agenda Is Chit-Chat

The reason learning teams’ meetings fizzle is mostly operational failure. No agenda turns it into chatter; no time cap lets one topic swallow the meeting. The standard operation of a 60-minute meeting:

There are two rules. When time is up, move to the next agenda item (unresolved items are recorded as "deferred" and moved on), and every decision gets an owner and a deadline. "Everyone try hard together" is not a decision but an evasion of one — an item with no owner’s name in the minutes does not get executed.

---

3. Follow Along

3-1. Finish Debrief Block A First

The team meeting must open after individual debriefs are done. Each person completes the debrief block A procedure set in Step 280 — pick 3 problems you couldn’t solve at this competition (CTF #3) and dig them to the bottom — and comes to the meeting.

Open the meeting first without debriefs and every answer to "why couldn’t we solve it" becomes guesswork. Each person bringing one line of failure cause found in their debrief is the meeting’s input.

Individual prep before attending the meeting:
- Debrief notes for 3 unsolved problems (Step 280 procedure)
- One line of failure cause per problem: pick one of "knowledge gap / time shortage / never attempted" + specifics

3-2. Organize Competition Results into CSV

Gather the problem lists of competitions #1–#3 into a table. Eight cells per problem — competition, field, difficulty, problem name, our attempt (0/1), success (0/1), time taken (minutes, if solved), overall solve rate (%). If the scoreboards survive, copy from captures; if not, pool memories to reconstruct.

Example data (the file below is fictional data made for this chapter’s explanation):

contest,category,difficulty,problem,attempted,solved,minutes,overall_solve_rate
CTF#1,Web,Easy,login-bypass,1,1,25,68
CTF#1,Web,Medium,ssti-note,1,0,90,22
CTF#1,Pwn,Easy,bof-basic,0,0,0,41
CTF#1,Crypto,Easy,rsa-small,1,1,40,55
CTF#2,Web,Easy,cookie-admin,1,1,20,71
CTF#2,Pwn,Easy,ret2win,0,0,0,38
CTF#3,Web,Easy,jwt-none,1,1,30,66
CTF#3,Crypto,Easy,base-layers,1,1,15,78

For unattempted problems, leave attempted, solved, and minutes all 0. If you didn’t record minutes, you may leave it blank this time — but from the next competition, add a time column to the board. Average solve time is the only material that measures the team’s speed.

3-3. Build and Run the Aggregation Script

You could count the table by eye, but past 20 problems hand-counting goes wrong. Save the script below as ctf_stats.py — this chapter’s code and output are measured, actually run and verified.

# ctf_stats.py — convert a competition-results CSV into a per-field weakness analysis table
# usage: python ctf_stats.py results.csv
import csv
import sys
from collections import defaultdict

def load(path):
    rows = []
    with open(path, encoding="utf-8") as f:
        for r in csv.DictReader(f):
            rows.append({
                "contest": r["contest"], "category": r["category"], "difficulty": r["difficulty"],
                "attempted": int(r["attempted"]), "solved": int(r["solved"]),
                "minutes": int(r["minutes"]), "overall": float(r["overall_solve_rate"]),
            })
    return rows

def main(path):
    rows = load(path)
    agg = defaultdict(lambda: {"offered": 0, "attempted": 0, "solved": 0,
                               "time_sum": 0, "overall_sum": 0.0})
    for r in rows:
        a = agg[r["category"]]
        a["offered"] += 1
        a["attempted"] += r["attempted"]
        a["solved"] += r["solved"]
        if r["solved"]:
            a["time_sum"] += r["minutes"]
        a["overall_sum"] += r["overall"]

    print("| category | offered | attempted | solved | coverage | per-attempt | avg time | overall solve rate |")
    print("|----------|---------|-----------|--------|----------|-------------|----------|--------------------|")
    total_solve = 0
    total_n = 0
    stats = {}
    for cat in sorted(agg):
        a = agg[cat]
        cov = a["solved"] / a["offered"] * 100
        att = a["solved"] / a["attempted"] * 100 if a["attempted"] else 0.0
        avg_t = a["time_sum"] / a["solved"] if a["solved"] else 0.0
        field = a["overall_sum"] / a["offered"]
        stats[cat] = (cov, field)
        total_solve += a["solved"]
        total_n += a["offered"]
        avg_t_s = f"{avg_t:.0f}min" if a["solved"] else "-"
        print(f"| {cat} | {a['offered']} | {a['attempted']} | {a['solved']} | "
              f"{cov:.0f}% | {att:.0f}% | {avg_t_s} | {field:.0f}% |")
    print(f"noverall score coverage: {total_solve}/{total_n} "
          f"({total_solve / total_n * 100:.0f}%)")

    print("n[weakness diagnosis]")
    for cat, (cov, field) in sorted(stats.items(), key=lambda x: x[1][0]):
        if cov == 0 and field >= 30:
            verdict = "⚠ skill weakness — other teams solve it, only we're at 0"
        elif cov == 0:
            verdict = "△ unattempted — no attempts at all; missing owner suspected"
        elif cov + 20 < field:
            verdict = "⚠ relative deficit — well below the overall solve rate"
        else:
            verdict = "○ normal range"
        print(f"- {cat}: ours {cov:.0f}% vs overall {field:.0f}% → {verdict}")

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "results.csv")

Measured result of running it on this chapter’s example data (24 problems). Your numbers will differ with your data.

python ctf_stats.py results.csv

| category | offered | attempted | solved | coverage | per-attempt | avg time | overall solve rate |
|----------|---------|-----------|--------|----------|-------------|----------|--------------------|
| Crypto | 6 | 5 | 4 | 67% | 80% | 39min | 42% |
| Forensics | 3 | 0 | 0 | 0% | 0% | - | 50% |
| Pwn | 5 | 0 | 0 | 0% | 0% | - | 25% |
| Reversing | 3 | 3 | 2 | 67% | 67% | 45min | 42% |
| Web | 7 | 6 | 3 | 43% | 50% | 25min | 37% |

overall score coverage: 9/24 (38%)

[weakness diagnosis]
- Forensics: ours 0% vs overall 50% → ⚠ skill weakness — other teams solve it, only we're at 0
- Pwn: ours 0% vs overall 25% → △ unattempted — no attempts at all; missing owner suspected
- Web: ours 43% vs overall 37% → ○ normal range
- Crypto: ours 67% vs overall 42% → ○ normal range
- Reversing: ours 67% vs overall 42% → ○ normal range

How to read it: this example team’s story is summarized in one table. Forensics has 0 attempts with a 50% overall solve rate — Easy problems never even opened, so a missing owner is the primary cause. Pwn is also unattempted, but with a low overall solve rate, whether it’s "our problem specifically" is judgment-reserved. Web sits at 43% coverage, above the overall average (37%), so it isn’t a weakness. Gut said "Pwn is the worst," but the data says "Forensics first."

3-4. Hold the 60-Minute Team Meeting

Proceed along 2-4’s timetable. Designate one facilitator (rotate the role), and start with 3-3’s table on screen. An example of the dialogue that actually passed per agenda item:

[5–20 min data review]
facilitator: "Forensics, 3 problems, 0 attempts. Correct?" → all confirm. Facts fixed.
[20–40 min cause analysis]
"why didn't anyone open them?" → "nobody on the team has done Forensics, so nobody volunteered"
→ class: missing owner (not a skill problem — it's never been done)
[40–55 min prescription decisions]
- Forensics: A owns it, learns the Step 244–247 route + solves 5 Easy problems before the next competition
- Pwn: can't staff it now → B does just the intro (pwnable.kr bof~collision); expectation is "attempts"
- 3 Web Medium failures: share in individual debriefs → hand over to Step 288's boot camp topics

In the last 5 minutes, read the minutes aloud. "If no objection the moment it’s heard, it’s fixed" — this procedure makes the meeting a device of decision.

3-5. Leave the Minutes

Four cells are enough for the minutes format.

# CTF #3 Debrief Team Meeting Minutes (date: ____)

### 1. Data summary
- Overall score coverage: __/__ (__%)
- Weakness diagnosis: Forensics (missing owner), Pwn (unattempted), Web Medium (skill)

### 2. Cause per weakness
| Category | Data | Cause class |
|------|--------|-----------|
| Forensics | 0 attempts / overall 50% | missing owner |
| Web Medium | 4 attempts, 0 solved | skill gap — SSTI·SSRF types |

### 3. Decisions (owner · deadline)
| To do | Owner | Deadline |
|-------|------|------|
| Forensics intro + 5 Easy problems | A | before CTF #4 |
| pwnable.kr intro, 3 problems | B | before CTF #4 |
| Pick Web boot-camp types | C | this week |

### 4. Deferred
- The question of a permanent Pwn owner — revisit at the next meeting

Why this format: minutes without table 3 are the same as not having met. Cell 4 keeps unresolved items from vanishing — even a deferral must be recorded to become the next meeting’s agenda.

---

4. Missions & Exercises

Mission — Complete Your Team’s Weakness Analysis Minutes

1. Organize the results of CTF #1–#3 into 3-2’s CSV format — including the overall solve rate column.
2. Run ctf_stats.py to pull the per-field table and weakness diagnosis.
3. Each member completes Step 280’s debrief block A and prepares one line of failure cause.
4. Hold the 60-minute meeting on 2-4’s timetable, and judge every weakness by the 3 classes (skill / owner / time).
5. Write minutes in 3-5’s format — every decision must have an owner and a deadline.
6. The next competition (CTF #4) becomes the place that verifies these minutes’ decisions — it continues in Step 285.

Exercises

Exercise 1. When the claim "our team is weak at Web" comes up in a meeting, which two pieces of data should verify it?

Exercise 2. Among the 3 weakness classes, what data difference separates "missing owner" from "skill gap"? What is each one’s prescription?

Exercise 3. Why is it a problem to close a team meeting with "let’s all work hard on Pwn together"?

Exercise 4. Why keep a separate "deferred" cell in the minutes?

---

Answers & completion criteria · expand/collapse

5. Model Answers & Completion Criteria

Mission Model Answer

Check against these verification criteria.

1. CSV completeness: are all problems from 3 competitions present as rows, with overall solve rates filled even on unattempted rows — rows missing the rate distort the diagnosis.
2. Table reproduction: do the script’s output numbers match the CSV — hand-verify one or two rows.
3. Grounds for cause judgments: does each weakness’s 3-class judgment carry both "data + debrief notes" as grounds — a tag attached by gut erases the meeting’s meaning.
4. Minutes executability: does every decision row have an owner and a deadline? If even one row has "everyone" as the owner, it’s unfinished.
5. Verification plan: is it written in the minutes or a separate memo what each decision will be verified by at the next competition (e.g., "Forensics attempts ≥ 1")?

Exercise Answers

Answer 1. First, our team’s Web score coverage (solved per offered), and second, the overall solve rate of those same problems. If coverage is low but the overall rate is also low, that competition’s Web was simply hard; if the overall rate is high and only ours is low, it’s a real weakness. Looking at just one catches you in the illusion of self-reference.

Answer 2. They’re separated by attempt count. Zero attempts means missing owner (never even opened); attempts with low success means skill gap. The prescription for a missing owner is designating an owner and assigning an intro route; the prescription for a skill gap is focused study of that type (Step 288’s boot camp). Same zero, different cause — swap the prescriptions and the next competition brings the same result.

Answer 3. Because a decision whose owner is "everyone" is a decision with no owner. Without who does what by when, the next meeting can’t ask whether it was achieved — and a plan that can’t be asked about is not training but a wish. "B does 3 pwnable.kr intro problems before CTF #4" — person, volume, deadline — is a verifiable decision.

Answer 4. To keep items that reach no conclusion within the time from evaporating. Without a deferred cell, an unresolved item vanishes with the words "let’s talk next time," and the next meeting restarts the same discussion from zero. A recorded deferral becomes the next meeting’s agenda and the discussion continues — the deferred cell creates the meetings’ continuity.

Completion Criteria Checklist

• [ ] I organized three competitions’ results into CSV (overall solve rate included)
• [ ] I printed the per-field table and weakness diagnosis with ctf_stats.py
• [ ] Each member finished debrief block A (3 unsolved problems) and brought one line of failure cause
• [ ] I ran the 60-minute meeting on the timetable (facilitator designated)
• [ ] I judged every weakness by the 3 classes (skill / owner / time)
• [ ] I filled the minutes’ four cells (data / causes / decisions / deferred)
• [ ] Every decision has an owner and a deadline
• [ ] I set the verification criteria for the next competition

---

6. Common Pitfalls & Fixes

Wall 1. Running the script throws FileNotFoundError

FileNotFoundError: [Errno 2] No such file or directory: 'results.csv'

Cause: there’s no results.csv in the folder where you ran the script. Most of the time, the folder where you made the CSV differs from the run folder.

Fix: move into the folder with the CSV and run there, or pass the full path as the argument — python ctf_stats.py C:ctfresults.csv. For reference, this message was actually reproduced and verified while writing this chapter.

Wall 2. Characters are garbled — the CSV opened with strange text

Symptom: a CSV saved from Excel throws UnicodeDecodeError when the script reads it, or conversely, the script’s output opens garbled in Excel.

Cause: Excel’s default save encoding is often not UTF-8 (CP949), and this script reads with encoding="utf-8".

Fix: open the CSV in Notepad or VS Code and re-save with "Save As → encoding UTF-8." Making CSVs in a text editor from the start is simplest.

Wall 3. The meeting ran past 60 minutes with zero decisions

Symptom: time passed trading impressions over the data.

Cause: no per-agenda time caps, or no facilitator.

Fix: from the next meeting, designate a facilitator whose sole mission is watching the clock. When time is up, they declare "this item is recorded as deferred; moving on." Recording a stalled item as deferred and moving on beats forcing it shut — a meeting’s purpose is not discussion but decisions.

Wall 4. I can’t get overall solve rate data

Symptom: the competition platform doesn’t publish per-problem solving team counts.

Cause: small competitions sometimes don’t provide statistics.

Fix: two substitutes. First, capture the scoreboard’s top-team score distribution right after the end and count yourself "how many teams hold this problem’s points." Second, estimate by the number of public write-ups — a problem with many write-ups was popularly solved. Less accurate, but sufficient to separate "only we couldn’t" from "it was hard for everyone."

Wall 5. The weakness analysis made things awkward between teammates

Symptom: a table saying "the Pwn owner didn’t solve anything" reads like blame of a specific person.

Cause: the table is fact, but the meeting read that fact as a person evaluation.

Fix: agree on one rule before the meeting starts — "the table points at problems, not people." And make the table’s rows per-field only, never per-person. Individual growth belongs to individual debriefs (Step 280); the team meeting’s territory is structure (owner placement, time rules). Only a meeting that keeps this boundary builds a team unafraid of data.

---

7. Summary

Today’s Concepts

Today’s Tools & Formats

The Instinct Beyond Commands

If the debrief is the device that fixes the individual, today’s meeting is the device that fixes the machine called the team. An individual’s skill rises through debriefs, but a structural defect like "a team where nobody opens Forensics" stays as-is no matter how hard individuals try — structure can only be fixed in meetings.

And this device’s fuel is data. A meeting opened on gut ends pleasantly and the next competition’s result is the same; a meeting opened on data ends uncomfortably and the next competition’s result differs. Accepting the weakness the table states — like the example team’s Forensics — is where growth begins. The next competition is the examination hall for today’s minutes.

---

Once every box is checked, Step 284 is complete.