TCP, Ports, and Why a Timeout Is Evidence

28.0 What this chapter gives you#

1. You will be able to say exactly what the internet’s basic delivery service promises, and the much longer list of things it does not promise.
2. You will be able to explain TCP and UDP as two answers to the same problem, and say why UDP is not a worse TCP but a different tool.
3. You will be able to explain a port as a 16-bit number, and a socket as the four numbers that make one conversation unique on the whole planet.
4. You will be able to draw the three-way handshake with real sequence numbers, walk the eleven states of a connection, and explain the four-way close.
5. You will be able to explain how TCP turns an unreliable service into a reliable one, and what it costs in round trips.
6. You will be able to separate flow control from congestion control, and say why a link that loses one packet in a hundred destroys throughput.
7. You will be able to name the three possible endings of a connection attempt, and tell them apart from the failure message alone.
8. You will be able to state precisely what the reader’s fifteen-second silence proves, and be equally precise about what it does not prove.
9. You will be able to run curl -v, nc -vz, nmap and tcpdump and say what each one adds to the evidence.

28.1 What IP alone gives you, and what it does not#

PLAIN28.1.1 in simple words#

1. The internet’s basic delivery service is called IP, short for Internet Protocol. It moves one small parcel of data from one address to another.
2. That parcel is called a packet. It carries a from-address, a to-address, and some data.
3. IP makes exactly one promise, and it is a weak one: it will try. The official phrase for it is best effort.
4. IP does not promise the packet will arrive, nor that ten packets sent in order will arrive in order, nor that a packet arrives only once.
5. Worst of all, IP does not promise to tell you when a packet is lost. Nobody rings a bell. The packet just stops existing.
6. So something had to be built on top of IP that turns “try” into “it arrived, complete, in order, exactly once”. That something is TCP.

PLAIN28.1.2 a picture in your head#

1. Imagine a postal system with no tracking, no receipts and no complaints desk.
2. Most postcards arrive. Some do not. Nobody tells you which.
3. Now send a 300-page book through this system, one page per postcard. The receiver gets a pile with pages missing, out of order, maybe one twice.
4. To fix it you would number every page, and the receiver would write back saying which numbers arrived, and you would re-post the gaps.
5. That numbering-and-writing-back scheme, done automatically, is TCP.

Where this comparison breaks: real post loses letters mostly by accident. IP loses packets mostly on purpose. A router with a full queue drops the newest packet deliberately, because dropping is the only signal it has to tell senders to slow down. Loss is not a malfunction in IP. Loss is the brake pedal.

PLAIN28.1.3 a worked example#

1. Suppose you send ten packets numbered 1 to 10 across the internet.
2. A router in the middle has a full queue for a few milliseconds and drops two.
3. Two others take a different path because the network is load balancing, the same behaviour the reader saw at hops 5, 6, 8, 9 and 11 of their traceroute.

4. The receiver has eight packets, two gaps, and one pair swapped. IP has done its job correctly. Nothing is broken.
5. Nobody sent any message saying “3 and 6 are gone”. The sender must work it out by noticing that 3 and 6 were never confirmed.
6. That is precisely what TCP does, and it is the whole reason TCP exists.

PLAIN28.1.4 what is really happening inside#

1. Each router looks only at the destination address in the packet header, decides the next hop, and forwards it.
2. Routers hold packets in a fixed-size queue while the outgoing link is busy. When the queue is full, an arriving packet is discarded. That rule is called tail drop, and no record of the packet is kept anywhere.
3. Sometimes the router sends back a complaint using ICMP, the Internet Control Message Protocol, the mechanism behind “host unreachable” and traceroute.
4. But ICMP is carried in IP packets, so complaints can be lost too, and many networks throttle or block ICMP on purpose.
5. The reader saw exactly that: hops 13 to 20 of their traceroute returned * * *, which usually means ICMP replies were suppressed, not a broken path.

TECHNICAL28.1.5 the engineer’s version#

1. IPv4 is RFC 791, published September 1981 by Jon Postel at the Information Sciences Institute for DARPA. IPv6 is RFC 8200, July 2017.
2. The IPv4 service model is unreliable, connectionless, best-effort datagram delivery. There is no connection state in any router.
3. The idea comes from Vint Cerf and Bob Kahn’s paper “A Protocol for Packet Network Intercommunication”, IEEE Transactions on Communications, May 1974.
4. In 1978 the design was deliberately split into IP for delivery and TCP for reliability, by Cerf, Postel and Danny Cohen, so packet voice could skip the reliability part. That split is why UDP exists. ARPANET switched from NCP to TCP/IP on 1 January 1983, a cutover still remembered as “flag day”.

5. Observe raw IP behaviour with ping -s for size effects, traceroute for the path, and tcpdump -n -v ip to read headers on the wire.

WORDS28.1.6 remember these#

1. Packet — one small parcel of data with an address on it — an IP datagram carrying a header and a payload.
2. Best effort — we will try, we promise nothing — a service model with no delivery, ordering or duplicate-suppression guarantees.
3. ICMP — the network’s complaint messages — the Internet Control Message Protocol, RFC 792, advisory and frequently filtered.

28.2 TCP and UDP: two ways to use IP#

PLAIN28.2.1 in simple words#

1. Sitting directly on top of IP there are two main choices, and almost everything you use picks one of them.
2. TCP, the Transmission Control Protocol, is the careful one. It sets up a connection first, numbers everything, checks everything arrived, and puts it back in order.
3. UDP, the User Datagram Protocol, is the plain one. It adds almost nothing. It sends your parcel and forgets about it.
4. It is tempting to say TCP is good and UDP is bad. That is wrong. TCP’s care costs time, and repairing a missing piece can be worse than living without it.
5. In a live voice call, a lost slice of sound from 300 milliseconds ago is worthless, so a voice call uses UDP and carries on. A file download uses TCP, because a file with a hole in it is not a file.
6. TCP trades time for completeness. UDP trades completeness for time.
7. A third name you will hear is QUIC. QUIC runs on top of UDP and rebuilds reliability in a smarter way. Chapter 32 covers it properly.

PLAIN28.2.2 a picture in your head#

1. TCP is a registered parcel service with signatures. Before sending, the courier phones ahead and confirms someone is home.
2. Every parcel is numbered and the receiver signs for each one. If a signature does not come back, the courier sends that parcel again.
3. UDP is a person throwing newspapers from a bicycle. No phone call, no signature, no numbering, no second attempt. Fast, cheap, mostly fine.
4. For a newspaper, mostly fine is fine. For a contract it is a disaster.

Where this comparison breaks: the courier does not really phone ahead to a human. The three-way handshake is between two pieces of software, and it exists mainly to agree on starting numbers, not to check that anyone is home. Also, the newspaper thrower does at least check the paper is not torn: UDP has a checksum, optional over IPv4 and mandatory over IPv6.

PLAIN28.2.3 a worked example#

1. Here are the two headers side by side, drawn to the same scale. Each row is 32 bits, which is 4 bytes.

TCP header, 20 bytes minimum, up to 60 with options

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-------------------------------+-------------------------------+
|          Source Port          |       Destination Port        |
+---------------------------------------------------------------+
|                        Sequence Number                        |
+---------------------------------------------------------------+
|                    Acknowledgement Number                     |
+-------+-----------+-----------+-------------------------------+
| Offset|  Reserved |   Flags   |            Window             |
+-------+-----------+-----------+-------------------------------+
|           Checksum            |        Urgent Pointer         |
+-------------------------------+-------------------------------+
|                    Options, 0 to 40 bytes                     |
+---------------------------------------------------------------+

UDP header, exactly 8 bytes, always

+-------------------------------+-------------------------------+
|          Source Port          |       Destination Port        |
+-------------------------------+-------------------------------+
|            Length             |           Checksum            |
+-------------------------------+-------------------------------+

2. UDP then stops. TCP keeps going for another 12 bytes minimum, and those 12 bytes are the entire machinery of reliability.
3. Sending 100 bytes by UDP costs 8 bytes of overhead. By TCP it costs at least 20 bytes plus a three-packet handshake: a bad deal for one short message, invisible for a 50 MB download.

PLAIN28.2.4 what is really happening inside#

1. When a program sends by UDP, the operating system writes an 8-byte header, hands the result to IP, and the call returns. Nothing is remembered, and if the message is lost, no software anywhere notices.
2. When a program sends by TCP, the operating system copies the data into a send buffer belonging to that connection, and it sits there until confirmed.
3. The kernel keeps a timer per connection. If a confirmation does not arrive in time, it sends the data again from the buffer. That is why TCP needs a connection: it is the place to keep the buffer, the timers and the numbering.
4. And TCP does not preserve your write boundaries. Three writes of 100 bytes may arrive as one read of 300. TCP is a stream of bytes, not messages.
5. UDP does preserve boundaries. One send equals one receive, or nothing at all. That single difference decides many design arguments.

TECHNICAL28.2.5 the engineer’s version#

1. TCP is now specified in RFC 9293, “Transmission Control Protocol (TCP)”, STD 7, published August 2022, edited by Wesley Eddy.
2. RFC 9293 obsoletes the original RFC 793 of September 1981 and folds in RFC 879, RFC 2873, RFC 6093, RFC 6429, RFC 6528 and RFC 6691. If you cite one document for TCP today, cite RFC 9293.
3. UDP is RFC 768, published 28 August 1980 by Jon Postel. It is three pages long. That is the entire specification and it has never needed replacing.
4. QUIC is RFC 9000, May 2021. It runs over UDP and implements its own loss recovery, congestion control and encryption. HTTP/3 over QUIC is RFC 9114, June 2022. Chapter 32 covers this in full.

5. Note also that streaming video is not a UDP application. Netflix and YouTube send video over TCP or QUIC, because a video file with holes will not decode. It is live conversation, not video, that needs UDP.
6. Observe both with ss -tan and ss -uan on Linux, or netstat -an -p tcp on macOS.

WORDS28.2.6 remember these#

1. TCP — the careful delivery service — a connection-oriented, reliable, ordered byte-stream transport protocol, RFC 9293.
2. UDP — the plain delivery service — a connectionless, unreliable, message-oriented transport protocol, RFC 768.
3. QUIC — a modern replacement — a UDP-based transport with integrated TLS 1.3 and per-stream loss recovery, RFC 9000.

28.3 Ports, sockets, and the four-tuple#

PLAIN28.3.1 in simple words#

1. An IP address gets a packet to the right machine. It does not say which program on that machine should receive it, and one laptop may run a browser, a mail client and a chat app all talking to the network at once.
2. So each packet also carries a number saying which program it is for. That number is called a port.
3. A port is a 16-bit number. Sixteen bits gives 65,536 values, 0 to 65,535.
4. Every TCP packet carries two: a source port, saying which program sent it, and a destination port, saying which program should get it.
5. Servers use fixed, agreed numbers so clients know where to knock. Web servers using TLS listen on port 443. Everyone knows this.
6. Clients need no fixed number. When your browser opens a connection, the operating system hands it a spare high-numbered port, used once and released.

PLAIN28.3.2 a picture in your head#

1. Think of a large office block with one street address, which is the IP address. It gets the post van to the building.
2. Inside there are 65,536 numbered pigeonholes. A pigeonhole number is a port.
3. Reception sorts every letter into a pigeonhole by the number on it. Some pigeonholes have a standing arrangement: 443 is always the web desk, 22 is always the remote-access desk, and anyone can look this up.

Where this comparison breaks: a real pigeonhole holds one item at a time. Port 443 on a busy server holds hundreds of thousands of simultaneous conversations, because a conversation is identified by four numbers together, not by the port alone. The next block fixes that picture properly.

PLAIN28.3.3 a worked example#

1. When the reader ran curl -v https://github.com, the operating system picked a spare local port. On macOS these come from the range 49152 to 65535.
2. Say it picked 51514. The connection is then described by four numbers.

source IP        192.168.0.101      the laptop, on the home LAN
source port      51514              picked by macOS, used once
destination IP   20.207.73.82       what DNS returned for github.com
destination port 443                the agreed port for HTTPS

3. Those four values together are the four-tuple, and they identify this one conversation and no other.
4. The laptop address here is an example inside the reader’s home range: the session record kept the router address 192.168.0.1 but not the laptop’s own. Everything else is exactly as recorded.

5. Look at column two and column four. Same source port, 51514, and no conflict, because the source IP differs. The tuple as a whole must be unique. That is how one server port holds hundreds of thousands of conversations at once.

PLAIN28.3.4 what is really happening inside#

1. Inside the operating system there are two kinds of socket, and confusing them causes real bugs.
2. A listening socket is a server saying “send me anything arriving on port 443”. It is identified by only two things: a local address and a local port. It never carries data; its only job is to catch new connection attempts.
3. A connected socket is one live conversation, identified by all four values: local IP, local port, remote IP, remote port.
4. When a connection request arrives, the kernel finds the listening socket, creates a brand new connected socket for that one client, and queues it.
5. The server program calls accept, which takes one connected socket off the queue and hands it over as a new file descriptor.
6. When a packet arrives, the kernel first matches all four values against connected sockets. Only if nothing matches does it try listening sockets.
7. If nothing matches at all, the kernel replies with a TCP RST, which produces the message “Connection refused”. Remember that reply. It is the centre of this whole chapter.

TECHNICAL28.3.5 the engineer’s version#

1. Port numbers are 16-bit unsigned integers. IANA divides the space by RFC 6335, published August 2011.
2. System Ports are 0 to 1023 and on Unix require root or a capability to bind. User Ports are 1024 to 49151. Dynamic or Private Ports are 49152 to 65535.
3. Ephemeral port ranges are an implementation detail, not a standard. Linux defaults to 32768 to 60999 in net.ipv4.ip_local_port_range. macOS and the BSDs default to 49152 to 65535 via net.inet.ip.portrange.first.

4. Port 443 carries HTTP over TLS, defined for that use in RFC 2818, May 2000. Port 853 for DNS over TLS is RFC 7858, May 2016. Port 465 was revoked and then restored for submission with implicit TLS by RFC 8314, January 2018; port 587 for submission is RFC 6409.
5. Port 443 became the universal port because firewalls and captive networks nearly always permit outbound 443, and because the traffic is encrypted, so middleboxes cannot easily inspect or rewrite it.
6. So HTTPS is now a general-purpose tunnel. Git, container registries, remote desktops, VPNs and messaging protocols all offer a 443 mode. That is a convention driven by firewall reality, not a design goal.
7. It is directly relevant to the reader’s session: git over SSH uses port 22, git over HTTPS uses port 443, and in the reader’s outage those two transports did not behave the same way.
8. Inspect with ss -ltn on Linux, lsof -i -P -n on macOS, and ss -tn state established for live connections.

WORDS28.3.6 remember these#

1. Port — a number saying which program a packet is for — a 16-bit unsigned field in the TCP or UDP header.
2. Four-tuple — the four numbers that make one conversation unique — source IP, source port, destination IP, destination port.
3. Ephemeral port — a temporary number for outgoing calls — a dynamically assigned local port, 49152 to 65535 on macOS, 32768 to 60999 on Linux.

28.4 The three-way handshake#

PLAIN28.4.1 in simple words#

1. Before TCP carries a single byte of your data, the two sides exchange three small packets. This is the three-way handshake.
2. Packet one, client to server: I want to open a connection, and my counting starts at this number.
3. Packet two, back: agreed, I heard your number, and my counting starts at this other number. Packet three, forward: agreed, I heard your number too.
4. After those three, both sides know the other is there, both know where the other’s numbering starts, and data can flow.
5. The names come from flags in the header. The first packet has the SYN flag set, for synchronize. The second has SYN and ACK. The third has ACK.
6. Notice the cost. The client cannot send data until packet three, and packet three needs packet two first. That is one complete round trip of pure waiting before any useful byte moves.
7. This handshake is also exactly where the reader’s problem lived. Their packet one went out and packet two never came back.

PLAIN28.4.2 a picture in your head#

1. Two people want to talk on a bad radio link, and must be sure both directions work before they start.
2. Aisha says: “Aisha here, my messages will be numbered starting at 481.”
3. Ben replies: “Ben here, I heard 481, so I expect 482 next. Mine start at 907.”
4. After Ben’s reply, Aisha knows the link works both ways, because she got an answer. But Ben knows his link works only one way until Aisha’s third message arrives. That is why three messages are needed and not two.

Where this comparison breaks: real people can repeat themselves and improvise. TCP cannot. If Ben’s reply is lost, Aisha has no way to tell whether Ben never heard her or Ben answered and the answer vanished. She can only send her opening line again and wait. Section 28.8 is about what she may conclude from silence.

PLAIN28.4.3 a worked example#

1. Here is the reader’s own connection, packet by packet, with realistic numbers filled in. The starting numbers are chosen unpredictably by each side.

laptop 192.168.0.101:51514        server 20.207.73.82:443
   state: CLOSED                     state: LISTEN
        |                                   |
        |  SYN                              |
        |  seq = 1842315776                 |
        |  ack = 0     win = 65535          |
        |  options: mss 1460, wscale 6, SACK|
        |---------------------------------->|
   SYN_SENT                            SYN_RECEIVED
        |                                   |
        |                     SYN, ACK      |
        |                     seq=3091447201|
        |                     ack=1842315777|
        |                     win = 65535   |
        |<----------------------------------|
        |                                   |
        |  ACK                              |
        |  seq = 1842315777                 |
        |  ack = 3091447202  win = 131712   |
        |---------------------------------->|
   ESTABLISHED                         ESTABLISHED
        |                                   |
        |  TLS ClientHello, 517 bytes       |
        |  seq = 1842315777                 |
        |---------------------------------->|

2. Read the acknowledgement numbers carefully. The server’s ack is 1842315777, the client’s start plus one.
3. That plus one is not a data byte. The SYN flag itself consumes one number in the sequence space, so the handshake can be acknowledged reliably. The client’s ack of 3091447202 does the same to the server’s start.
4. The window value 65535 is a raw 16-bit field. The wscale 6 option means “multiply my future windows by 2 to the power 6”, which is 64. So the later window of 131712 means about 8.4 megabytes of buffer space.

PLAIN28.4.4 what is really happening inside#

1. The application calls connect. The kernel allocates a connection control block, picks a source port, picks an initial sequence number, moves the socket to state SYN_SENT, and sends the SYN.
2. A retransmission timer starts, initially one second.
3. At the server, the SYN arrives and the kernel checks whether any socket is listening on port 443. If yes, it creates a small half-open entry, replies with SYN-ACK, and enters SYN_RECEIVED.
4. When the client’s ACK arrives, the server moves the entry into the accept queue, marks it ESTABLISHED, and wakes any program waiting in accept.
5. On the client, the SYN-ACK cancels the retransmission timer, moves the socket to ESTABLISHED, and unblocks the connect call.
6. Only now does the application start writing. For HTTPS the first thing written is a TLS ClientHello, not the web request.
7. If the SYN-ACK never arrives, the timer fires and the SYN is sent again with the wait doubled. That behaviour is the whole of section 28.8.

TECHNICAL28.4.5 the engineer’s version#

1. RFC 9293 section 3.5 defines the handshake. The three segments carry these exact control bits.

2. Initial sequence numbers must not be predictable. RFC 9293 section 3.4.1, folding in RFC 6528 by Fernando Gont and Steven Bellovin, February 2012, recommends this construction.

ISN = M + F(localip, localport, remoteip, remoteport, secretkey)

M  = a timer that ticks about every 4 microseconds
F  = a cryptographic hash, keyed with a host secret

3. The reason is security. If an attacker can guess your next sequence number, they can inject data or forge a connection without ever seeing your packets. That is a blind in-window attack.
4. The famous demonstration was the attack on Tsutomu Shimomura’s machines on 25 December 1994, attributed to Kevin Mitnick, which used sequence-number prediction with IP source spoofing.
5. Steven Bellovin had described the weakness in his 1989 paper “Security Problems in the TCP/IP Protocol Suite”, and later wrote RFC 1948, May 1996, the first randomization recommendation. RFC 5961, August 2010, adds hardening against blind RST and SYN injection.
6. The SYN and FIN flags each consume one sequence number, so they can be acknowledged and retransmitted with the same machinery as data.
7. Handshake cost is exactly one round-trip time before the first application byte. TLS 1.3 adds one more round trip and TLS 1.2 adds two, as Chapter 29 covers.
8. Observe with tcpdump -n 'tcp[tcpflags] & tcp-syn != 0'.

WORDS28.4.6 remember these#

1. Handshake — the short exchange that opens a connection — the SYN, SYN-ACK, ACK sequence defined in RFC 9293 section 3.5.
2. SYN — the flag meaning “let us start counting” — the synchronize control bit, which consumes one sequence number.
3. ACK — the flag meaning “I received up to here” — the acknowledgement control bit, making the acknowledgement number field valid.
4. Initial sequence number — the random starting count — the ISN, chosen per RFC 6528 as a timer plus a keyed hash of the four-tuple.

28.5 The TCP state machine and the four-way close#

PLAIN28.5.1 in simple words#

1. A TCP connection is not just a pipe. It is a small machine with a memory of where it currently is. That memory is called its state.
2. There are eleven states worth knowing, and a connection moves between them only when a specific packet arrives or the program does something.
3. Opening is fast: three packets and you are in ESTABLISHED, the state where data flows.
4. Closing is slower, because a TCP connection is really two one-way streams, and each direction must be closed separately.
5. So closing takes four packets, not three. One side says “I am done sending”. The other says “understood”. Then the same happens in the other direction.
6. After the last packet, the side that started the close sits in a state called TIME_WAIT for a while, doing nothing. That looks like waste. It is not: it catches late packets from the connection that just ended.

PLAIN28.5.2 a picture in your head#

1. Think of a phone call between two polite people who both hate hanging up first.
2. Aisha says “that is all from me”. Ben says “understood, noted”. Ben may still talk, and Aisha listens. The call is half-closed.
3. But Aisha does not put the phone down immediately. She holds the line for a couple of minutes, because if her final “understood” was not heard, Ben will repeat his goodbye, and somebody has to answer it.

Where this comparison breaks: real people can tell a repeated goodbye from a new call. A computer reusing the same four numbers cannot, which is the real reason for the wait. It is not politeness, it is identity. The next connection using the same four-tuple must not receive a stray packet meant for the last one.

PLAIN28.5.3 a worked example#

1. Here is the four-way close. The client closes first, so the client ends up waiting.

client (closes first)              server
  ESTABLISHED                        ESTABLISHED
        |                                  |
        |  FIN, ACK   seq = x              |
        |--------------------------------->|
   FIN_WAIT_1                         CLOSE_WAIT
        |                                  |
        |            ACK   ack = x + 1     |
        |<---------------------------------|
   FIN_WAIT_2                              |
        |     (server may still send data) |
        |            FIN, ACK  seq = y     |
        |<---------------------------------|
   TIME_WAIT                           LAST_ACK
        |                                  |
        |  ACK   ack = y + 1               |
        |--------------------------------->|
        |                               CLOSED
   wait 2 x MSL
        |
     CLOSED

2. Count the packets: FIN, ACK, FIN, ACK. Four. Often the middle two are combined into one FIN-ACK, when the server has nothing left to send.
3. 2 x MSL means twice the maximum segment lifetime. RFC 9293 sets MSL at 2 minutes, so the specified wait is 4 minutes.
4. Linux uses a fixed 60 seconds instead, compiled in as TCP_TIMEWAIT_LEN. That is an implementation detail, not a standard.

PLAIN28.5.4 what is really happening inside#

1. Here is the full state machine. Read it as: you are in a box, an event happens, you move along an arrow.

                          +--------+
              +---------->| CLOSED |<--------------+
              |           +--------+               |
              |            |      |                |
     app calls listen      |    app calls connect  |
     (passive open)        |    send SYN           |
              |            |      |                |
              v            |      v                |
        +--------+         |  +----------+         |
        | LISTEN |         |  | SYN_SENT |         |
        +--------+         |  +----------+         |
             |             |       |               |
      recv SYN             |   recv SYN,ACK        |
      send SYN,ACK         |   send ACK            |
             v             |       |               |
     +--------------+      |       |               |
     | SYN_RECEIVED |------+       |               |
     +--------------+  recv ACK    |               |
             |                     |               |
             +--------+   +--------+               |
                      v   v                        |
                +---------------+                  |
                |  ESTABLISHED  |                  |
                +---------------+                  |
                  |           |                    |
        app close |           | recv FIN           |
        send FIN  |           | send ACK           |
                  v           v                    |
          +------------+  +------------+           |
          | FIN_WAIT_1 |  | CLOSE_WAIT |           |
          +------------+  +------------+           |
            |        |          |                  |
   recv ACK |        | recv FIN | app close        |
            |        | send ACK | send FIN         |
            v        |          v                  |
   +------------+    |     +----------+            |
   | FIN_WAIT_2 |    |     | LAST_ACK |            |
   +------------+    |     +----------+            |
            |        v          |                  |
   recv FIN |   +---------+     | recv ACK         |
   send ACK |   | CLOSING |     +------------------+
            |   +---------+                        |
            |        | recv ACK                    |
            v        v                             |
        +-------------+                            |
        |  TIME_WAIT  |----------------------------+
        +-------------+   2 x MSL timer expires

2. SYN_SENT is a client that has sent its SYN and is waiting. This is exactly where the reader’s connection was stuck.
3. SYN_RECEIVED is the server’s half-open state between SYN-ACK and the final ACK. ESTABLISHED is the working state where everything useful happens.
4. FIN_WAIT_1 and FIN_WAIT_2 belong to the side that closed first, before and after its FIN is acknowledged. CLOSE_WAIT belongs to the side that was told first, and thousands of CLOSE_WAIT sockets is nearly always an application bug: the program never called close itself.

TECHNICAL28.5.5 the engineer’s version#

1. The eleven states are normative in RFC 9293 section 3.3.2. The names are the same strings printed by netstat and ss on every platform.
2. TIME_WAIT exists for two distinct reasons, and people usually remember only the first.
3. Reason one: to let any duplicate or delayed segment from this incarnation of the connection expire before the same four-tuple can be reused.
4. Reason two: to be able to retransmit the final ACK. If that ACK is lost, the peer in LAST_ACK resends its FIN and somebody must answer. Without TIME_WAIT the answer would be an RST, which the peer logs as an error.
5. Only the side performing the active close enters TIME_WAIT. Whoever calls close first pays the cost.
6. This is why servers accumulate TIME_WAIT. An HTTP server that closes each connection after responding, or a proxy that opens and closes short-lived back-end connections, is the active closer every time.

7. At 60 seconds of TIME_WAIT that caps sustained new connections at roughly 470 per second to a single destination. The fix is connection reuse, not tuning.
8. net.ipv4.tcp_tw_recycle was different, was unsafe behind NAT, and was removed from Linux in version 4.12, July 2017. Advice telling you to enable it is out of date and should be ignored.
9. Inspect with ss -tan state time-wait | wc -l on Linux, or netstat -an -p tcp | grep -c TIME_WAIT on macOS.

WORDS28.5.6 remember these#

1. State — where a connection currently is — one of the eleven values in the RFC 9293 state machine.
2. ESTABLISHED — the working state — both directions open, data may flow.
3. TIME_WAIT — the pause after hanging up — the 2 x MSL wait held by the active closer to absorb stray segments and resend the final ACK.
4. CLOSE_WAIT — waiting for your own program to hang up — the state after receiving a FIN, exited only when the application calls close.

28.6 Making TCP reliable and fast#

PLAIN28.6.1 in simple words#

1. TCP’s reliability rests on one habit: number everything, and confirm what you received.
2. Every byte in the stream has a number. Not every packet, every byte.
3. The receiver keeps saying “I have everything up to number N”. That message is the acknowledgement, or ACK.
4. The sender keeps a copy of everything sent but not yet confirmed, and if a confirmation does not arrive in time, sends it again.
5. Doing that one piece at a time would be safe but slow, so the sender is allowed to send many pieces before waiting for any confirmation. The amount allowed in flight at once is called the window, and as confirmations come in it slides forward, letting new data out.
6. The receiver controls the window size, so a fast sender cannot drown a slow receiver. That is flow control.

PLAIN28.6.2 a picture in your head#

1. Think of a conveyor belt between a warehouse and a shop.
2. The shop tells the warehouse how many crates it has room for. Say twelve. The warehouse may put up to twelve crates on the belt without hearing back.
3. Each crate is numbered. The shop calls out “received everything up to 7”, which frees the warehouse to put crates 13 to 19 on the belt.
4. If crate 5 falls off, the shop keeps calling “everything up to 4”, no matter how many later crates arrive, and the warehouse works out that 5 is missing.

Where this comparison breaks: the numbering in TCP counts bytes, not crates, so a crate’s number is the number of the first item inside it. Also the belt is not a single fixed-speed belt: the sender is guessing how fast the road can take traffic, which is section 28.7, and that guess is separate from the storeroom.

PLAIN28.6.3 a worked example#

1. Suppose the sender’s window is 5 segments of 1,460 bytes, starting at sequence number 1000, and segment 3 is lost.

sent:  seq 1000  1460 bytes   arrives
       seq 2460  1460 bytes   arrives
       seq 3920  1460 bytes   LOST
       seq 5380  1460 bytes   arrives
       seq 6840  1460 bytes   arrives

acks:  ack 2460                after first
       ack 3920                after second
       ack 3920   (duplicate)  after fourth
       ack 3920   (duplicate)  after fifth
       ack 3920   (duplicate)  after a sixth segment

2. The receiver cannot say “I have 5380”, because that would claim everything below it, and 3920 is missing. So it repeats 3920. This is a cumulative acknowledgement: all bytes below this number are safely here.
3. The sender sees the same acknowledgement three times over. Three duplicates is the agreed signal.
4. It immediately re-sends just the segment at 3920, without waiting for any timer. That is fast retransmit.
5. The extra mechanism is selective acknowledgement, SACK. The receiver adds a note saying “I also have the block from 5380 to 8300”.
6. With SACK the sender re-sends exactly one segment. Without it, up to three.

PLAIN28.6.4 what is really happening inside#

1. The sender keeps three markers over its send buffer: the oldest unconfirmed byte, the next byte to send, and the end of the window.
2. The waiting time before declaring a loss is measured, not fixed. The sender keeps a smoothed average of the round-trip time and a measure of how much that time jumps around, then sets the timeout to average plus four times the jumpiness.
3. On the receiving side, out-of-order segments are held in a reorder queue and handed to the application only once the gap is filled.
4. If the application is slow to read, that queue fills and the advertised window shrinks. At zero the sender stops sending data entirely.
5. The sender then sends a tiny window probe every so often, because if it simply waited for an update and that update were lost, both sides would wait forever.

TECHNICAL28.6.5 the engineer’s version#

1. The retransmission timeout is computed per RFC 6298, June 2011, by Paxson, Allman, Chu and Sargent, with alpha = 1/8, beta = 1/4 and K = 4.

first measurement R:
    SRTT   = R
    RTTVAR = R / 2

later measurements R':
    RTTVAR = (1 - beta) * RTTVAR + beta * |SRTT - R'|
    SRTT   = (1 - alpha) * SRTT  + alpha * R'
    RTO    = SRTT + max(G, K * RTTVAR)

then clamp:  RTO = max(RTO, 1 second)

2. Here is that running on a path of about 30 ms with one spike.

3. Every computed value is well under one second, so the RFC 6298 minimum of 1 second governs. On short paths the timer is almost always the floor, which is why fast retransmit matters: waiting for the RTO would cost thirty round trips.
4. Fast retransmit and fast recovery are in RFC 5681, September 2009. The trigger is three duplicate ACKs, chosen because one or two duplicates are commonly caused by simple reordering.
5. Selective acknowledgement is RFC 2018, October 1996, by Mathis, Mahdavi, Floyd and Romanow. It is negotiated in the SYN and carries three or four blocks per segment because of the 40-byte option limit.
6. The receive window field is 16 bits, capping the raw value at 65,535 bytes. Window scaling, RFC 7323, September 2014, obsoleting RFC 1323, adds a shift count of up to 14, giving a maximum window just under 1 GiB. The scale factor is exchanged only in the SYN and SYN-ACK; if either side omits it, no scaling is used for the whole connection.
7. Zero window handling: the sender must send a one-octet probe on the persist timer, backing off exponentially, so a lost window update cannot deadlock the connection.
8. Nagle’s algorithm is RFC 896, “Congestion Control in IP/TCP Internetworks”, John Nagle, 6 January 1984. Rule: if unacknowledged data is outstanding, buffer small writes until an ACK arrives or a full segment can be sent.
9. Delayed acknowledgement is in RFC 1122, October 1989. An ACK may be delayed by less than 500 ms, and there must be an ACK for at least every second full-sized segment.
10. The two together create a well-known stall. The sender holds a small final piece under Nagle. The receiver holds its ACK under delayed ACK. Neither moves until the delayed-ACK timer fires.
11. On Linux that timer is adaptive between about 40 ms and 200 ms, from TCP_DELACK_MIN and TCP_DELACK_MAX. A protocol that writes a header and a body in two separate calls can add 40 ms per exchange for no reason.
12. The fixes, in order of preference: write the whole message in one call or use writev; or set TCP_NODELAY to disable Nagle. Disabling Nagle is normal for request-response traffic and is the default in most RPC libraries.
13. Inspect with ss -tin on Linux, which prints rtt, rttvar, cwnd, ssthresh, retrans and the negotiated window scale for every socket.

WORDS28.6.6 remember these#

1. Sequence number — the count of bytes sent so far — a 32-bit field naming the first octet of this segment’s payload.
2. Acknowledgement — a receipt for what arrived — a cumulative ACK naming the next octet expected, implying everything below it arrived.
3. SACK — a note saying which pieces beyond the gap arrived — the selective acknowledgement option of RFC 2018.
4. Fast retransmit — resending early because of repeated receipts — resending the missing segment after three duplicate ACKs, per RFC 5681.
5. RTO — how long to wait before assuming loss — the retransmission timeout of RFC 6298, SRTT plus four times RTTVAR, floored at 1 second.

28.7 Congestion control in plain words#

PLAIN28.7.1 in simple words#

1. Flow control stops you drowning the person you are talking to. Congestion control stops you drowning the road between you. Mixing them up is one of the most common mistakes in this subject.
2. The receiver tells you its buffer size, so flow control is easy. Nobody tells you the capacity of the network, so the sender has to guess and keep re-guessing.
3. The guess is called the congestion window, a second limit on how much may be in flight, kept alongside the receiver’s window.
4. The strategy is simple: start small, increase while things go well, cut hard when a packet is lost.
5. Loss is the signal, because on a plain IP network loss is the only message the network sends about being full.
6. That works beautifully when loss means congestion, and terribly when loss means a bad radio link, because then TCP slows for no reason.

PLAIN28.7.2 a picture in your head#

1. Imagine driving into an unfamiliar city at night with no speed signs.
2. You start slowly. Nothing bad happens, so you go faster. At some point you scrape a kerb. You brake hard, then creep back up.
3. That is congestion control: millions of senders, no central authority, all probing for a limit and backing off when they find it.
4. Now imagine your car reports a scrape whenever you drive over a bump. You crawl through an empty city, convinced it is full. That is TCP on a lossy wireless link.

Where this comparison breaks: a driver can see the road ahead. TCP cannot see anything. It learns about the road only by driving into it and noticing what happened one round trip later. Every decision is made with information that is already old.

PLAIN28.7.3 a worked example#

1. Slow start is badly named. It is not slow; it is a doubling. Start with a congestion window of 10 segments, the modern default.

2. Doubling continues until a loss happens or the window passes the slow start threshold, after which it grows by about one segment per round trip. So the shape is rocket, then crawl, then a cut on loss, then crawl again.
3. Now the bandwidth-delay product: how much data must be in flight to keep a link busy, which is rate multiplied by round-trip time.

4. The 30 millisecond figure is the right order for the reader’s own path, which ended at Microsoft routers in Pune. The 250 millisecond figure is typical for India to the east coast of the United States. Both are approximate.
5. Without window scaling the maximum window is 65,535 bytes. Divide that by the round-trip time to get the ceiling on speed: about 17 megabits per second at 30 milliseconds, and about 2.1 megabits per second at 250 milliseconds.
6. A gigabit connection from India to a US server, without window scaling, delivers about two megabits per second. The link is not the problem. The window is.

PLAIN28.7.4 what is really happening inside#

1. The sender keeps two numbers: cwnd, the congestion window, and ssthresh, the slow start threshold.
2. While cwnd is below ssthresh, the sender is in slow start and cwnd grows by one segment per segment acknowledged, which doubles it per round trip.
3. Above ssthresh, the sender is in congestion avoidance and cwnd grows by about one segment per round trip, however many acknowledgements arrive.
4. When three duplicate acknowledgements arrive, the sender halves ssthresh, sets cwnd near the new ssthresh, retransmits, and continues. That is fast recovery, a mild reaction.
5. When the retransmission timer fires instead, the reaction is severe: cwnd drops all the way back to one segment and slow start begins again. That is why fast retransmit was such an important invention.

TECHNICAL28.7.5 the engineer’s version#

1. The problem was not theoretical. In October 1986 the link between Lawrence Berkeley Laboratory and the University of California, Berkeley, a distance of about 400 yards, collapsed from 32 kbit/s to 40 bit/s.
2. Van Jacobson diagnosed it and published “Congestion Avoidance and Control” at ACM SIGCOMM in August 1988, introducing slow start, congestion avoidance and the RTT variance estimator.
3. 4.3BSD-Tahoe, 1988, shipped slow start, congestion avoidance and fast retransmit. 4.3BSD-Reno, 1990, added fast recovery. The algorithm names come from the release names.
4. The current standard is RFC 5681, “TCP Congestion Control”, September 2009, with NewReno loss recovery in RFC 6582, April 2012. The initial window of 10 segments comes from RFC 6928, April 2013; before that it was 2 to 4.
5. CUBIC is the Linux default since kernel 2.6.19 in November 2006, from the 2008 work of Sangtae Ha, Injong Rhee and Lisong Xu. It is now standards track as RFC 9438, August 2023, obsoleting the informational RFC 8312.
6. CUBIC grows the window as a cubic function of the time since the last reduction, making it far more aggressive than Reno on high bandwidth-delay-product paths and roughly as polite on short ones.
7. BBR, from Google in 2016 by Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh and Van Jacobson, takes a different view: it estimates the bottleneck bandwidth and the minimum round-trip time and paces to that, rather than treating loss as the signal.
8. Established fact: BBR usually outperforms loss-based algorithms on paths with non-congestive loss and deep buffers. Active dispute: how fairly BBR shares capacity with CUBIC in shallow buffers. BBR version 3 was presented in 2023 and is being standardized in the IETF; it is a draft, not an RFC.
9. The Mathis equation, from the 1997 paper by Matthew Mathis, Jeffrey Semke, Jamshid Mahdavi and Teunis Ott, gives the throughput ceiling of a loss-based algorithm.

rate  <=  (MSS / RTT) * (C / sqrt(p))

MSS = 1460 bytes,  C = 1.22,  p = loss probability

10. Read the bottom-right cell. One in twenty packets lost on a long path gives you 260 kbit/s regardless of how fat the pipe is. This is why “the link is 1 Gbit/s” tells you almost nothing on its own.
11. Inspect and change with sysctl net.ipv4.tcp_congestion_control on Linux, ss -tin to see cwnd and ssthresh per socket, and sysctl net.inet.tcp.cc.available on macOS and FreeBSD.

WORDS28.7.6 remember these#

1. Congestion window — the sender’s guess at what the road can take — cwnd, a sender-side limit measured in segments or bytes.
2. Bandwidth-delay product — how much must be in the air to keep a pipe full — rate multiplied by round-trip time, in bytes.
3. CUBIC — today’s usual algorithm — the cubic-growth loss-based controller of RFC 9438, default on Linux since 2006.
4. BBR — the model-based alternative — a controller estimating bottleneck bandwidth and minimum RTT, from Google in 2016, still an IETF draft.

28.8 The three outcomes of a connection attempt#

PLAIN28.8.1 in simple words#

1. This section is the heart of the chapter. Everything before it was preparation.
2. When you try to open a TCP connection, exactly three things can happen. Not four. Three.
3. Success. The far side answers with SYN-ACK. You are connected, in about one round trip.
4. Refused. Something answers, but the answer is no. You get an error almost immediately, well under a second on any normal path.
5. Dropped. Nothing answers. Not a yes, not a no, not an error. Silence. Your machine tries again a few times and eventually gives up.
6. Here is the rule to carry with you: a refusal is a reply. A drop is not. So the speed of a failure tells you which kind it was, before you know anything else about the network.
7. Fast failure means something on the path is alive, received your packet, and decided against you. Slow silent failure means your packet, or the answer to it, disappeared without anyone saying so.
8. The reader’s failure was the slow silent kind. That one fact rules out a whole category of explanations, which is what section 28.9 does.

PLAIN28.8.2 a picture in your head#

1. You knock on a door.
2. Someone opens it and lets you in. That is success.
3. Someone shouts through the door “go away” and you leave immediately. That is refusal. Unpleasant, but fast, and it proves someone is home.
4. Nothing happens. You knock again. Nothing. Eventually you walk away. That is a drop.
5. Notice how much less you learned in case three. You do not know if the house is empty, if the bell is broken, if there is a wall you cannot see, or if someone inside is deliberately ignoring you.

Where this comparison breaks: with a real door you can look at the windows and check whether the lights are on. On a network you often cannot get a second opinion from the same vantage point, which is why testing from another network, as the reader did with mobile data, is worth more than any amount of retrying.

PLAIN28.8.3 a worked example#

   You send:  SYN ------------------------------>  ?

   (a) SUCCESS
       <-------------------------------- SYN, ACK
       result: connected in about one round trip

   (b) REFUSED
       <-------------------------------- RST, ACK
         or  <----------------- ICMP dest unreachable
       result: error returned in well under a second

   (c) DROPPED
       ...nothing...
       SYN ------------------------->  ...nothing...
       SYN ------------------------->  ...nothing...
       SYN ------------------------->  ...nothing...
       result: timeout after tens of seconds

1. Try this on your own machine to feel the difference in speed.

# (b) refused: nothing is listening on port 9 locally
nc -vz 127.0.0.1 9
# nc: connect to 127.0.0.1 port 9 failed: Connection refused
# returns instantly

# (c) dropped: a documentation address that goes nowhere
nc -vz -w 30 192.0.2.1 443
# nc: connect to 192.0.2.1 port 443 failed: Operation timed out
# returns only after the full wait

2. The two commands differ only in the address. The difference in how long they take is the entire diagnostic.

PLAIN28.8.4 what is really happening inside#

1. In the refused case, a machine received your SYN, found no socket listening on that port, and replied with a TCP segment carrying the RST flag.
2. RST means reset. It says: this connection does not exist and never will.
3. Your kernel receives that RST while in SYN_SENT, moves the socket to CLOSED, and returns the error ECONNREFUSED to your program at once.
4. A firewall can also produce a refusal, by sending an ICMP destination unreachable message, most often code 3 for port unreachable or code 13 for administratively prohibited.
5. In the dropped case, no such packet exists. Your kernel sits in SYN_SENT with a retransmission timer running and nothing to react to.
6. The timer fires, the SYN is sent again, and the timer is doubled. Then again, and doubled again. This is exponential backoff.
7. After a fixed number of attempts, or a fixed total time, the kernel gives up and returns ETIMEDOUT.

TECHNICAL28.8.5 the engineer’s version#

1. Errors as your program sees them: ECONNREFUSED for a refusal, ETIMEDOUT for a drop, EHOSTUNREACH or ENETUNREACH when ICMP or the local routing table says the destination cannot be reached at all.
2. RFC 9293 requires an RST in response to a SYN for a port with no listener, unless local policy says otherwise, with hardening rules in RFC 5961.
3. The relevant ICMP type is 3, Destination Unreachable, from RFC 792. Code 3 is port unreachable; codes 9, 10 and 13 are the administratively prohibited family added by RFC 1812, June 1995; code 4 is fragmentation needed.
4. Here is the Linux SYN retry schedule with the default net.ipv4.tcp_syn_retries of 6 and the RFC 6298 initial RTO of 1 second.

5. The Linux kernel documentation states this exactly: six retries corresponds to 63 seconds until the last retransmission, and a final timeout at 127 seconds.
6. The server side has its own counter, net.ipv4.tcp_synack_retries, default 5, giving 31 seconds to the last SYN-ACK retransmission and a final timeout at 63 seconds.
7. macOS and the BSDs use a total-time cap instead of a retry count. The sysctl is net.inet.tcp.keepinit, default 75000 milliseconds, so a connection attempt is abandoned after 75 seconds.

8. The Windows row is approximate, because the retransmission count and the 3 second initial timeout have changed across versions.
9. So a 5 second timeout in a log may be the application’s limit, not the kernel’s, and tells you less than a full kernel-length timeout does.
10. The mapping onto scanner vocabulary is exact: success is open, refusal is closed, and a drop is filtered. Section 28.12 returns to this.

WORDS28.8.6 remember these#

1. RST — the packet that means “no, and stop” — the TCP reset control bit, aborting a connection immediately.
2. Connection refused — someone answered no — the ECONNREFUSED error, caused by an RST or an ICMP port unreachable.
3. Connection timed out — nobody answered at all — the ETIMEDOUT error, caused by exhausting SYN retransmissions.
4. Exponential backoff — waiting twice as long after each failure — the doubling of the retransmission timer, 1, 2, 4, 8 seconds and onward.
5. Silent drop — a packet discarded with no notification — a filtering or forwarding decision that generates no ICMP and no RST.

28.9 What the reader’s timeout actually proves#

PLAIN28.9.1 in simple words#

1. Here is what the reader saw, exactly as recorded. curl -v https://github.com printed Trying 20.207.73.82:443..., then stopped for 15 seconds and failed with nothing coming back at all.
2. No refusal. No error message from any router. No reset. Silence.
3. Now let us extract every single thing that observation proves, and refuse to claim anything more.
4. One. DNS worked. The name github.com became the address 20.207.73.82, so the resolver at 1.1.1.1 was reachable and answered.
5. Two. Routing worked at least part of the way. The traceroute reached hop 12, ae106-0.rwa04.pnq20.ntwk.msn.net, at 104.44.20.42. Packets from this laptop were leaving the flat, crossing the ISP and entering Microsoft’s network in Pune.
6. Three. The TCP handshake never completed. The connection never reached ESTABLISHED. It stayed in SYN_SENT until it was abandoned.
7. Four. Nothing actively refused it. No RST came back. If one had, curl would have said “Connection refused” in a fraction of a second.
8. Five. No router complained either. No ICMP destination unreachable arrived. If one had, curl would have failed quickly with a different message.
9. Six. Therefore something swallowed a packet in silence. Either the SYN never reached a machine willing to answer, or the SYN-ACK it produced never made it home. One of those two packets died without a word.
10. That is the complete list. Everything else people usually say about a timeout is guessing.

PLAIN28.9.2 a picture in your head#

1. You post a letter that needs a signature on delivery. Three weeks later you have no signature and no returned envelope.
2. Compare this with getting the envelope back stamped “no such address”. That stamp would tell you a sorting office read your address and rejected it.
3. But silence is not useless. It definitively rules out every story that would have produced a stamp.

Where this comparison breaks: with post you could telephone the recipient and ask. The network equivalent is testing from a different vantage point, and the reader did exactly that with mobile data. That is why the mobile test was the most valuable single action in the whole session.

PLAIN28.9.3 a worked example#

1. Here is the evidence laid out as a ladder, from most certain to least.

2. Note the line about hop 12. The traceroute used ICMP or UDP probes, not TCP packets to port 443, and networks routinely treat those differently. A device can forward ICMP happily and drop TCP SYNs to port 443, or the reverse.
3. So “the trace reached hop 12” proves that some packets from this laptop reached Pune. It does not prove that this SYN did.
4. The session record notes several utun interfaces on the macOS machine, which are tunnel interfaces created by VPN software.
5. A stale route pointing 20.207.73.82 into a utun interface whose tunnel is not up produces precisely this symptom: a SYN that goes nowhere and a silent timeout.
6. That is not a claim that this happened. It is a candidate the collected evidence cannot rule out, and one command would have settled it.

PLAIN28.9.4 what is really happening inside#

1. Walk the SYN’s possible fates in order, from the laptop outward.
2. It could have been dropped by the local packet filter on macOS, or routed into a tunnel interface that leads nowhere because a VPN left a route behind.
3. It could have crossed the ISP’s access network and been dropped in the private core at hops 2 to 6, which are carrier addresses in the RFC 1918 ranges.
4. Or, and this is the one people forget, it could have arrived perfectly and produced a SYN-ACK that was dropped on the way back. The return path is not necessarily the mirror of the outbound path.
5. From the client’s chair, all of these look identical: silence for 15 seconds. That is the fundamental limitation, and no amount of retrying from the same chair fixes it.

TECHNICAL28.9.5 the engineer’s version#

1. First, an honest note about the 15 seconds. The macOS kernel default is net.inet.tcp.keepinit at 75000 milliseconds, so the kernel would have kept trying for 75 seconds.
2. A stop at 15 seconds therefore came from something above the kernel: a --connect-timeout or --max-time flag, a timeout wrapper, or a tool default. It does not change the conclusion, but the number should be attributed correctly.
3. Second, the candidate causes, with the evidence that would separate them.

4. Take the MTU case seriously, because it is the one that does not fit. A path MTU black hole occurs when a router must fragment, sends ICMP type 3 code 4 “fragmentation needed”, and that ICMP is filtered. Path MTU Discovery is RFC 1191, November 1990; the probing alternative is RFC 4821, March 2007.
5. But a SYN is about 60 bytes. It fits through any MTU on earth. An MTU black hole therefore produces a connection that completes and then hangs, usually at the TLS ClientHello or the first large response.
6. The reader’s symptom was the opposite: the connection never opened at all. So MTU is a poor fit here and should be ranked low. Naming it and then ruling it out is better practice than never mentioning it.
7. The single most informative missing command is a capture, which answers “did the SYN leave, and on which interface”.

sudo tcpdump -n -i any \
  'host 20.207.73.82 and tcp port 443'

# expected on a drop, repeated with growing gaps:
# 12:00:00.000  192.168.0.101.51514 > 20.207.73.82.443: [S]
# 12:00:01.001  192.168.0.101.51514 > 20.207.73.82.443: [S]
# 12:00:03.005  192.168.0.101.51514 > 20.207.73.82.443: [S]
# and nothing at all coming back

8. The second most informative is a TCP-based traceroute, because it probes with the same protocol and port that is failing.

sudo traceroute -T -p 443 20.207.73.82
sudo nmap -sS -Pn -p 443 --traceroute 20.207.73.82

9. Now the rigour about what is not proven, stated flatly.
10. The evidence does not identify where the drop happened. A silent drop leaves no marker anywhere in the client’s view.
11. The evidence does not prove intent. A drop can come from a deliberate policy, a misconfiguration, a failed line card, a full state table in a carrier-grade NAT, or a route withdrawn during a reconvergence.
12. The evidence does not name a responsible party, and this book will not name one. It is the reader’s ISP and it is Microsoft’s edge network, and neither fact establishes who dropped anything.
13. Nor does it establish that the condition was permanent. A single 15 second sample says nothing about five minutes later.
14. What the evidence does support, and supports strongly: a silent discard somewhere on that path in that moment, and not a server that was down.

WORDS28.9.6 remember these#

1. Black hole — a place packets go and never return — a forwarding state that discards traffic without generating ICMP.
2. Asymmetric routing — the way back is not the way there — a path where forward and reverse traffic traverse different links.
3. MTU black hole — big packets vanish, small ones do not — a path where fragmentation-needed ICMP is filtered, breaking Path MTU Discovery.
4. Vantage point — where you are standing when you measure — the network location from which an observation was taken, which bounds what it can prove.

28.10 Why “the server is down” was ruled out#

PLAIN28.10.1 in simple words#

1. The first thing most people say when a site does not load is “the site must be down”. Here that explanation is wrong, on two independent grounds.
2. Ground one: the failure had the wrong shape. A server that is down usually still answers something. It answers badly, but it answers.
3. If the web program has crashed but the machine is running, the machine’s network software receives your SYN, finds no listener, and sends back a reset. You get “Connection refused” in milliseconds.
4. If the machine itself is off, the last router on that network usually cannot find it and sends back a “host unreachable” message. Again, fast.
5. If the site is behind a load balancer whose machines are all unwell, the load balancer answers you. It completes the connection and sends an error page.
6. None of those look like fifteen seconds of nothing.
7. Ground two: it worked elsewhere at the same time. The same site loaded instantly on the reader’s phone over mobile data. Same destination, same moment, different path.
8. If the destination were down, the phone would have failed too. So the problem was not at the destination. It was on one particular path.

PLAIN28.10.2 a picture in your head#

1. You phone a shop and hear nothing at all. Not a ring, not an engaged tone, not a recorded message. Dead air for fifteen seconds.
2. A shop closed for the day still makes your phone ring. A shop out of business gives you “this number is not in service”, promptly.
3. Dead air is not a property of the shop. Dead air is a property of the line.
4. Then your friend, standing beside you, phones the same number on a different network and speaks to an assistant. Now you know. The shop is fine.

Where this comparison breaks: phone networks are far more consistent about generating tones than IP networks are about generating errors. Much of why network faults are hard is that the internet’s equivalent of the engaged tone is optional, and is frequently switched off.

PLAIN28.10.3 a worked example#

1. Here is the same user-visible failure under five different causes.

2. Only the last row matches what the reader saw.
3. A service the size of GitHub is fronted by many machines behind anycast addressing and load balancers. For every one of them to go silent at once, while a phone two feet away reached it instantly, is not a coherent story.

PLAIN28.10.4 what is really happening inside#

1. Consider a server whose web software has died. The kernel is still running, still owns the IP address, and still runs the TCP state machine.
2. Your SYN arrives, the kernel looks for a socket in LISTEN on port 443, finds none, and generates an RST. RFC 9293 requires that.
3. Now consider a load balancer with no healthy backends. It accepted your TCP connection before checking anything, completes TLS, reads your request, finds nowhere to send it, and returns 502 Bad Gateway or 503 Service Unavailable.
4. In every case some device took ownership of your packet and produced a response. Silence means no device took ownership, which is a different category of fault, living in the network rather than in the service.

TECHNICAL28.10.5 the engineer’s version#

1. The general principle: a service failure produces a response with the wrong content; a path failure produces no response. Distinguishing them is the first branch of any network diagnosis.
2. Independent-vantage testing is the strongest cheap evidence available. The mobile-data test is a control experiment: it changes exactly one variable, the network path, and holds the destination constant.
3. The honest version: the mobile test is slightly weaker than it looks. Mobile data may resolve github.com to a different address entirely, since large services return different answers to different resolvers and regions.
4. To make it airtight, pin the same address from the other network.

curl -v --resolve github.com:443:20.207.73.82 https://github.com
nc -vz -w 10 20.207.73.82 443

5. There is one real exception, where a healthy service does go silent. A large provider under a volumetric attack may drop SYNs from an entire source prefix rather than answer them, which to the client looks like a broken path.
6. So the correct conclusion is not “the network operator is at fault”. It is “the destination was serving, and packets on this path were being discarded without notification”. Only the second claim is supported.

WORDS28.10.6 remember these#

1. Service failure — the thing you wanted is broken but reachable — an error response generated by a machine that received your request.
2. Path failure — you cannot reach the thing at all — loss or misdelivery in the network, typically with no error generated.
3. Control experiment — change one thing and hold the rest fixed — testing the same destination and port from a second network path.

28.11 Reading curl -v line by line#

PLAIN28.11.1 in simple words#

1. curl fetches a URL. The -v flag makes it narrate what it is doing, naming each stage as it starts and finishes.
2. There are three kinds of line. A line beginning with * is curl talking about itself. Nothing on the network looks like that.
3. A line beginning with > is something curl sent to the server. A line beginning with < is something the server sent back, so the first < line is the first proof that the far side is really talking to you.
4. The reader’s run stopped after Trying 20.207.73.82:443..., the line curl prints just before it starts the TCP handshake. So the name lookup finished and the handshake never did. The output pinpoints the stage.

PLAIN28.11.2 a picture in your head#

1. Think of curl’s verbose output as a running commentary from a courier.
2. “Looking up the address.” “Driving there.” “Arrived at the gate.” “Showing my papers.” “Papers accepted.” “Handing over the request.” “Receiving a reply.”
3. If the commentary stops after “driving there”, the courier never arrived, and you stop worrying about the papers. If it stops after “papers accepted”, the route and the gate were fine and the problem is later.

Where this comparison breaks: a courier reports its own progress honestly. curl only knows what the operating system tells it, and the operating system only knows what came back. Neither can report on a packet that vanished outbound. The commentary stopping is an absence of news, not news of a failure.

PLAIN28.11.3 a worked example#

1. A successful run, trimmed, with the meaning of each line beside it.

*   Trying 20.207.73.82:443...      starting TCP handshake
* Connected to github.com port 443  handshake done, ESTABLISHED
* ALPN: curl offers h2,http/1.1     offering HTTP versions in TLS
* TLSv1.3 (OUT), Client hello (1)   first TLS message sent
* TLSv1.3 (IN), Server hello (2)    first TLS message received
*  subject: CN=github.com           the name on the certificate
*  issuer: C=US; O=DigiCert Inc     who signed it
> GET / HTTP/2                      request line sent
> Host: github.com                  request header sent
< HTTP/2 200                        first byte back from server
< content-type: text/html           response header received

2. Now the reader’s failing run, which is much shorter.

*   Trying 20.207.73.82:443...
* connect to 20.207.73.82 port 443 failed: Operation timed out
* Failed to connect to github.com port 443 after 15002 ms
* Closing connection 0
curl: (28) Failed to connect to github.com port 443

3. Two facts jump out. There is no Connected to line, so the handshake never finished. And there is no < line anywhere, so nothing came back at all.

4. Code 7 and code 28 are the refusal and the drop from section 28.8, wearing curl’s clothing.

PLAIN28.11.4 what is really happening inside#

1. curl -v does not measure time. For numbers you use -w, the write-out option, which prints variables after the transfer.

curl -sS -o /dev/null -w '
  dns         %{time_namelookup}
  tcp         %{time_connect}
  tls         %{time_appconnect}
  ttfb        %{time_starttransfer}
  total       %{time_total}
  address     %{remote_ip}:%{remote_port}
' https://github.com

2. Every time value is measured from the start of the request, so they are cumulative, not durations. You subtract to get each stage.

3. In the reader’s case time_namelookup would be small and every later field would read zero. A resolved address with a time_connect of zero is the machine-readable signature of the drop case.

TECHNICAL28.11.5 the engineer’s version#

1. --connect-timeout limits only the setup phase, which curl defines as DNS plus the TCP, TLS or QUIC handshakes. The libcurl default for CURLOPT_CONNECTTIMEOUT is 300 seconds.
2. --max-time limits the whole operation including the body. Use both: a short connect timeout and a longer overall one.
3. -4 forces IPv4 and -6 forces IPv6. This matters constantly on dual-stack networks, where a broken IPv6 path produces long stalls. The reader’s machine reported IPv6: (none), so -4 would have changed nothing here.
4. --resolve host:port:address overrides DNS for one name and port, letting you test one specific server without changing any system configuration.
5. Here are the exact commands the reader should have run.

# 1. is it TCP, or is it TLS or HTTP above it
nc -vz -w 5 20.207.73.82 443

# 2. is it this port, or this address
nc -vz -w 5 20.207.73.82 80
nc -vz -w 5 140.82.121.4 443

# 3. is a name problem hiding as a network problem
dig +short github.com @1.1.1.1
curl -v --resolve github.com:443:20.207.73.82 https://github.com

# 4. did the SYN leave the laptop, and by which interface
sudo tcpdump -n -i en0 'host 20.207.73.82 and tcp port 443'
route get 20.207.73.82

# 5. how far does a TCP SYN to 443 actually get
sudo nmap -sS -Pn -p 443 --reason --traceroute 20.207.73.82

# 6. timing signature, in one line
curl -sS -o /dev/null -m 20 \
  -w 'dns %{time_namelookup} tcp %{time_connect}\n' \
  https://github.com

6. Command 1 separates the transport from everything above it: if nc also hangs, the fault is at or below TCP. Command 2 distinguishes a port-specific filter from an address-specific one, for ten seconds of effort.
7. Command 4 is the only one that can prove the packet left the machine, which is the assumption every other test quietly makes. Command 5 probes with a SYN to port 443, so unlike an ICMP traceroute it measures the failing path.

WORDS28.11.6 remember these#

1. Write-out — print measured values after the transfer — curl’s -w option with variables such as time_connect.
2. Resolve override — pin a name to an address for one command — curl’s --resolve host:port:address.
3. Exit code — the number the command returns — curl’s documented failure codes, 7 for connect failure and 28 for timeout.

28.12 The other tools that answer this question#

PLAIN28.12.1 in simple words#

1. curl tests a whole stack at once: DNS, TCP, TLS and HTTP. When it fails you still have to work out which layer failed.
2. nc, usually called netcat, opens a bare TCP connection and reports whether it opened. Nothing above TCP is involved.
3. telnet did the same job for decades, and old documentation still recommends it. It works, but modern systems are removing it.
4. nmap tries many ports at once and classifies each. Its vocabulary maps exactly onto the three outcomes of section 28.8.
5. tcpdump shows the actual packets on the wire, and is the only tool here that can prove a packet left your machine.

PLAIN28.12.2 a picture in your head#

1. Think of a doctor with four instruments.
2. curl is asking the patient how they feel: rich information, hard to localize. nc is a single specific test: does this one joint bend.
3. nmap is a broad screen: check twenty things quickly and flag the odd ones.
4. tcpdump is the X-ray: it shows what is actually there rather than what anyone reports. You would not start with it, and you would not finish without it when the reports disagree.

Where this comparison breaks: an X-ray sees inside the patient. tcpdump only sees packets passing your own machine’s interfaces. It can prove your SYN left; it can never show you what a router 2,000 kilometres away did with it.

PLAIN28.12.3 a worked example#

# netcat, bare TCP connect, 5 second limit
nc -vz -w 5 20.207.73.82 443
# open:     Connection to 20.207.73.82 port 443 succeeded
# closed:   nc: connect to ... failed: Connection refused
# filtered: nc: connect to ... failed: Operation timed out

# nmap, with the reason printed
sudo nmap -sS -Pn -p 443 --reason 20.207.73.82
# PORT    STATE     SERVICE  REASON
# 443/tcp open      https    syn-ack ttl 52
# 443/tcp closed    https    reset ttl 52
# 443/tcp filtered  https    no-response

1. syn-ack means a handshake reply arrived. reset means an RST arrived. no-response means nothing arrived. That is section 28.8, word for word.

PLAIN28.12.4 what is really happening inside#

1. nc -z and telnet perform a normal connect call through the operating system, so they see exactly what any ordinary program would see.
2. nmap -sS builds raw packets itself, sends a SYN, reads the reply, and never completes the handshake. That is why it needs administrator rights and is called a half-open or stealth scan. nmap -sT uses connect instead.
3. tcpdump asks the kernel for a copy of packets at the interface, so it shows outbound packets even when nothing comes back. A capture showing three or four outgoing SYNs with growing gaps and nothing inbound is direct proof of the drop case.

TECHNICAL28.12.5 the engineer’s version#

1. Telnet is RFC 854, May 1983, by Jon Postel and Joyce Reynolds. Apple removed the telnet and ftp clients from macOS in version 10.13 High Sierra, September 2017. On Windows the Telnet Client has been an optional feature, off by default, since Windows Vista. It is unencrypted and must never be used for real remote login; as a reachability probe nc is better in every way.
2. Nmap’s authoritative state definitions: open means an application is actively accepting connections; closed means the port is reachable but no application is listening; filtered means packet filtering prevents probes from reaching the port, so nmap cannot determine the state.
3. Always pass -Pn. Without it, a host that ignores pings is recorded as down and never scanned at all, which yields a misleading empty result.

# only SYN packets, either direction, to one host
sudo tcpdump -n 'host 20.207.73.82 and tcp[tcpflags] & tcp-syn != 0'

# only resets, useful for spotting refusals
sudo tcpdump -n 'tcp[tcpflags] & tcp-rst != 0'

# all ICMP, for unreachables and MTU messages
sudo tcpdump -n icmp

# write a file for later analysis in Wireshark
sudo tcpdump -n -w capture.pcap 'host 20.207.73.82'

4. Complementary socket-state tools: ss -tan state syn-sent on Linux shows every connection stuck in exactly the reader’s state; on macOS use netstat -an -p tcp and look for SYN_SENT. lsof -i -P -n maps sockets to process names on both.

WORDS28.12.6 remember these#

1. netcat — a bare TCP connect tester — nc, opening a socket and reporting the result without any higher protocol.
2. Filtered — nmap’s word for silence — a port state where probes get no response, matching the dropped outcome.
3. Packet capture — recording the actual packets — tcpdump or Wireshark reading frames from an interface, selected with a filter such as tcp port 443.

28.13 Sockets from the programmer’s side#

PLAIN28.13.1 in simple words#

1. Everything in this chapter is reached from a program through about eight function calls, with the same names in almost every language.
2. socket creates an endpoint and returns a small number, a handle. bind attaches that endpoint to a local address and port. Servers do this; clients usually skip it and let the system choose.
3. listen turns a server endpoint into a door and sets how many callers may queue. accept takes one waiting connection off that queue and gives you a new handle for that one client.
4. connect is the client’s call. It sends the SYN and waits for the handshake. This is the call that took 15 seconds and failed for the reader.
5. send and recv move bytes. On a stream socket they may move fewer bytes than you asked for, and you must handle that.
6. close releases the handle and starts the four-way close. And separately you set a timeout, because without one your program inherits the operating system’s very long default.

PLAIN28.13.2 a picture in your head#

1. socket is renting a telephone. bind is being assigned an extension number.
2. listen is telling the switchboard to put callers through, with a queue of at most so many people on hold. accept is picking up the next caller from that queue; the main line stays free for the next one.
3. connect is dialling out. send and recv are talking and listening, and close is the polite mutual goodbye of section 28.5.

Where this comparison breaks: a phone conversation preserves the boundaries between sentences. A TCP stream does not. If you send three times, the other side may recv all of it in one go, or in seven pieces. Every real protocol therefore carries its own length prefix or delimiter.

PLAIN28.13.3 a worked example#

1. A minimal server, in C, that echoes back whatever it is sent.

int s = socket(AF_INET, SOCK_STREAM, 0);

struct sockaddr_in a;
memset(&a, 0, sizeof a);
a.sin_family      = AF_INET;
a.sin_addr.s_addr = htonl(INADDR_ANY);
a.sin_port        = htons(8080);

int one = 1;
setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);

bind(s, (struct sockaddr *)&a, sizeof a);
listen(s, 128);            /* 128 = backlog queue length */

for (;;) {
    int c = accept(s, NULL, NULL);   /* one client */
    char buf[1024];
    ssize_t n = recv(c, buf, sizeof buf, 0);
    if (n > 0) send(c, buf, (size_t)n, 0);
    close(c);              /* this side does the active close */
}

2. A client with a timeout, in Python, that reports which of the three outcomes happened.

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(5.0)          # applies to connect and to I/O

try:
    s.connect(("20.207.73.82", 443))
    print("SUCCESS: handshake completed")
except ConnectionRefusedError:
    print("REFUSED: an RST came back, something said no")
except socket.timeout:
    print("DROPPED: nothing came back at all")
except OSError as e:
    print("OTHER:", e)
finally:
    s.close()

3. Those three branches are exactly the three outcomes of section 28.8, in twelve lines of code.

PLAIN28.13.4 what is really happening inside#

1. socket allocates a kernel structure and a file descriptor, and bind writes a local address and port into it. Neither sends any packet.
2. listen moves the socket into LISTEN state and creates two queues: one for half-open connections awaiting the final ACK, one for completed connections awaiting accept, which hands one over as a new descriptor.
3. The number passed to listen is the backlog. If the completed queue is full, new connections are dropped or reset, and clients see a timeout or a refusal.
4. connect builds and sends the SYN, puts the socket in SYN_SENT, and sleeps until the SYN-ACK arrives, an RST arrives, or the timer runs out.
5. send copies your bytes into the kernel send buffer and returns without waiting, so a successful send proves nothing about delivery. recv copies out of the receive buffer and returns 0 exactly when the peer has sent a FIN.
6. close releases the descriptor and begins the FIN exchange. shutdown is the finer tool: it closes one direction while leaving the other open.

TECHNICAL28.13.5 the engineer’s version#

1. The Berkeley sockets interface first shipped in 4.2BSD in August 1983 and is standardized in POSIX.1-2001. That is why network code looks the same in C, Python, Go, Java and Rust.
2. Timeouts are the part people get wrong. SO_RCVTIMEO and SO_SNDTIMEO apply to data operations. On Linux they do not reliably bound connect.
3. The portable way to bound connect is to set the socket non-blocking, call connect, expect EINPROGRESS, wait on select or poll for writability with your own deadline, then read SO_ERROR for the result.
4. Linux additionally offers TCP_USER_TIMEOUT for established connections and TCP_SYNCNT to override tcp_syn_retries per socket, which is the clean way to make one connection give up in 5 seconds instead of 127.

5. A successful return from send means the data is in your kernel’s buffer. It does not mean it left the machine, arrived, or was read by the far end.
6. This is the same principle the reader met in git: a disconnect while reading a response tells you nothing about whether the write landed. The only way to know is to ask again.

WORDS28.13.6 remember these#

1. EINPROGRESS — “started, not finished” — the error returned by connect on a non-blocking socket, the basis of a custom connect timeout.
2. shutdown — close one direction only — the call that sends a FIN without releasing the descriptor.
3. TCP_NODELAY — send small writes at once — the socket option that disables Nagle’s algorithm, standard in request-response systems.

28.98 Common wrong ideas#

1. Wrong: a timeout means the server is down. Right: a timeout means nothing came back. A down server usually sends a reset or produces a fast error page. Silence points at the path, not at the service.
2. Wrong: if ping works, the site should work. Right: ping uses ICMP, not TCP. A device can forward ICMP and drop TCP port 443, and many do. ping failing also proves nothing, because ICMP is often blocked on purpose.
3. Wrong: TCP guarantees delivery. Right: TCP guarantees that data arrives complete and in order, or that the connection fails and you are told. It turns silent loss into correct delivery or a reported error. It cannot deliver across a cut cable.
4. Wrong: closed and filtered are the same thing. Right: closed means something replied with a reset, which proves the host is reachable. filtered means nothing replied, which proves almost nothing. They differ in evidence and in how long they take.
5. Wrong: UDP is just a worse TCP. Right: UDP is TCP with the reliability removed on purpose, so applications that cannot use late data are not forced to wait for it. Live voice, video calls, DNS and NTP all want it.
6. Wrong: thousands of TIME_WAIT sockets are a leak. Right: TIME_WAIT is normal for whoever closes first, and lasts 60 seconds on Linux. Thousands of CLOSE_WAIT sockets, by contrast, usually are a real application bug.
7. Wrong: * * * at the end of a traceroute proves the fault is at that hop. Right: many routers deprioritise or suppress ICMP time-exceeded replies. Silence at the end of a trace is the normal case, not a diagnosis.
8. Wrong: a successful send means the other side received the data. Right: it means the bytes are in your own kernel’s buffer. Delivery is confirmed only by an acknowledgement you cannot see, or by a reply from the application.
9. Wrong: raising the timeout will fix a connection that times out. Right: a longer timeout only makes you wait longer for the same silence. It helps only when the path is slow, not when it is discarding packets.

28.99 Chapter summary in 20 lines#

1. IP promises only to try. It gives no delivery, no ordering, no duplicate suppression, and no notification when a packet is lost.
2. TCP turns that into a reliable, ordered byte stream, and UDP is the layer that deliberately does not.
3. UDP is not a worse TCP. It is right whenever late data is worthless, such as live voice, live video, DNS queries and clock synchronization.
4. A port is a 16-bit number, 0 to 65,535, naming which program a packet is for.
5. A connection is identified by a four-tuple: source IP, source port, destination IP, destination port. That is why one server port serves millions.
6. A listening socket is matched on two values; a connected socket is matched on all four. Confusing them is the root of many bugs.
7. Port 443 is HTTPS, and it became the universal port because outbound 443 is nearly always permitted and its contents cannot easily be inspected.
8. Opening a connection costs a three-way handshake, SYN, SYN-ACK, ACK, and one full round trip before any application byte moves.
9. Initial sequence numbers are chosen unpredictably, per RFC 6528, because predictable ones allow blind injection attacks.
10. TCP is defined today by RFC 9293, STD 7, August 2022, which replaced the original RFC 793 of September 1981.
11. A connection walks eleven states. Closing takes four packets because each direction closes separately, and the active closer waits in TIME_WAIT.
12. Reliability comes from numbering every byte, cumulative acknowledgements, selective acknowledgement, retransmission timers, and fast retransmit on three duplicate acknowledgements.
13. Speed comes from the sliding window; the receiver’s window is flow control and the sender’s congestion window is congestion control. They are different.
14. Without window scaling, 65,535 bytes over a 250 millisecond path caps you at about 2 megabits per second regardless of link speed.
15. Loss-based congestion control treats every loss as congestion, so one percent loss on a long path cuts throughput to a fraction of a megabit per second.
16. A connection attempt has exactly three endings: SYN-ACK and success, RST or ICMP and a fast refusal, or silence and a slow timeout.
17. The diagnostic rule: fast failure means something answered; slow silent failure means something swallowed it.
18. The reader’s curl -v https://github.com printed Trying 20.207.73.82:443 and returned nothing for 15 seconds, proving DNS worked, some path existed as far as hop 12, the handshake never completed, and nothing refused it.
19. It does not prove where the drop happened, who caused it, or that it was deliberate. It does contradict “the server is down”, because the same site loaded instantly on mobile data at the same moment.
20. nc isolates the transport, nmap names the outcome as open, closed or filtered, and tcpdump is the only tool that can prove the SYN ever left the machine.