Packets and Routing

26.0 What this chapter gives you#

1. You will be able to explain why data is chopped into small pieces at all, instead of being sent as one long stream, and name the four separate reasons.
2. You will be able to draw a real packet, layer by layer, with byte counts, and work out how much of the wire is spent on wrapping rather than content.
3. You will be able to read every field of an IPv4 header, say what each one is for, and decode a real header from the reader’s laptop by hand.
4. You will be able to explain what happens when a packet is too big for a link, what IPv4 does, what IPv6 refuses to do, and why a filtered ICMP message can make a connection die in silence.
5. You will be able to list, in order, the eight things a router does to a packet, and say which addresses change at every hop and which never change.
6. You will be able to read a real routing table, apply longest prefix match to a real address, and explain the reader’s default route through 192.168.0.1.
7. You will be able to say how routers learn routes, from static lines typed by hand up to BGP, and explain what convergence is and why it is not instant.
8. You will be able to explain autonomous systems, transit, peering and internet exchanges, look up which network owns an address, and read AS8075 correctly.
9. You will be able to explain why BGP is about money and policy rather than distance, and describe three real incidents where it went badly wrong.
10. You will be able to explain why the reader can see twelve hops outward and still know nothing at all about the way back.

26.1 Why data is split up at all#

PLAIN26.1.1 in simple words#

1. When your computer sends something big, it does not send it as one lump.
2. It cuts the data into many small pieces and sends the pieces one after another.
3. Each piece is called a packet. A packet is a small parcel of data with a label on the front saying where it is going.
4. A typical packet on a home network carries about 1,500 bytes at most.
5. There are four reasons for doing it this way, and all four matter.
6. Reason one is fairness. A wire is shared. If one person sent a whole film in one unbroken lump, everyone else would wait until it finished.
7. Reason two is the cost of mistakes. If something goes wrong, you only resend the small piece that was damaged, not the whole file.
8. Reason three is memory. A machine in the middle only needs room to hold a few small pieces at a time, not the whole file.
9. Reason four is mixing. Small pieces from many different conversations can be interleaved on one wire, so many things happen at once.

PLAIN26.1.2 a picture in your head#

1. Think about how you would move a house full of furniture across a country.
2. You would pack it into boxes. Each box has a label with the new address.
3. The boxes travel separately. Some go on one lorry, some on another.
4. If one box is dropped, you replace the contents of that box only. The other boxes are unaffected.
5. Other people’s boxes ride on the same lorries at the same time. The lorry is never held for one customer.
6. This is exactly what packets do, and it is why the postal system is a much better picture of a network than a pipe is.
7. A pipe suggests a continuous stream that arrives in order, complete, with nothing else in it. None of that is true of a network.
8. Where this comparison breaks: real parcels are almost never lost or reordered, and the post office never sends two boxes by two different routes on purpose. A network does all three, routinely, every second.

PLAIN26.1.3 a worked example#

1. Suppose two people share one 100 megabit per second link.
2. Person A wants to send a one gigabyte file. Person B wants to load a web page of 50 kilobytes.
3. One gigabyte is 8,000,000,000 bits. At 100 megabits per second that takes 80 seconds to push onto the wire.
4. If the link carried whole files one at a time, person B would wait 80 seconds for a page that needs half a second of transfer.
5. With packets, the link carries a piece from A, then a piece from B, then a piece from A, and so on. Person B finishes in about a second.
6. Now the error case. Assume 1 packet in 1,000 is lost.
7. Sent as packets of 1,448 bytes each, the file is 690,608 packets, and one lost packet costs 1,448 bytes, which is about 0.116 milliseconds of link time.

PLAIN26.1.4 what is really happening inside#

1. There are two old ways to share a network, and they are worth naming clearly.
2. The first is circuit switching. Before you send anything, the network reserves a path from end to end and holds it for you until you hang up.
3. The classic telephone call worked this way. When you dialled, exchanges along the way set up a chain of connections, and that chain was yours alone.
4. A voice circuit was fixed at 64,000 bits per second. That capacity was yours whether you were talking or silent.
5. The bad part: silence is wasted. Two people on a call are silent about half the time, and that half of the capacity is thrown away.
6. The worse part: setup takes time, and if any link on the reserved path fails, the call drops. There is no way to slip round the break.
7. The second way is packet switching. Nothing is reserved. Every packet carries its own destination address and is dealt with on arrival.
8. A machine in the middle looks at the address, picks an outgoing link, and sends it. If the link is busy, the packet waits briefly in a queue.
9. Nothing about the earlier packets is remembered. Each packet is judged on its own. That property is called being stateless.
10. The bad part: delay is not steady. A packet may wait behind others. Under heavy load, packets are dropped rather than delayed forever.

TECHNICAL26.1.5 the engineer’s version#

1. Packet switching was invented twice, independently, in the early 1960s.
2. Leonard Kleinrock published queueing analysis of message flow in large communication networks at MIT in 1961, later expanded into his 1962 thesis.
3. Paul Baran at the RAND Corporation published the eleven-volume report On Distributed Communications in August 1964, describing what he called distributed adaptive message block switching, designed to survive damage.
4. Donald Davies at the National Physical Laboratory in the United Kingdom reached the same design in 1965 and gave it the name that stuck: packet.
5. The ARPANET, the first large working packet network, carried its first message on 29 October 1969 from UCLA to the Stanford Research Institute.
6. Vint Cerf and Robert Kahn published A Protocol for Packet Network Intercommunication in IEEE Transactions on Communications in May 1974, which is the direct ancestor of TCP and IP.
7. The ARPANET switched from the older NCP to TCP/IP on 1 January 1983, a single cut-over now remembered as flag day.
8. Serialization delay is the time to clock a frame onto the wire, and it is exactly frame bits divided by link rate.

9. Tools: ping -s varies packet size, iperf3 measures throughput, and tc qdisc on Linux shows the queue behaviour underneath.

WORDS26.1.6 remember these#

1. Packet — a small labelled parcel of data — a protocol data unit carrying a header and a payload, forwarded independently.
2. Circuit switching — reserve a path first, then talk — connection-oriented networking with resources committed for the call duration.
3. Packet switching — no reservation, address on every parcel — connectionless store-and-forward networking with per-packet forwarding decisions.
4. Multiplexing — many conversations sharing one wire — statistical multiplexing of independent flows onto a shared link.
5. Serialization delay — the time to push a packet onto the wire — frame size in bits divided by the link rate in bits per second.

26.2 What a packet actually is#

PLAIN26.2.1 in simple words#

1. Every packet has exactly two parts: a header and a payload.
2. The header is the label on the outside. It says where the packet is going, where it came from, and what kind of thing is inside.
3. The payload is the thing being carried. The network does not read it.
4. Here is the part that surprises people: the payload of one packet is usually another whole packet, with its own header.
5. The outermost label is used by the local wire. The next one in is used by the routers. The next one in is used by the two computers at the ends.
6. The next one in is used by the encryption layer. The innermost one is the thing you actually wanted, like a web request.
7. Putting a box inside a box is called encapsulation.
8. Taking the boxes apart again at the far end is called decapsulation.

PLAIN26.2.2 a picture in your head#

1. Think of a diplomatic pouch being sent between two countries.
2. Inside the outer canvas sack there is a sealed courier bag.
3. Inside the envelope there is a single sheet of paper with the message.
4. The airline handles the sack. It reads the sack’s tag and nothing else.
5. The courier company handles the bag. It reads the bag’s tag and nothing else.
6. The embassy opens the case. The ambassador opens the envelope. Only the ambassador reads the paper.
7. Where this comparison breaks: in a real network the outer wrapper is thrown away and rebuilt at every single hop, not carried end to end. The airline tag is replaced at every airport. That is section 26.5.

PLAIN26.2.3 a worked example#

1. Let us draw one real packet: the reader’s laptop sending an HTTPS request to github.com at 20.207.73.82.
2. Assume the request itself, the plain text a web server would read, is 512 bytes.

+--------------------------------------------------------+
| Ethernet frame       14 B header  +  4 B trailer       |
|  +--------------------------------------------------+  |
|  | IPv4 packet       20 B header                    |  |
|  |  +--------------------------------------------+  |  |
|  |  | TCP segment    32 B header                 |  |  |
|  |  |  +--------------------------------------+  |  |  |
|  |  |  | TLS record   5 B header + 17 B seal  |  |  |  |
|  |  |  |  +--------------------------------+  |  |  |  |
|  |  |  |  | HTTP request      512 B        |  |  |  |  |
|  |  |  |  +--------------------------------+  |  |  |  |
|  |  |  +--------------------------------------+  |  |  |
|  |  +--------------------------------------------+  |  |
|  +--------------------------------------------------+  |
+--------------------------------------------------------+

3. Now add it up from the inside out.

4. The TCP 32 bytes are the fixed 20 byte header plus 12 bytes of options, which on a modern machine are two padding bytes and a timestamp.
5. So 604 bytes go on the wire to deliver 512 useful bytes.
6. The overhead is 604 minus 512, which is 92 bytes, or 15.2 percent.

PLAIN26.2.4 what is really happening inside#

1. Encapsulation happens in a fixed order on the way out, top to bottom.
2. The browser writes the HTTP request bytes and hands them to the TLS layer.
3. TCP cuts the stream into segments no bigger than the agreed maximum, puts its header on each one, and hands each segment to IP.
4. IP puts its own 20 byte header on the front, with the source and destination addresses, and hands the packet to the network card driver.
5. The driver wraps it in an Ethernet or Wi-Fi frame with hardware addresses, and the card turns the bits into signals.
6. Decapsulation at the far end is the same list read backwards.
7. IP checks the destination is itself, strips the IP header, reads the protocol number, and hands the rest to TCP.
8. TCP checks the port number, strips its header, puts the bytes back in order, and hands a byte stream to TLS.

TECHNICAL26.2.5 the engineer’s version#

1. Ethernet II framing carries a 6 byte destination MAC, a 6 byte source MAC and a 2 byte EtherType, followed by payload and a 4 byte frame check sequence.
2. EtherType 0x0800 means IPv4, 0x86DD means IPv6 and 0x0806 means ARP. These are assigned by the IEEE Registration Authority.
3. The standard Ethernet payload maximum is 1,500 bytes, so the standard frame is 1,518 bytes including the check sequence, and the wire slot is 1,538.
4. On 802.11 Wi-Fi the picture differs: a data frame header is 24 to 30 bytes plus an 8 byte LLC/SNAP shim, so the same IP packet costs more.
5. The reader is on home broadband over a router at 192.168.0.1. If the laptop is on Wi-Fi, the first hop uses 802.11 framing and only the wired hops beyond the router use Ethernet framing. The IP packet inside is identical.
6. TLS 1.3, specified in RFC 8446 from August 2018, adds 5 bytes of record header, 1 byte of inner content type and a 16 byte AEAD tag per record.

7. Tools: tcpdump -e -x prints the frame header and the raw bytes; Wireshark shows the same nesting as a tree you can expand layer by layer.

WORDS26.2.6 remember these#

1. Header — the label on the front — the fixed-format control fields a protocol layer prepends to the data it is carrying.
2. Payload — the thing being carried — the opaque body of a protocol data unit, normally the entire packet of the layer above.
3. Encapsulation — putting a box inside a box — wrapping an upper-layer PDU as the payload of a lower-layer PDU on transmission.
4. Decapsulation — unwrapping — stripping a header and demultiplexing the payload to the correct upper-layer protocol on reception.
5. Frame — the outermost wrapper on one wire — a link-layer PDU with hardware addresses, valid only between two directly connected devices.

26.3 The IPv4 header field by field#

PLAIN26.3.1 in simple words#

1. The IP header is the only part of the packet that routers in the middle read.
2. It is 20 bytes long in almost every packet you will ever see.
3. The two most important fields are the source address and the destination address. Together they take 8 of the 20 bytes.
4. The next most important is a countdown number that stops packets circling forever if the routers get confused.
5. There is a number saying what is inside, so the receiver knows whether to hand the payload to TCP, to UDP, or to something else.
6. The rest is machinery for cutting a packet into smaller pieces, plus a couple of bits for marking traffic as more or less urgent.

PLAIN26.3.2 a picture in your head#

1. Picture a customs declaration form stuck to the outside of a parcel.
2. Line one says which version of the form this is, so the clerk knows the layout.
3. Line three says how urgent the parcel is and whether the depot was crowded.
4. Lines five to seven exist only for parcels that had to be split: a batch number, a flag saying whether more pieces follow, and where this piece sits.
5. Line eight is a stamp saying how many more depots this parcel may pass through before being thrown away.
6. Line nine says what is inside so the right department gets it.
7. Where this comparison breaks: a customs form is read once. An IP header is read, altered and re-stamped at every single depot, and the checksum has to be recomputed each time.

PLAIN26.3.3 a worked example#

1. Here is the IP header of the first packet the reader’s laptop sent towards github.com, the TCP SYN that opens the connection to port 443.
2. The laptop’s own address on the home network is not recorded in the session notes, so we write it as 192.168.0.14. Treat that one number as a stand-in. Every other value here is exactly what such a packet carries.

0000   45 00 00 40 4a 3b 40 00 40 06 d1 a5 c0 a8 00 0e
0010   14 cf 49 52

3. Read it byte by byte.

4. Total length 64 makes sense: a 20 byte IP header plus a 44 byte TCP header, because a SYN carries options and no data.
5. TTL 64 is the macOS and Linux default. Windows uses 128. Many routers use 255 for their own packets. These are conventions, not requirements.
6. When the reader’s router at 192.168.0.1 forwards this packet, TTL drops to 63 and the checksum becomes 0xd2a5.

PLAIN26.3.4 what is really happening inside#

1. The checksum is a one’s complement sum, and it is worth seeing once.
2. Split the header into ten 16 bit words, with the checksum field set to zero.
3. Add them all. Whenever the total goes past 16 bits, wrap the carry round and add it back at the bottom.
4. Flip every bit of the result. That is the checksum you store.
5. For the header above the running total is 0x2e5a, and flipping every bit gives 0xd1a5, which is exactly what is in the packet.
6. A receiver checks it by summing all ten words including the checksum. If the header is intact the answer is always 0xffff.
7. This checksum covers the header only, not the payload. TCP and UDP have their own checksums for the data.
8. That split is intentional. Routers change header fields at every hop, so the header checksum must be recomputed at every hop, and it is kept cheap.

TECHNICAL26.3.5 the engineer’s version#

1. The IPv4 header is defined in RFC 791, Internet Protocol, September 1981, and its layout has not changed since.

2. IHL is measured in 32 bit words, so the minimum legal value is 5, meaning 20 bytes, and the maximum is 15, meaning 60 bytes.
3. The 8 bits after version and IHL were originally a Type of Service byte. RFC 2474, December 1998, redefined the top 6 bits as the Differentiated Services Code Point, and RFC 3168, September 2001, took the bottom 2 for Explicit Congestion Notification.
4. Total Length is 16 bits, so an IPv4 datagram cannot exceed 65,535 bytes.
5. RFC 6864, February 2013, updated the Identification field: it is only meaningful when a datagram is actually fragmented, and senders that always set DF may leave it as anything, including zero.
6. Common Protocol numbers, assigned by IANA: 1 is ICMP, 6 is TCP, 17 is UDP, 41 is IPv6 encapsulation, 47 is GRE, 50 is ESP, 89 is OSPF, 132 is SCTP.
7. RFC 1812, Requirements for IP Version 4 Routers, June 1995, is the document that tells a router exactly how to treat each of these fields.
8. Tools: tcpdump -vvn -x host 20.207.73.82 prints the decoded fields and the raw hex; in Wireshark the display filter ip.ttl < 5 finds nearly expired packets, and ip.flags.df == 1 finds Do Not Fragment packets.

WORDS26.3.6 remember these#

1. TTL — a countdown that stops packets circling forever — time to live, an 8 bit hop budget decremented by every forwarding router.
2. DF bit — do not cut this packet up — the Don’t Fragment flag, which forces a router to drop rather than fragment an oversized packet.
3. Protocol number — a label saying what is inside — an 8 bit IANA-assigned value selecting the next-layer protocol.
4. Header checksum — a small sum that catches corruption — a 16 bit one’s complement checksum over the IPv4 header only.
5. DSCP — a priority marking — the 6 bit Differentiated Services Code Point from RFC 2474.

26.4 Fragmentation and MTU#

PLAIN26.4.1 in simple words#

1. Every link has a limit on how big a single packet may be.
2. That limit is called the MTU, short for maximum transmission unit.
3. A packet travels over many links in a row, and they need not all have the same limit.
4. So a packet can start out legal, travel happily for six hops, and then arrive at a link where it is too big.
5. There are only three things a router can do with a packet that is too big.
6. It can cut the packet into smaller pieces and send them separately.
7. It can throw the packet away and send a complaint back to the sender saying “too big, the limit here is this number”.
8. Or it can throw the packet away and say nothing at all.

PLAIN26.4.2 a picture in your head#

1. Imagine driving a lorry across a country with a series of low bridges.
2. Option one: the bridge keeper unloads your lorry into three smaller vans and sends them on. Slower, more paperwork, but the goods arrive.
3. Option two: the bridge keeper turns you back and hands you a note saying “maximum height here is two point five metres”. You repack and try again.
4. Option three: the bridge keeper waves you into a ditch and goes back to reading the newspaper. You are simply never seen again.
5. Where this comparison breaks: a lorry driver knows immediately that they hit a bridge. A packet sender has no way to know unless somebody tells it.

PLAIN26.4.3 a worked example#

1. Take a 1,500 byte IPv4 packet: 20 bytes of header and 1,480 bytes of data.
2. It reaches a link with an MTU of 1,400 bytes, and the DF bit is not set, so the router is allowed to cut it up.
3. Every fragment except the last must be a multiple of 8 bytes, because the offset field counts in 8 byte units.
4. The biggest data chunk that fits in 1,400 bytes is 1,380, and the biggest multiple of 8 not above 1,380 is 1,376.

5. Offset 172 means 172 times 8, which is 1,376 bytes into the original data. That is exactly where fragment 1 stopped.
6. 1,376 plus 104 is 1,480, which is the original data. Nothing is lost.

PLAIN26.4.4 what is really happening inside#

1. IPv4 lets any router on the path fragment a packet. IPv6 does not.
2. IPv6 made a deliberate decision: routers never fragment. If a packet is too big, the router drops it and sends back a “packet too big” message.
3. Modern IPv4 senders behave the same way voluntarily. They set the DF bit on every packet, which forbids any router from cutting it up.
4. That means the sender must find out for itself how big it may go. The method is called path MTU discovery.
5. It works like this. Send a full-size packet with DF set. If a link on the way cannot take it, that router drops it and sends back an ICMP message saying “fragmentation needed” together with the MTU it can accept.
6. The sender reads that number, shrinks its packets to fit, and retransmits.
7. The whole scheme depends on one thing: the ICMP message must get back.
8. When that happens, the big packets vanish and no message comes back. The sender keeps retransmitting the same too-big packet forever.
9. This is called a path MTU black hole, and it produces a very particular symptom: small things work, big things hang.

TECHNICAL26.4.5 the engineer’s version#

1. Path MTU discovery for IPv4 is RFC 1191, November 1990. For IPv6 it is RFC 8201, July 2017, which obsoleted RFC 1981.
2. The IPv4 signal is ICMP type 3, code 4, Destination Unreachable with Fragmentation Needed and DF set. The next-hop MTU is carried in the message.
3. The IPv6 signal is ICMPv6 type 2, Packet Too Big. IPv6 forbids router fragmentation entirely, by RFC 8200, so the sender must react or fail.
4. IPv6 sets a hard floor: every link must support at least 1,280 bytes. IPv4’s equivalent floor is 68 bytes to forward and 576 bytes to reassemble.

5. Because ICMP filtering is so common, the IETF specified a method that does not need ICMP at all: Packetization Layer Path MTU Discovery, RFC 4821, March 2007, extended to datagram transports in RFC 8899, September 2020.
6. A quick manual probe on macOS, which sends a 1,472 byte payload plus 8 bytes of ICMP header plus 20 bytes of IP header, exactly 1,500 bytes:

ping -D -s 1472 20.207.73.82
ping -D -s 1400 20.207.73.82
ping -D -s 1200 20.207.73.82

7. If the 1,472 probe fails and the 1,200 probe succeeds, you have found an MTU restriction on the path. On Linux the flag is -M do instead of -D.
8. Tunnels are the usual cause, and the reader’s machine showed several utun interfaces, which on macOS are tunnel devices. A tunnel always reduces the usable MTU.
9. Now the honest part, because it matters. A path MTU black hole is a genuine and common cause of exactly the kind of silent failure the reader saw, and it belongs on the candidate list.
10. But the evidence here points elsewhere. curl printed Trying 20.207.73.82:443... and then nothing for 15 seconds. curl prints Connected to only after the TCP handshake completes, and it never did.
11. So the packet that got no answer was the SYN, which is 64 bytes. An MTU problem does not stop a 64 byte packet.
12. What is PROVEN: the first small packet of the connection received no reply of any kind. What is merely SUGGESTED: something on that path silently discarded it.
13. The classic MTU black hole signature is the opposite shape: handshake succeeds, small responses arrive, and the connection stalls on the first full-size packet. That is not what happened.

WORDS26.4.6 remember these#

1. MTU — the biggest packet a link will carry — maximum transmission unit in bytes, 1,500 for standard Ethernet.
2. Path MTU — the smallest limit anywhere along the route — the minimum MTU over all links on a given path.
3. Fragmentation — cutting a packet into smaller ones — splitting an IPv4 datagram across multiple packets sharing an identification value.
4. Path MTU discovery — asking the network how big you may go — RFC 1191 for IPv4 and RFC 8201 for IPv6, using ICMP too-big messages.
5. Black hole — packets vanish and nobody says why — a path where oversized packets are dropped and the ICMP notification is filtered.

26.5 What a router actually does, mechanically#

PLAIN26.5.1 in simple words#

1. A router is a machine with two or more network connections that passes packets from one to another.
2. Step one: a frame arrives on a wire and the router checks it is not damaged.
3. Step two: it throws away the outer wrapper, the one with hardware addresses, because that wrapper was only ever for that one wire.
4. Step three: it reads the destination IP address inside.
5. Step four: it looks that address up in its routing table to decide which way out to send it, and to whom.
6. Step five: it subtracts one from the hop counter. If the counter hits zero, the packet is destroyed and a complaint is sent back.
7. Step six: because it changed the hop counter, it fixes the small checksum.
8. Step seven: it finds the hardware address of the next machine along.
9. Step eight: it builds a brand new outer wrapper with that hardware address and pushes the packet out of the chosen wire.
10. The single most important thing to hold on to: the outer wrapper is destroyed and rebuilt at every hop, but the address inside stays the same.

PLAIN26.5.2 a picture in your head#

1. Imagine a parcel travelling by a chain of couriers, each covering one leg.
2. The parcel has the final address written on it in permanent ink. Nobody ever changes that.
3. Each courier puts the parcel in their own bag with their own routing sticker: “from depot 4, to depot 9”.
4. When it arrives at depot 9, that bag is thrown in the bin. The sticker was only ever about that one leg.
5. Depot 9 reads the permanent address, decides depot 14 is next, puts the parcel in a fresh bag with a fresh sticker, and sends it on.
6. Where this comparison breaks: real couriers know the whole route. A router knows only the next depot. It has no idea how many more legs remain.

PLAIN26.5.3 a worked example#

1. Follow the reader’s first packet across three real hops from the traceroute.

laptop           192.168.0.1        172.31.0.17     137.97.29.249
   |                  |                   |                |
   |== frame A ======>|                   |                |
   |   MAC: me->rtr   |== frame B =======>|                |
   |                  |   MAC: rtr->isp   |== frame C ====>|
   |                  |                   |  MAC: isp->hop3|
   |                  |                   |                |
 TTL 64            TTL 63              TTL 62           TTL 61

2. Three different frames carry one packet. Each frame lives for exactly one link and is then discarded.
3. The MAC addresses below are illustrative, since the session notes do not record them, but the pattern is exact.

4. Now the same journey seen from the IP header.

5. Notice the checksum column. Each hop adds 0x0100, because the TTL is the high byte of a 16 bit word and it went down by one.
6. The honest version: the source IP address is not really unchanged. The reader’s home router does network address translation, so it rewrites the source from 192.168.0.14 to its own public-facing address.

PLAIN26.5.4 what is really happening inside#

1. Step seven above, finding the hardware address of the next machine, deserves its own explanation, because it is where two different address systems meet.
2. The routing table gives the router an IP address for the next hop, not a hardware address.
3. To build the new frame it needs a hardware address. It gets one by asking on the local wire: “who has this IP address, tell me your hardware address”.
4. That question and answer is the Address Resolution Protocol. The answers are kept in a small table so the question is only asked occasionally.
5. If the next hop is not a router at all but the final destination on the same wire, the same lookup happens for the destination address itself.
6. Now the speed problem. Doing a full table lookup in software for every packet is far too slow for a serious router.
7. So real routers split in two. The control plane works out what the routes should be, slowly, in software.
8. The data plane forwards packets, fast, using a stripped-down copy of the answers built into dedicated hardware.

TECHNICAL26.5.5 the engineer’s version#

1. The full forwarding path for an IPv4 unicast packet, per RFC 1812:

1) receive frame, verify FCS, check dst MAC is ours
2) demultiplex on EtherType 0x0800 -> IPv4
3) verify IHL, total length and header checksum
4) if TTL <= 1: drop, send ICMP time exceeded
5) longest prefix match on destination in the FIB
6) decrement TTL, update header checksum incrementally
7) resolve next-hop IP to MAC via ARP cache or request
8) encapsulate in a new L2 header, enqueue on egress

2. Those eight steps are the forwarding plane. Nothing in the list consults any state about previous packets, which is what makes IP stateless.
3. The routing table is the Routing Information Base, the RIB. The hardware copy is the Forwarding Information Base, the FIB.
4. The RIB holds every route learned from every source, including losers. The FIB holds only the winner for each prefix, in a form the silicon can search.
5. FIB lookup in hardware is typically a TCAM, ternary content addressable memory, which compares a key against every stored prefix in one clock cycle.
6. TCAM is expensive and power hungry, which is why routers have hard limits on route counts. A switch sold as supporting 16,000 IPv4 routes cannot hold a full internet table.
7. As of the CIDR Report for 13 August 2026, the global IPv4 table held about 1,071,273 prefixes originated by 79,217 autonomous systems.
8. Address resolution is ARP for IPv4, RFC 826 from November 1982. IPv6 uses Neighbor Discovery instead, RFC 4861.
9. Tools: arp -a on macOS lists the resolved neighbours; netstat -rn shows the routing table; ip route get 20.207.73.82 on Linux shows the exact decision for one destination.

WORDS26.5.6 remember these#

1. Hop — one link between two routers — a single layer 3 forwarding step, which decrements TTL by one.
2. Next hop — who to hand the packet to next — the IP address of the adjacent router selected by the routing lookup.
3. Control plane — the part that decides routes — routing protocol software building the RIB.
4. Data plane — the part that moves packets — hardware forwarding using the FIB.
5. RIB — the full list of everything learned — Routing Information Base, holding all candidate routes.
6. FIB — the short list actually used — Forwarding Information Base, one best next hop per prefix, held in fast memory.

26.6 The routing table#

PLAIN26.6.1 in simple words#

1. A routing table is a list of rules of the form “packets going here should be sent that way”.
2. Each rule has a destination pattern, a way out, and sometimes a next machine to hand the packet to.
3. The destination pattern is not one address. It is a whole block of addresses, written as an address and a slash number.
4. 192.168.0.0/24 means “any address whose first 24 bits match”, which is the 256 addresses from 192.168.0.0 to 192.168.0.255.
5. Several rules can match the same address at once. That is normal and deliberate.
6. When several match, the winner is the one with the biggest slash number, the most specific one. That rule is called longest prefix match.
7. There is almost always one rule with a slash number of zero, written 0.0.0.0/0. Zero bits must match, so it matches everything.
8. That is the default route, meaning “if nothing else matched, send it here”. On the reader’s laptop it points at 192.168.0.1.

PLAIN26.6.2 a picture in your head#

1. Think of the signs in a large airport.
2. One sign says “All departures, this way”. That is the default route.
3. Further on, a sign says “International departures, this way”. More specific, so it overrides the general one.
4. Further on again, “Flights to Mumbai, gate 34”. More specific still.
5. All four signs point at your flight. You obey the most specific one that applies to you, and you ignore the vaguer ones.
6. Where this comparison breaks: an airport arranges the signs so you meet the general one first and the specific one later. A router sees all its rules at once and picks the most specific in a single step.

PLAIN26.6.3 a worked example#

1. Take a router with these five rules, and a packet for 20.207.73.82, which is the address the reader’s github.com resolved to.

2. All five match. Now check the bits and see why.

20.207.73.82  = 00010100 11001111 01001001 01010010

/0    (nothing to compare)            match
/8    00010100                        match
/10   00010100 11                     match
/18   00010100 11001111 01            match
/24   00010100 11001111 01001001      match

3. Take the third one slowly. 20.192.0.0/10 compares the first 10 bits.
4. The route’s second byte is 192, which is 11000000. Its first two bits are 11.
5. The address’s second byte is 207, which is 11001111. Its first two bits are also 11. So the first 10 bits agree.
6. Now a rule that does not match, for contrast: 20.207.128.0/18. The third byte 128 is 10000000, whose first two bits are 10, and the address’s third byte 73 is 01001001, whose first two bits are 01. No match.
7. Winner: 20.207.73.0/24, because 24 is the largest number of matching bits. The packet goes to 10.9.0.13.
8. The default route loses every time anything else matches, which is exactly what “default” means.

PLAIN26.6.4 what is really happening inside#

1. Routes get into the table in four different ways, and the difference matters.
2. Connected routes appear automatically. Give an interface the address 192.168.0.14/24 and the machine knows 192.168.0.0/24 is reachable directly on that interface, with no next hop at all.
3. Static routes are typed in by a person or set by a script. They stay until someone removes them.
4. Dynamic routes are learned from a routing protocol talking to other routers. Section 26.7 covers those.
5. When two sources offer a route to the very same prefix, longest prefix match cannot break the tie, because the prefixes are identical.
6. Two more tie-breakers exist, in this order.
7. First, administrative distance: a trust score per source. A lower number means “trust this source more”. Connected beats static beats OSPF beats RIP.
8. Second, metric: within one protocol, the cheaper route wins. What “cheaper” means depends entirely on the protocol.

TECHNICAL26.6.5 the engineer’s version#

1. Here is a realistic netstat -rn from the reader’s macOS machine. Only the IPv4 section is shown, because the machine reported IPv6: (none).

Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.0.1        UGScg        en0
127                127.0.0.1          UCS          lo0
127.0.0.1          127.0.0.1          UH           lo0
169.254            link#12            UCS          en0      !
192.168.0          link#12            UCS          en0      !
192.168.0.1/32     link#12            UCS          en0      !
192.168.0.1        ac:22:0b:4e:1f:2a  UHLWIir      en0    978
192.168.0.14/32    link#12            UCS          en0      !
192.168.0.255      ff:ff:ff:ff:ff:ff  UHLWbI       en0      !
224.0.0/4          link#12            UmCS         en0      !
255.255.255.255/32 link#12            UCS          en0      !

2. Line by line:
   1. default 192.168.0.1 is 0.0.0.0/0 via the home router. Every packet for 20.207.73.82 used this line.
   2. 127 and 127.0.0.1 are loopback, so the machine can talk to itself without touching any wire.
   3. 169.254 is the link-local block from RFC 3927, used when DHCP fails.
   4. 192.168.0 is the connected route for the home network, reachable directly on en0 with no gateway.
   5. 192.168.0.1/32 and 192.168.0.14/32 are host routes macOS creates for the gateway and for the machine’s own address.
   6. 192.168.0.1 ac:22:0b:4e:1f:2a is the resolved ARP entry for the router, with 978 seconds left before it expires.
   7. 192.168.0.255 is the broadcast address for the subnet.
   8. 224.0.0/4 is the multicast block.
   9. 255.255.255.255/32 is the all-hosts broadcast address.
3. The Flags column is a set of letters, documented in the macOS netstat manual page.

4. So UGScg on the default route reads: usable, via a gateway, static, protocol-cloning, global. The ! on some lines marks a reject route for addresses inside the block with no more specific entry.
5. Administrative distance is a Cisco convention, not an internet standard. Other vendors use the term route preference with different numbers. Juniper’s defaults, for instance, differ from the list below.

6. Tools: netstat -rn on macOS and BSD, ip route show on Linux, route print on Windows, show ip route on Cisco IOS, and route get 20.207.73.82 on macOS to see exactly which line wins.

WORDS26.6.6 remember these#

1. Prefix — a block of addresses written with a slash — an address plus a mask length, such as 20.192.0.0/10.
2. Longest prefix match — the most specific sign wins — selecting the matching route with the greatest mask length.
3. Default route — where to send anything unrecognized — 0.0.0.0/0, matching every destination with zero bits of agreement required.
4. Connected route — the network you are plugged into — a route derived automatically from an interface address and mask.
5. Static route — a line typed by hand — an administratively configured route with no protocol maintaining it.

26.7 How routers learn routes#

PLAIN26.7.1 in simple words#

1. A small network can have its routes typed in by hand. A large one cannot.
2. So routers talk to each other and tell each other what they can reach.
3. There are two families of ways to do this, and they think very differently.
4. The first family is distance vector. Each router tells its neighbours a list: “I can reach these places, and here is how far away each one is”.
5. A router receiving that list adds the cost of the link it came over and compares the result with what it already knew.
6. The second family is link state. Every router describes only its own direct connections, and that description is flooded to every other router.
7. Every router therefore ends up with an identical picture of the whole network, and computes the best paths itself.
8. Distance vector is “ask a neighbour for directions”. Link state is “everyone gets the same map and works it out”.

PLAIN26.7.2 a picture in your head#

1. Imagine a village with no maps, where you find your way by asking people.
2. Distance vector is asking your neighbour, “how far is the market?” They say “twelve minutes that way”. You add the two minutes to reach them and tell the next person “fourteen minutes”.
3. This works, but nobody can check anything. If your neighbour is wrong, everyone downstream of you is wrong too, and confidently so.
4. Worse: if the market closes, your neighbour hears from you that the market is fourteen minutes away, and tells you it is sixteen, and you tell them eighteen. The number climbs forever while the market stays shut.
5. Link state is different. Every person writes down only what they can see from their own doorstep, and copies of every note are posted to everybody.
6. Now everyone holds the same stack of notes, draws the same map, and works out their own shortest path.
7. Where this comparison breaks: real people would not re-post the entire note stack every thirty minutes, but link state protocols do refresh, and the flooding costs real bandwidth on very large networks.

PLAIN26.7.3 a worked example#

1. Four routers, A, B, C and D, in a line, with A connected to a network N.

   N --- A --- B --- C --- D

2. With RIP, which counts hops, the table for network N fills in over time. A says N = 0 throughout, because N is connected to it.

3. Each round is one update interval. RIP’s default update interval is 30 seconds, so full knowledge of a four-router line takes about a minute and a half.
4. Now break the link between A and B. A knows immediately. B does not.
5. Without protection, B hears from C that N is reachable at distance 2, adds one, and announces N at distance 3. C then hears 3 and announces 4.
6. The count climbs: 3, 4, 5, and onward, one per update round, until it reaches 16, which RIP defines as unreachable. That takes many rounds.
7. This is the famous count to infinity problem, and it is why RIP declares 16 to mean unreachable rather than allowing large numbers.

PLAIN26.7.4 what is really happening inside#

1. Link state protocols need a way to turn a map into best paths, and they all use the same method: Dijkstra’s algorithm.
2. Here it is in plain words, with no mathematics.
3. Write down every router. Give yourself a cost of 0 and everyone else a cost of infinity. Mark nobody as finished.
4. Pick the unfinished router with the smallest cost. The first time round that is you.
5. Look at each of its neighbours. For each one, work out the cost of reaching it through the router you just picked.
6. If that total beats the neighbour’s current cost, write down the better total and remember which router you came from.
7. Mark the picked router finished. It can never improve, because every other route to it would have to go through something more expensive.
8. Repeat: pick the smallest unfinished one, relax its neighbours, mark it finished.
9. Stop when everyone is finished. Following the “came from” notes backwards gives the shortest path to every destination.
10. The output is a tree rooted at you, called the shortest path tree, and the first step of each branch is the next hop you put in the routing table.
11. Convergence is the time from a change happening to every router having finished redoing this and agreeing again.

TECHNICAL26.7.5 the engineer’s version#

1. Interior gateway protocols run inside one administrative domain. Exterior gateway protocols run between domains. Sections 26.8 and 26.9 cover the exterior side.

2. RIP runs over UDP port 520, updates every 30 seconds, times a route out after 180 seconds, and treats metric 16 as infinity. RIPv2 added subnet masks and uses multicast address 224.0.0.9.
3. OSPFv2 was specified by John Moy. Version 1 appeared as RFC 1131 in 1989 and the current version 2 as RFC 2328 in April 1998. OSPFv3 for IPv6 is RFC 5340.
4. OSPF runs directly over IP as protocol number 89. On a broadcast network it sends hellos every 10 seconds and declares a neighbour dead after 40 seconds.
5. OSPF divides a large network into areas, all joined to area 0, the backbone. Link state flooding is confined within an area, which bounds both the database size and the computation.
6. IS-IS comes from the OSI world, standardized as ISO/IEC 10589, and was adapted to carry IP by RFC 1195 in December 1990. Radia Perlman designed it at Digital Equipment Corporation for DECnet Phase V.
7. IS-IS runs directly on the link layer rather than inside IP, which is one reason very large carriers favour it: it is harder to attack from an IP packet, and it carries IPv4 and IPv6 in one protocol without change.
8. Dijkstra’s algorithm was conceived by Edsger Dijkstra in 1956 and published in Numerische Mathematik in 1959. Its cost with a binary heap is on the order of E log V, where E is links and V is routers.

9. Tools: show ip ospf neighbor and show isis adjacency on router command lines, and birdc show route or vtysh for the open source stacks BIRD and FRRouting.

WORDS26.7.6 remember these#

1. Distance vector — ask a neighbour how far it is — a protocol where routers exchange destination and metric pairs, without topology knowledge.
2. Link state — everyone gets the same map — a protocol where routers flood descriptions of their own links and each computes paths independently.
3. Count to infinity — a distance that climbs while the destination is gone — a distance vector loop where metrics increment until reaching the infinity value.
4. Split horizon — do not tell a neighbour what it told you — suppressing advertisement of a route out of the interface it was learned on.
5. Dijkstra’s algorithm — repeatedly finish the cheapest unfinished node — a shortest path first algorithm, published 1959, used by OSPF and IS-IS.
6. Convergence — everyone agreeing again after a change — the interval from a topology change to a consistent forwarding state network-wide.

26.8 Autonomous systems#

PLAIN26.8.1 in simple words#

1. The internet is not one network. It is about eighty thousand separate networks that have agreed to carry each other’s traffic.
2. Each of those separate networks is called an autonomous system, usually shortened to AS.
3. Autonomous means it decides its own internal routing for itself. Nobody outside gets a say in how it moves packets inside its own borders.
4. Each AS has a number. Microsoft’s main one is AS8075. That number is how networks refer to each other.
5. The reader’s packets started in the ISP’s autonomous system and ended up in Microsoft’s, which is why the names at hop 7 change to ntwk.msn.net.
6. Networks connect to each other in two very different commercial ways.
7. Transit is buying. You pay a bigger network to carry your traffic to the whole internet and to bring the whole internet back to you.
8. Peering is swapping. Two networks connect directly and exchange traffic between their own customers, usually with no money changing hands.

PLAIN26.8.2 a picture in your head#

1. Think of the world’s postal services.
2. Each country runs its own post inside its borders however it likes. Different vans, different sorting rules, different rates. That is autonomy.
3. For letters leaving the country, two arrangements exist.
4. A small country might pay a big neighbour to handle everything international. That is transit. One bill, global reach.
5. Two countries that exchange a lot of post might set up a direct exchange at the border and simply swap sacks, no invoice. That is peering.
6. A border exchange building where twenty countries all bring sacks and swap with whoever they like is an internet exchange point.
7. Where this comparison breaks: postal countries have fixed geography and cannot choose their neighbours. Networks connect to whoever they wish, anywhere, so an AS in India can peer directly with one in Germany.

PLAIN26.8.3 a worked example#

1. Take the address at the end of the reader’s story, 20.207.73.82.
2. Any of these three commands tells you which AS owns it.

whois -h whois.cymru.com " -v 20.207.73.82"
dig +short 82.73.207.20.origin.asn.cymru.com TXT
whois 20.207.73.82

3. The second one reverses the four octets and asks a special DNS zone. It is the fastest, because it is just a DNS lookup.
4. The answer for this address, checked in August 2026: AS8075, Microsoft Corporation, with the covering route 20.192.0.0/10, registered in the United States, and the address itself geolocating to Pune, India.
5. That last detail lines up neatly with the trace. Hops 10, 11 and 12 carry pnq20 and pnq21 in their names, and pnq is the airport code for Pune.
6. AS8075 is registered under the name MICROSOFT-CORP-MSN-AS-BLOCK, which is also why the hop names use msn.net.

PLAIN26.8.4 what is really happening inside#

1. Networks are informally sorted into tiers, and the definition is about whether you pay anyone.
2. A tier 1 network can reach every destination on the internet without buying transit from anybody. It peers with all the other tier 1 networks.
3. A tier 2 network peers with many networks but still buys transit from at least one provider to reach everything.
4. A tier 3 network buys transit and does little or no peering. Most small ISPs are here.
5. Experts disagree about who counts as tier 1, because the list depends on private contracts nobody publishes. Treat any published list as an informed estimate.
6. An internet exchange point, or IXP, is a neutral room with a large ethernet switch in it.
7. Every member plugs one port into that switch. Once connected, a member can exchange traffic with any other member that agrees to peer.
8. The exchange does not route anything. It provides a shared layer 2 fabric. The peering itself is BGP sessions between members over that fabric.
9. Most exchanges also run a route server, so a new member can get routes from dozens of others by setting up one session instead of dozens.

TECHNICAL26.8.5 the engineer’s version#

1. AS numbers were originally 16 bit, giving 0 to 65,535. RFC 6793, December 2012, extended them to 32 bit, giving 0 to 4,294,967,295.

2. AS23456 is reserved as AS_TRANS, a placeholder used when a 32 bit AS number must be shown to a router that only understands 16 bit numbers.
3. IANA allocates blocks of AS numbers to the five regional internet registries, ARIN, RIPE NCC, APNIC, LACNIC and AFRINIC, which then assign them.
4. Real AS numbers worth knowing:

5. The reader’s resolver, 1.1.1.1, is announced by AS13335. The name the resolver returned, 20.207.73.82, sits inside AS8075. Two entirely separate networks were involved before a single packet went anywhere.
6. Real internet exchange figures, all checkable:

7. Tools: whois -h whois.cymru.com, the PeeringDB directory for who peers where, bgp.he.net and bgp.tools for AS relationships, and the CIDR Report for daily table statistics.

WORDS26.8.6 remember these#

1. Autonomous system — one network under one routing policy — a set of prefixes with a single clearly defined routing policy, identified by an AS number.
2. ASN — the number that names a network — a 16 or 32 bit autonomous system number allocated by IANA through a regional registry.
3. Transit — paying someone for global reach — a paid service where a provider announces your prefixes to the whole internet and carries your traffic.
4. Peering — swapping traffic directly — an interconnection carrying only the two parties’ own customer routes.
5. Settlement-free — peering with no invoice — peering where neither side pays the other, the normal arrangement between similar-sized networks.
6. IXP — a shared room where networks plug in and swap — an internet exchange point providing a neutral layer 2 fabric for bilateral BGP peering.

26.9 BGP, the protocol that glues the internet together#

PLAIN26.9.1 in simple words#

1. Inside one network, routers use protocols like OSPF to find the best path.
2. Between networks, none of that works, because the two sides do not trust each other, do not share a map, and have contracts to honour.
3. So there is a second protocol just for the boundaries between autonomous systems. It is called the Border Gateway Protocol, or BGP.
4. BGP is how one network announces to another: “I can reach these address blocks, and here is the list of networks a packet would cross to get there”.
5. That list of networks is called the AS path, and it is the heart of the protocol.
6. There are only two things BGP ever says. “I can reach this block”, which is an announcement, and “I can no longer reach this block”, which is a withdrawal.
7. Here is the part that surprises engineers: BGP does not pick the fastest path. It picks the path its operator’s policy prefers.
8. Policy usually means money. A route through a network you do not pay is preferred over a route through a network you do pay, even if it is longer.
9. And here is the uncomfortable part: in the original design, BGP has no way to check whether an announcement is true.

PLAIN26.9.2 a picture in your head#

1. Imagine a shipping company deciding how to send goods to Chennai.
2. Three freight brokers phone in. Each says “I can get it there”, and each gives you a chain of firms it would pass through.
3. Broker A quotes a chain of two firms. Broker B quotes three. Broker C quotes two, but one of them charges you per container while the others do not.
4. You do not pick the shortest chain. You pick the one that costs you least, then use chain length only to break ties.
5. Now the flaw. Nobody verifies that any broker can actually reach Chennai. Whoever claims the most direct-sounding route gets the goods.
6. Where this comparison breaks: a shipping company would notice quickly that its goods never arrived. On the internet, the sender often sees only silence, which is exactly the shape of the reader’s failed connection.

PLAIN26.9.3 a worked example#

1. This is the YouTube hijack of Sunday 24 February 2008, one of the clearest real examples ever recorded, documented by the RIPE NCC.
2. YouTube, then AS36561, announced 208.65.152.0/22, a block containing its servers.
3. Pakistan’s government ordered access to YouTube blocked. Pakistan Telecom, AS17557, decided to do it by announcing a more specific route internally so its own users would be sent nowhere.
4. At 18:47 UTC, AS17557 began announcing 208.65.153.0/24. That is a /24 inside YouTube’s /22, so it is more specific.
5. The announcement escaped. Pakistan Telecom’s upstream provider, PCCW Global, AS3491, accepted it and passed it to the rest of the world.
6. Longest prefix match then did what it always does. A /24 beats a /22, so routers everywhere sent YouTube traffic to Pakistan Telecom.

7. Notice YouTube’s counter-move at 20:07. It announced the same /24 itself, so the two announcements were equally specific and the tie fell to AS path length. That recovered some traffic but not all.
8. It ended only when PCCW stopped accepting the announcement at 21:01.

PLAIN26.9.4 what is really happening inside#

1. BGP is a path vector protocol. Each announcement carries the full list of autonomous systems it has crossed, not just a distance.
2. That list has two jobs. It breaks loops, because a router discards any announcement that already contains its own AS number. And it is a tie-break, because a shorter list is preferred.
3. Two kinds of BGP session exist and they behave differently.
4. eBGP runs between two different autonomous systems. Every time an announcement crosses an eBGP session, the sender adds its own AS number to the front of the path.
5. iBGP runs between routers inside one autonomous system. The AS path is not changed, because the announcement has not left the AS.
6. iBGP has a rule that catches everyone out: a router will not pass a route learned from one iBGP peer on to another iBGP peer. That prevents loops, but it means every iBGP router must talk to every other one, or you need route reflectors.
7. When several announcements offer the same prefix, BGP runs a fixed list of tie-breakers, in strict order, and stops at the first one that decides.
8. The first real tie-break is local preference, a number set by your own policy. It exists to encode “prefer the free path over the paid path”.
9. Only after local preference does AS path length matter. This is why the internet’s routes are frequently not the shortest ones available.

TECHNICAL26.9.5 the engineer’s version#

1. BGP-4 is specified in RFC 4271, January 2006, which obsoleted RFC 1771 from
   1995. The first version, RFC 1105, dates from June 1989 and was written by Kirk Lougheed of Cisco and Yakov Rekhter of IBM.
2. The best path selection order, as implemented by Cisco IOS. Steps 1 and parts of the ordering are vendor conventions, not parts of RFC 4271.

3. Read that table as a statement about economics. The two strongest knobs, weight and local preference, are both purely local policy. Distance does not appear until step 4, and physical latency never appears at all.
4. Three incidents worth knowing exactly:
   1. 25 April 1997, the AS 7007 incident. A router at a small Florida provider re-announced large parts of the internet as /24s, and much of the global routing system followed the more specific routes into it.
   2. 24 February 2008, the Pakistan Telecom YouTube hijack described above.
   3. 4 October 2021, the Facebook outage. Cloudflare recorded a burst of BGP updates from Facebook’s network beginning around 15:39 UTC, including withdrawal of the routes covering all of its authoritative DNS servers.
5. The Facebook case is the cleanest illustration of withdrawal. Facebook’s DNS servers were designed to withdraw their own routes if they lost contact with the data centres, and a backbone configuration change triggered exactly that.
6. Resolvers worldwide, including the 1.1.1.1 service the reader uses, then returned SERVFAIL for facebook.com, because the name servers were not merely down, they were unreachable at the routing layer.
7. BGP activity resumed shortly before 21:00 UTC, routing was restored at about 21:50 UTC, and services were generally usable again by about 22:45 UTC. Six to seven hours, from a routing change, not an attack.
8. A route leak is different from a hijack. The prefixes are genuine, but they are passed to a party that should never have received them, usually by a network announcing its providers’ routes to its other providers.
9. On 16 April 2021, AS55410, Vodafone Idea in India, leaked more than 30,000 prefixes it did not own, drawing a large multiple of its normal inbound traffic and disrupting reachability well beyond India.
10. The current defence is the Resource Public Key Infrastructure, RPKI, architecture in RFC 6480, February 2012.
11. An address holder publishes a signed Route Origin Authorization, a ROA, saying “AS number X is authorized to originate this prefix, up to this length”. 20.207.73.82 is covered by a valid ROA today.
12. Routers then perform Route Origin Validation, RFC 6811, January 2013, marking each route Valid, Invalid or NotFound, and operators normally drop the Invalid ones.
13. Deployment figures from the NIST RPKI Monitor for 1 August 2026, IPv4:

14. Measured by address space rather than by prefix count, 61.64 percent of announced /24 equivalents were covered by a valid ROA on the same date.
15. Coverage is not the same as enforcement. Published assessments in 2026 put the share of autonomous systems actually enforcing ROV at roughly a quarter, and this is the number that matters.
16. RPKI origin validation only proves who may originate a prefix. It says nothing about whether the rest of the AS path is real, so it stops the 2008 style hijack but not every route leak.
17. Tools: birdc show protocols, RIPE RIS and RouteViews for historical BGP data, BGPlay for animated replays of incidents, and public looking glasses for a live view from another network.

WORDS26.9.6 remember these#

1. BGP — how networks tell each other what they can reach — the Border Gateway Protocol, RFC 4271, running over TCP port 179.
2. Path vector — an announcement carrying its full route history — a protocol where each update lists the AS numbers traversed.
3. AS path — the list of networks a route crossed — the AS_PATH attribute, used for loop detection and as a tie-break.
4. Announcement — “I can reach this block” — a BGP UPDATE advertising NLRI with path attributes.
5. Withdrawal — “I can no longer reach this block” — a BGP UPDATE removing a previously advertised prefix.
6. Local preference — a policy dial for choosing paid versus free — the LOCAL_PREF attribute, compared before AS path length.
7. eBGP — a session between two networks — external BGP, which prepends the local AS number to the path.
8. iBGP — a session inside one network — internal BGP, which does not change the AS path and does not re-advertise between iBGP peers.
9. Hijack — announcing address space that is not yours — an unauthorized origin announcement, valid to every router that hears it.
10. Route leak — passing routes to somebody who should not get them — propagating routes in violation of the intended business relationship.

26.10 Why nobody knows the whole path in advance#

PLAIN26.10.1 in simple words#

1. There is no plan for a packet’s journey. Nowhere is one written down.
2. Each router knows only one thing about a destination: which neighbour to hand the packet to next.
3. It does not know how many hops remain, or which countries they are in, or whether the far end is even up.
4. This is called hop-by-hop forwarding, and it is the whole model.
5. The path exists only as the sum of many independent local decisions, made fresh for every packet.
6. So the path can change in the middle of a conversation, without warning, and nothing has to be renegotiated.
7. And the way back need not use the same routers as the way out. The two directions are decided by different networks with different policies.

PLAIN26.10.2 a picture in your head#

1. Imagine posting a letter and asking every person you meet, “which way to Pune?”, then walking one street in the direction they point.
2. At the next corner you ask again. Nobody you meet knows the whole route. Each knows only the next turn.
3. You will get there, because each person is right about their own street.
4. If a street closes while you are walking, the next person simply points a different way. Nothing had to be cancelled.
5. Where this comparison breaks: you can see where you have been. A packet cannot. It carries no record of its journey unless a rare option is enabled.

PLAIN26.10.3 a worked example#

1. The reader’s traceroute shows 12 hops outward before the replies stop.
2. Hop 1 is the router in the flat. Hops 2 to 6 are private ISP addresses. Hop 3 is the ISP’s public address 137.97.29.249. From hop 7 the names become ntwk.msn.net, so the packets are inside Microsoft’s network.
3. The names then read Delhi, then Mumbai, then Pune. That is a real geographic journey, visible in the site codes del01, bom01, pnq20 and pnq21.
4. Now the honest limit. Every one of those 12 lines was learned from a reply sent back to the reader.
5. A reply is generated when a packet’s TTL hits zero. The reply is a brand new packet, sent from that router to the reader, over whatever path that router chooses.
6. So each line tells you one address on the forward path, and nothing whatever about how the answer travelled home.
7. If the fault lies on the return path, a traceroute from the reader’s laptop cannot see it at all. That is a limit of the model, not of the tool.

PLAIN26.10.4 what is really happening inside#

1. Paths differing in the two directions is called asymmetric routing, and it is the normal case on the internet, not the exception.
2. It happens because forward and reverse decisions are made by different parties. Your ISP decides how to reach Microsoft. Microsoft decides how to reach your ISP.
3. Both apply their own local preference, so both may pick a cheaper path than the other side chose.
4. A stateful firewall that only sees one direction of a flow will drop it, because it never saw the handshake it expects.
5. To diagnose properly you need a trace from both ends, or a looking glass at the far network that can trace back towards you.

TECHNICAL26.10.5 the engineer’s version#

1. Traceroute was written by Van Jacobson in 1987 and released in 1988. It works by sending probes with TTL 1, 2, 3 and upward, and reading the ICMP time exceeded messages, ICMP type 11 code 0, from RFC 792.
2. Classic Unix traceroute uses UDP to high destination ports. macOS supports -I for ICMP echo probes and -T for TCP probes.
3. traceroute -T -p 443 20.207.73.82 is the version worth running for the reader’s fault, because it probes the exact port that failed. Firewalls frequently treat UDP, ICMP and TCP port 443 completely differently.
4. Silence at the end of a trace proves nothing on its own. Routers commonly rate limit ICMP generation, and many networks suppress it entirely. Hops 13 to 20 showing * * * is consistent with a healthy path.
5. Reverse path visibility requires cooperation: a public looking glass, a RIPE Atlas measurement from a probe near the target, or mtr run from both ends.
6. mtr is the better everyday tool, because it runs continuously and shows per-hop loss and jitter rather than a single snapshot.

WORDS26.10.6 remember these#

1. Hop-by-hop forwarding — each router only knows the next step — destination based forwarding with no end-to-end path state.
2. Asymmetric routing — the way back is not the way out — forward and reverse paths differing because each direction is chosen by a different AS.
3. Time exceeded — the message a router sends when the counter runs out — ICMP type 11 code 0, the mechanism traceroute depends on.
4. Looking glass — a window into another network’s view — a public interface for running traceroute or BGP queries from a remote AS.
5. ICMP rate limiting — a router answering only some probes — deliberate throttling of ICMP generation, which shows as false loss in traceroute.

26.11 Load balancing inside the network#

PLAIN26.11.1 in simple words#

1. Between two big routers there is rarely one cable. There are often four, eight or sixteen, run in parallel for capacity and safety.
2. When a router has several equally good ways to reach a destination, it uses all of them. This is called equal cost multipath, or ECMP.
3. Now the question: how does it decide which cable each packet takes?
4. The safe method is to keep each conversation on one cable. All packets of one connection follow the same link, in order.
5. The router works out which cable by mixing a few fields from the packet header into a number and taking the remainder. The same conversation always produces the same number.
6. This is why the reader’s traceroute shows two or three different addresses at hops 5, 6, 8, 9 and 11.
7. Traceroute sends many probe packets, and probes are usually not all part of one conversation, so they scatter across the parallel links and come back with different router addresses.
8. That is not a fault. It is the visible fingerprint of a healthy, redundant network, and it is PROVEN evidence of parallel paths rather than of trouble.

PLAIN26.11.2 a picture in your head#

1. Think of a toll plaza with eight lanes going the same way.
2. If you sent every car down whichever lane was shortest right now, cars from the same convoy would arrive out of order.
3. So the plaza uses a rule instead: take the number plate, add up the digits, and send the car to lane number remainder-of-that.
4. Every car from the same fleet, with related plates, ends up in the same lane, in order. Different fleets spread evenly across all eight.
5. Where this comparison breaks: a lane that closes forces a re-computation, and with a naive rule that reshuffles everybody, not just the affected cars. Real networks use hashing schemes designed to limit that damage.

PLAIN26.11.3 a worked example#

1. The fields used are called the five-tuple: source address, destination address, protocol number, source port and destination port.
2. For the reader’s connection those are 192.168.0.14, 20.207.73.82, protocol 6, some ephemeral source port, and destination port 443.
3. Suppose a router has four equal paths. It hashes the five-tuple to a number and takes that number modulo 4.
4. A second connection from the same laptop to the same server uses a different source port, so a different hash, so possibly a different path.

5. Now look at the trace again. Hop 8 shows be23.rwa02.bom01.ntwk.msn.net and po22.rwa04.bom01.ntwk.msn.net. Two different Mumbai routers answered probes with the same TTL.
6. That means the probes took two different equal-cost paths through Mumbai. Both are correct answers.

PLAIN26.11.4 what is really happening inside#

1. There are two ways to spread traffic, and the difference matters enormously.
2. Per-flow hashing keeps a conversation on one link. Packets arrive in order. Each individual conversation is limited to the speed of one link.
3. Per-packet spreading sends each packet down whichever link is free. It uses capacity perfectly and reorders packets badly.
4. Reordering hurts TCP. Three out-of-order acknowledgements look exactly like a lost packet, so TCP slows down even though nothing was lost.
5. That is why per-flow is the default nearly everywhere, and per-packet is used only in controlled environments.
6. Bundling several physical links into one logical link, which is what ae, be and po refer to, uses the same hashing idea one layer lower down.

TECHNICAL26.11.5 the engineer’s version#

1. ECMP is described in RFC 2991, Multipath Issues in Unicast and Multicast Next-Hop Selection, and RFC 2992, Analysis of an Equal-Cost Multi-Path Algorithm, both from November 2000.
2. RFC 2992 analyses hash-threshold assignment, chosen specifically because it disturbs fewer existing flows when the number of paths changes than a naive modulo does.
3. Typical hash inputs by layer:

4. Tunnels break flow spreading. If thousands of flows share one outer header, the hash sees one flow. MPLS entropy labels and the UDP source port in VXLAN and in QUIC exist to restore variety.
5. Traceroute has a specific defence. The Paris traceroute technique keeps the flow identifier constant across probes so that all probes follow one path, which makes multipath fan-out disappear and reveals the true single path.
6. On macOS and Linux, traceroute -T -p 443 fixes the destination port and protocol, which alone removes much of the scatter.

WORDS26.11.6 remember these#

1. ECMP — using several equally good paths at once — equal cost multipath forwarding across next hops with identical metrics.
2. Five-tuple — the five fields that identify a conversation — source IP, destination IP, protocol, source port and destination port.
3. Flow hashing — arithmetic that pins a conversation to one link — hashing selected header fields to choose among equal cost next hops.
4. Per-flow versus per-packet — keep a conversation together, or spread every packet — order-preserving versus capacity-maximizing load distribution.
5. Link aggregation — several cables acting as one — bundling physical links, called ae, be or po depending on the vendor.

26.12 Quality of service and the real world#

PLAIN26.12.1 in simple words#

1. When packets arrive at a router faster than they can leave, they wait in a queue. That queue is called a buffer.
2. A small buffer smooths out short bursts. That is useful.
3. A large buffer holds packets for a long time before sending them. That is harmful, and it is why a fast connection can feel slow.
4. The packet is not lost. It is delayed, sometimes by a whole second, which for a video call or a game is worse than losing it.
5. This problem has a name: bufferbloat.
6. When a buffer finally fills completely, the simplest routers throw away whatever arrives next. That is called tail drop.
7. Better routers drop or mark a few packets early, on purpose, before the queue gets long, to tell senders to slow down sooner.
8. That deliberate early dropping is called active queue management.

PLAIN26.12.2 a picture in your head#

1. Think of a single checkout in a shop with a very long roped queue behind it.
2. Adding more rope does not make the checkout faster. It only lets more people stand in line, so everybody waits longer.
3. Somebody who wants one item now waits behind a trolley shopper’s whole load.
4. The fix is not more rope. It is a shorter permitted queue, and a rule that turns people away early rather than letting the line grow without limit.
5. Where this comparison breaks: shoppers who are turned away go home. A sender whose packet is dropped simply slows down and tries again, which is exactly the message the drop was meant to send.

PLAIN26.12.3 a worked example#

1. Take a home connection with a 10 Mbit/s upload and a 1 megabyte buffer in the modem, which is not unusual.
2. One megabyte is 8,000,000 bits. At 10,000,000 bits per second, a full buffer takes 0.8 seconds to drain.
3. So while a large upload is running, every other packet from that house waits up to 800 milliseconds behind it.
4. A video call needs round trips under about 150 milliseconds to feel natural. The call breaks even though the link is not full.

5. Reducing the buffer from 1 MB to 64 KB cuts the worst case from 800 milliseconds to 51, on the same link, with no extra bandwidth.

PLAIN26.12.4 what is really happening inside#

1. Two different tools are often confused: shaping and policing.
2. Shaping delays traffic to make it fit a target rate. Packets queue and come out smoothly. Nothing is thrown away unless the queue overflows.
3. Policing simply discards anything above the target rate. There is no queue, so there is no added delay, but there is more loss.
4. Shaping is kinder to TCP. Policing is cheaper to implement and common on provider edges.
5. Marking is a third idea. The DSCP field in the IP header lets a sender label a packet as more or less urgent.
6. The honest version: your marking usually does not survive. Most networks rewrite or zero the DSCP field at their boundary, because they will not let an outsider decide their queueing.

TECHNICAL26.12.5 the engineer’s version#

1. Jim Gettys named bufferbloat in 2010 and wrote it up as Bufferbloat: Dark Buffers in the Internet in ACM Queue in 2011.
2. Kathleen Nichols and Van Jacobson published CoDel, Controlling Queue Delay, in ACM Queue in 2012. It became RFC 8289, and the flow-queueing variant fq_codel became RFC 8290, both in January 2018.
3. CoDel does not target a queue length. It targets a queue delay, with a default target of 5 milliseconds measured over a 100 millisecond interval.
4. fq_codel adds per-flow queues with round-robin service, so a bulk transfer cannot delay a DNS lookup or a video call sharing the same link.
5. Earlier active queue management used Random Early Detection, from the 1993 paper by Sally Floyd and Van Jacobson, which drops with rising probability as the average queue grows. RED needs tuning; CoDel does not.
6. ECN, RFC 3168, lets a router set a bit instead of dropping. The receiver echoes it, the sender slows down, and nothing is lost.

7. Tools: tc qdisc on Linux shows and sets the queue discipline, and fq_codel is the default on most current distributions. The flent test suite measures bufferbloat directly.
8. Whether providers should deprioritize particular traffic is a policy argument, not a technical one, and experts disagree. In India, the regulator TRAI barred discriminatory pricing of data services in February 2016 and net neutrality rules followed in 2018.

WORDS26.12.6 remember these#

1. Buffer — the waiting room for packets — memory holding packets queued for an outgoing interface.
2. Bufferbloat — a fast link that feels slow — excessive latency caused by oversized unmanaged buffers.
3. Tail drop — throw away whatever arrives when full — the default queue discipline, dropping arrivals once the buffer is full.
4. AQM — dropping early on purpose — active queue management, signalling congestion before the buffer fills.
5. fq_codel — many short queues, each kept short — the flow queue CoDel scheduler, RFC 8290, targeting 5 ms of queue delay.
6. Shaping — delay traffic to fit a rate — queueing excess traffic for later transmission.

26.13 Reading a packet capture#

PLAIN26.13.1 in simple words#

1. Everything in this chapter can be watched directly. You do not have to guess.
2. A packet capture is a recording of the actual frames going in and out of a network card.
3. The command line tool is tcpdump. The graphical tool is Wireshark. They read the same file format.
4. There are two kinds of filter, and mixing them up wastes hours.
5. A capture filter decides what gets recorded. It runs inside the kernel and what it rejects is gone forever.
6. A display filter decides what you are shown from an existing recording. Nothing is lost, and you can change it as often as you like.

PLAIN26.13.2 a picture in your head#

1. A capture filter is the security guard at the door of a building, deciding who is allowed in at all.
2. A display filter is you, later, searching the visitor book for one name.
3. If the guard turned somebody away, no amount of searching the book will find them. The record simply does not exist.
4. Where this comparison breaks: the guard is fast and free, and letting everybody in has a real cost. On a busy link, capturing everything can fill a disk in minutes and drop packets while doing so.

PLAIN26.13.3 a worked example#

1. Here is what the reader’s failure looks like on the wire.

$ sudo tcpdump -ni en0 host 20.207.73.82
09:14:02.118 IP 192.168.0.14.51422 > 20.207.73.82.443:
   Flags [S], seq 1829384756, win 65535, length 0
09:14:03.121 IP 192.168.0.14.51422 > 20.207.73.82.443:
   Flags [S], seq 1829384756, win 65535, length 0
09:14:05.128 IP 192.168.0.14.51422 > 20.207.73.82.443:
   Flags [S], seq 1829384756, win 65535, length 0
09:14:09.142 IP 192.168.0.14.51422 > 20.207.73.82.443:
   Flags [S], seq 1829384756, win 65535, length 0

2. Read the four things that matter.
3. Every line is outbound. Nothing at all comes back. Not a SYN-ACK, not a reset, not an ICMP message.
4. The sequence number is identical every time, so these are retransmissions of one SYN, not four separate attempts.
5. The gaps are 1, 2 and 4 seconds. That is TCP’s exponential backoff, which is the operating system trying harder and harder before giving up.
6. The source port 51422 stays the same, confirming one connection attempt.
7. That capture, on its own, PROVES the request left the laptop and no answer returned. It does not prove where the packet died.
8. Compare with a refused connection, which replies at once with Flags [R], a reset, usually within milliseconds. The reader saw nothing of the kind.

PLAIN26.13.4 what is really happening inside#

1. Capturing works by putting the network card driver into a mode where it hands a copy of every frame to a special kernel facility.
2. That facility applies the capture filter, compiled into a tiny program, and copies only matching frames into a ring buffer for the tool to read.
3. Because the filter runs in the kernel, a narrow capture filter costs almost nothing even on a fast link.
4. Two cautions. On Wi-Fi you normally see only your own traffic unless the card is put into monitor mode. And on a switch you see only frames addressed to you, unless a port is configured to mirror others.
5. Encrypted payloads stay encrypted. You can see that a TLS handshake happened, its size and its timing, but not the HTTP request inside it.

TECHNICAL26.13.5 the engineer’s version#

1. tcpdump was written in 1988 at Lawrence Berkeley Laboratory by Van Jacobson, Craig Leres and Steven McCanne. The filtering engine underneath it, the BSD Packet Filter, was described by McCanne and Jacobson in 1993.
2. Wireshark began as Ethereal, released by Gerald Combs in 1998, and was renamed Wireshark in 2006 after a trademark problem.
3. Capture filters use BPF syntax. Display filters use Wireshark’s own syntax. They are not interchangeable, and this trips up nearly everyone once.

sudo tcpdump -ni en0 -s0 -w gh.pcap \
  'host 20.207.73.82 and tcp port 443'
tcpdump -r gh.pcap -vvn

4. Useful display filters in Wireshark for this fault: ip.addr == 20.207.73.82, tcp.flags.syn == 1 && tcp.flags.ack == 0, tcp.analysis.retransmission, and icmp.
5. -s0 captures whole frames rather than truncating them. -w writes a pcap file. Analysis should always be done on a file, never on live scrolling text.
6. A checklist for diagnosing silence, in order:
   1. Is there an outbound SYN at all, and to the right address?
   2. Is anything at all coming back, including ICMP?
   3. Are the retransmissions identical, or is the stack trying new ports?
   4. Does a capture on another network, such as mobile data, differ?
7. Tools worth having beside tcpdump: mtr for continuous per-hop loss, dig for name resolution, curl -v --trace-time for per-stage timing, and tshark for scripted analysis of a saved capture.

WORDS26.13.6 remember these#

1. Packet capture — a recording of real traffic — a stored copy of frames read from an interface, normally in pcap or pcapng format.
2. Capture filter — decides what is recorded — a BPF expression evaluated in the kernel before frames are copied to userspace.
3. Display filter — decides what is shown — a Wireshark expression applied to an already captured set of packets.
4. Promiscuous mode — the card keeps frames not addressed to it — an interface setting required to see other stations’ traffic.
5. Retransmission — the same data sent again — a repeat of an unacknowledged segment, identifiable by an identical sequence number.

26.98 Common wrong ideas#

1. Wrong: packets travel down a fixed path that is set up when the connection starts. Right: nothing is set up. Each router decides independently, per packet, and the path can change mid-conversation without anyone being told.
2. Wrong: the internet routes around damage automatically and instantly. Right: it routes around damage eventually, if an alternative exists and if policy allows it. Convergence takes seconds inside one network and can take minutes across BGP, during which traffic is lost.
3. Wrong: a traceroute shows the return path. Right: it shows one address per hop on the way out, learned from replies that came back by an unknown route. The return path is invisible from one end.
4. Wrong: more hops means slower. Right: hop count and latency are only loosely related. Twelve hops inside one well-run backbone can beat four hops over a congested link or a satellite.
5. Wrong: stars at the end of a traceroute prove a fault. Right: many routers rate limit or block ICMP replies. The reader’s hops 13 to 20 showing * * * is consistent with a perfectly healthy path.
6. Wrong: seeing two or three addresses for one hop means something is broken. Right: it means parallel links are in use, which is normal and desirable.
7. Wrong: BGP picks the shortest path. Right: BGP picks the path its operator’s policy prefers. Local preference, which is pure business policy, is compared before AS path length, and physical distance is never compared at all.
8. Wrong: my ISP can see which pages I loaded because it carries my packets. Right: it sees addresses, sizes and timing. With TLS it does not see the content, though it can often infer a great deal from the pattern.

26.99 Chapter summary in 20 lines#

1. Data is split into packets for fairness, cheap error recovery, small buffers and the ability to interleave many conversations on one wire.
2. Circuit switching reserves a path and wastes capacity. Packet switching reserves nothing and accepts variable delay. The internet chose the second.
3. A packet is a header plus a payload, and the payload is normally another whole packet with its own header.
4. A 512 byte HTTPS request from the reader’s laptop becomes a 604 byte Ethernet frame, which is 15.2 percent overhead.
5. The IPv4 header is 20 bytes and 13 fields, unchanged since RFC 791 in 1981.
6. TTL stops packets circling forever. Decrementing it forces the header checksum to be recomputed, which routers do incrementally.
7. Every link has an MTU. Oversized packets are fragmented, rejected with an ICMP message, or silently dropped, and the third case is the painful one.
8. IPv6 never fragments in the network. Modern IPv4 senders set the Do Not Fragment bit and discover the path MTU for themselves.
9. A router receives a frame, strips it, looks up the destination, decrements TTL, fixes the checksum, resolves the next hop and builds a brand new frame.
10. MAC addresses change at every hop. IP addresses do not, unless a device is performing NAT.
11. A routing table is matched by longest prefix. 20.207.73.0/24 beats 20.192.0.0/10 beats 0.0.0.0/0 for the address 20.207.73.82.
12. The reader’s default route, 0.0.0.0/0 through 192.168.0.1, carried every one of these packets out of the flat.
13. Inside a network, routes are learned by RIP, OSPF, IS-IS or EIGRP. Link state protocols flood a map and run Dijkstra’s algorithm on it.
14. Between networks, routes are learned by BGP, which carries a full AS path and chooses by policy, with local preference outranking path length.
15. 20.207.73.82 belongs to AS8075, Microsoft, inside the route 20.192.0.0/10, which is why hop 7 onwards reads ntwk.msn.net.
16. BGP has no built-in way to verify an announcement. RPKI adds signed origin authorizations, and about 69 percent of IPv4 prefix-origin pairs were valid in August 2026, but only about a quarter of networks enforce checking.
17. Nobody knows the whole path in advance. Each router knows only the next hop, and the return path is chosen by different networks entirely.
18. Multiple addresses at hops 5, 6, 8, 9 and 11 are equal cost multipath at work, selected by hashing the five-tuple, and are healthy rather than faulty.
19. Oversized buffers turn loss into delay and make fast links feel slow. Short managed queues, such as fq_codel, fix it without more bandwidth.
20. A capture showing four identical SYNs at 1, 2 and 4 second gaps with no reply of any kind PROVES the request left and nothing came back, and SUGGESTS a silent drop somewhere on that path.