Firewalls, VPNs, CDNs, IPv6 and the Gaps

34.0 What this chapter gives you#

1. You will be able to say exactly what localhost, 127.0.0.1 and 0.0.0.0 each mean, and why 0.0.0.0 means two different things depending on where you write it.
2. You will be able to explain the three port ranges, why ports below 1024 need root on a UNIX machine, and why a busy machine can run out of ports even though it has 65,535 of them.
3. You will be able to describe what a firewall is, what it is not, and the difference between a stateless filter and a stateful one, with real rules written in pf, iptables, nftables and a cloud security group.
4. You will be able to explain the difference between DROP and REJECT well enough to diagnose a fault from the shape of the failure alone, and you will recognize the reader’s own 15-second silence as the DROP signature.
5. You will be able to name the middleboxes sitting between you and any server, and say what each one can and cannot see.
6. You will be able to explain what a VPN really does at the packet level, choose between IPsec, OpenVPN and WireGuard for a reason, and state plainly what a commercial VPN does not protect you from.
7. You will be able to tell a forward proxy from a reverse proxy for the rest of your life, and write the SSH commands for local, remote and dynamic port forwarding without looking them up.
8. You will be able to describe how a CDN handles one request end to end, and explain anycast and DNS steering as two separate mechanisms that are constantly confused.
9. You will be able to compute the theoretical minimum round trip from India to the United States from the speed of light in glass, and say why buying more bandwidth will not fix a slow page.
10. You will be able to read and write IPv6 addresses, explain SLAAC and neighbour discovery, and say what IPv6: (none) in the reader’s own output did and did not cost them.

34.1 localhost, 127.0.0.1 and 0.0.0.0#

PLAIN34.1.1 in simple words#

1. Three things get confused constantly, and all three appear in error messages every day. We will separate them once and for all.
2. 127.0.0.1 is an address that means this machine. Data sent there never leaves the computer.
3. localhost is a name for that address. It is a word, not an address, and something has to turn it into an address first.
4. 0.0.0.0 is not really an address at all. It is a placeholder, and it means two different things in two different places.
5. When a program listens on 0.0.0.0, it means “accept connections on every network the machine has”.
6. When something sends to 0.0.0.0, it means “this host, or an address I do not know yet”.
7. That double meaning is the whole confusion. Same four numbers, opposite jobs, decided by whether you are listening or sending.
8. The practical result trips up almost every beginner. A server started on 127.0.0.1 works perfectly from the same machine and is completely invisible from any other machine.
9. That is not a bug and not a firewall. It is the address doing exactly what it says.

PLAIN34.1.2 a picture in your head#

1. Think of a large office building with a postal address on the street, and an internal mail system for moving envelopes between desks.
2. 127.0.0.1 is the internal mail system. You write a note, drop it in the internal tray, and it reaches another desk in the same building. It never goes near the street.
3. localhost is the phrase “internal mail”. It is a word people use. The mail room has to look it up to know which tray that is.
4. Binding to 127.0.0.1 is like telling reception “I only accept internal mail”. A courier from outside is turned away at the door.
5. Binding to 0.0.0.0 is like saying “I accept mail from any entrance the building has, including the loading bay and any door they add next year”.

Where this comparison breaks:

6. In a real building, a courier could still walk in and hand you the envelope in person. On a computer there is no such route. A packet addressed to 127.0.0.1 that arrives from a cable is thrown away by the operating system before any program sees it.
7. And the office has one internal mail system. Your machine has sixteen million loopback addresses, all valid, all pointing at itself.

PLAIN34.1.3 a worked example#

1. Here is the loopback interface on a Mac, and the same idea on Linux.

$ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1

2. Notice the MTU is 16384 bytes, not the usual 1500. There is no cable to respect, so the operating system allows much bigger chunks.
3. Notice ::1. That is the IPv6 version of 127.0.0.1, and it matters more than people expect, as we will see in a moment.
4. Now start a web server two ways and look at the difference.

$ python3 -m http.server 8000 --bind 127.0.0.1
$ lsof -nP -iTCP:8000 -sTCP:LISTEN
COMMAND  PID  USER  FD  TYPE  NODE NAME
Python  4411   you   4u  IPv4  TCP  127.0.0.1:8000 (LISTEN)

$ python3 -m http.server 8000 --bind 0.0.0.0
$ lsof -nP -iTCP:8000 -sTCP:LISTEN
COMMAND  PID  USER  FD  TYPE  NODE NAME
Python  4498   you   4u  IPv4  TCP  *:8000 (LISTEN)

5. The * in the second output is the whole story. The star means “any local address”. The first case names one address and only one.
6. From a phone on the same home WiFi, the first server is unreachable and the second answers on the laptop’s address, for example 192.168.0.24:8000.
7. Here is the modern trap. Ask for localhost and you may be handed ::1 before 127.0.0.1.

$ curl -v http://localhost:8000/
*   Trying [::1]:8000...
* connect to ::1 port 8000 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000

8. If the server had been listening only on the IPv4 loopback and the client had given up after the first attempt, you would see “connection refused” from a server that is plainly running. This exact problem hit many Node.js users after version 17, released in October 2021, changed the default order in which resolved addresses are tried.

PLAIN34.1.4 what is really happening inside#

1. The loopback interface is a fake network card built into the operating system. It has no hardware, no driver for a chip, and no cable.
2. When a program sends to 127.0.0.1, the kernel builds a normal IP packet, hands it to the loopback device, and the loopback device immediately hands it back up the receive path.
3. So the packet goes down the sending half of the stack and straight up the receiving half, inside the same machine, in microseconds.
4. Everything else still happens. There is a real TCP handshake, real sequence numbers, real ports. You can capture it with tcpdump -i lo0.
5. The whole block 127.0.0.0 to 127.255.255.255 is loopback, not just the one address. That is 16,777,216 addresses, and every one of them means “this machine”.
6. On Linux you can use any of them straight away. Running one test service on 127.0.0.2:80 and another on 127.0.0.3:80 is a genuinely useful trick.
7. On macOS only 127.0.0.1 is configured by default, and you must add the others yourself with sudo ifconfig lo0 alias 127.0.0.2 up.
8. A packet from the outside world claiming to be from or to 127.x.x.x is called a martian packet, and the kernel discards it. That rule is why binding to loopback is a real isolation boundary and not just a hint.
9. 0.0.0.0 as a listening address is a wildcard. The socket is not tied to any one address, so it accepts anything arriving on any interface, including interfaces created later, such as a VPN tunnel that comes up afterwards.
10. /etc/hosts is a plain text file that maps names to addresses, and it is consulted before DNS on almost every system. Two columns: the address first, then one or more names.

TECHNICAL34.1.5 the engineer’s version#

1. 127.0.0.0/8 is reserved for loopback by RFC 1122, October 1989, section 3.2.1.3, which states that a host must never send a datagram with such an address outside itself, and must discard one that arrives from a link. RFC 6890, April 2013, records it in the special-purpose address registry.
2. ::1/128 is the IPv6 loopback, defined in RFC 4291. IPv6 spends exactly one address on it rather than a sixteen-million block.
3. localhost is a special-use domain name under RFC 6761, February 2013. Resolvers are supposed to answer it locally with a loopback address and never send it to a DNS server. Not every stack obeys this.
4. 0.0.0.0 is INADDR_ANY in the sockets API. Passing it to bind() gives a wildcard binding. In IPv6 the equivalent is in6addr_any, written ::.
5. A dual-stack listener bound to :: will, on Linux with net.ipv6.bindv6only set to 0, also accept IPv4 connections through IPv4-mapped addresses of the form ::ffff:192.168.0.24. On some BSDs the default is the opposite. This is a per-system default, not a standard.
6. 0.0.0.0 as a destination or source means “this host on this network” per RFC 1122. A DHCP client sends its first DISCOVER from source 0.0.0.0 because it has no address yet. Most stacks quietly rewrite a connect() to 0.0.0.0 as a connect to loopback, which is an implementation detail and not something to rely on.
7. 0.0.0.0/0 in a routing table is the default route: the least specific match possible, used when nothing else matches.

8. The /etc/hosts file is the direct descendant of HOSTS.TXT, the single file distributed by the Stanford Research Institute Network Information Center before DNS existed. On Linux the order of name sources is set in /etc/nsswitch.conf, usually hosts: files dns. On Windows the file is at C:\Windows\System32\drivers\etc\hosts. On macOS mDNSResponder reads /etc/hosts and it takes precedence over DNS.
9. Docker is where this bites hardest. Inside a container, 127.0.0.1 is the container’s own loopback namespace, not the host’s. A server bound to 127.0.0.1 inside a container is unreachable even with -p 8080:8080, because the published port forwards to the container’s external address, not its loopback. Bind to 0.0.0.0 inside containers.
10. The reverse form -p 127.0.0.1:8080:8080 publishes the port only on the host’s loopback, which is the correct way to expose a database container to the host and to nothing else.
11. Observation commands: lsof -nP -iTCP -sTCP:LISTEN on macOS, ss -tlnp on Linux, netstat -ano | findstr LISTENING on Windows, dscacheutil -q host -a name localhost to see what macOS resolves.

WORDS34.1.6 remember these#

1. Loopback — a fake network card inside the machine — a virtual interface that returns transmitted packets to the local receive path.
2. localhost — the word for “this machine” — a special-use domain name reserved by RFC 6761, resolving to 127.0.0.1 or ::1.
3. Wildcard bind — listen on everything — binding a socket to INADDR_ANY, 0.0.0.0, or in6addr_any, ::.
4. Martian packet — an impossible address arriving from outside — a datagram whose source or destination is reserved for local use, discarded on ingress.
5. Hosts file — a small local address book — a text file of address-to-name mappings consulted ahead of DNS.

34.2 Ports, properly this time#

PLAIN34.2.1 in simple words#

1. An address gets a packet to a machine. A port gets it to the right program on that machine.
2. A port is just a number between 0 and 65,535, carried in the TCP or UDP header. It is not a physical thing.
3. The numbers are split into three groups by agreement, and the groups have different rules.
4. The first group, 0 to 1023, holds the famous ones: 22 for SSH, 80 for plain web, 443 for secure web, 53 for DNS.
5. On Mac and Linux, only an administrator may open a program on those low numbers. That rule is old and it exists for a reason we will explain.
6. The second group, 1024 to 49151, is for registered services. Anyone may use these. Databases and development servers live here.
7. The third group, 49152 upward, is for temporary use. Every time your machine makes an outgoing connection, it borrows one number from this pool, uses it, and gives it back.
8. That borrowed number is why two browser tabs can both talk to the same website without the replies getting mixed up. Each tab’s connection has a different borrowed number.

PLAIN34.2.2 a picture in your head#

1. Think of a company with one street address and a switchboard with numbered extensions.
2. The street address is the IP address. The extension is the port. The switchboard is the operating system.
3. Extensions 1 to 1023 are reserved for official departments: reception, accounts, security. Only management may assign them.
4. Extensions in the middle range belong to named teams who have registered their number in the company directory.
5. The top range is a pool of temporary extensions. When someone in the building phones out, the switchboard lends them a spare extension so the reply comes back to the right desk.
6. When the call ends, the extension goes back in the pool, but only after a short cooling-off period, so a late reply to the old call does not ring a new person’s phone.

Where this comparison breaks:

7. A real switchboard could reuse the same temporary extension for calls to two different companies at once. A computer can too, and that is the part people get wrong. The uniqueness rule is not about the port alone. It is about the whole set of four values: your address, your port, their address, their port.
8. Also, a phone extension has one line. A listening port can hold thousands of simultaneous connections, all with the same local port number.

PLAIN34.2.3 a worked example#

1. Take the reader’s own failed request. macOS lends an ephemeral port from its range, and the connection is described by four values.

Source address: 192.168.0.24     (the laptop on the home LAN)
Source port:    54318            (borrowed from 49152-65535)
Dest address:   20.207.73.82     (github.com, as resolved)
Dest port:      443              (HTTPS)

2. That set of four is the four-tuple. It is what makes the connection unique on the machine. The kernel matches arriving packets against it.
3. Now the exhaustion arithmetic, which is the thing worth remembering. Suppose a Linux server opens many short-lived connections to one database at one address and one port.
4. Linux offers 32768 to 60999 by default, which is 28,232 ports. All four other values in the tuple are fixed, so the port number is the only thing that can vary.
5. After each connection closes, the port sits in TIME_WAIT for 60 seconds on Linux before it can be reused.
6. So the sustainable rate is 28,232 divided by 60, which is about 470 new connections per second to that single destination. Above that, connect() starts failing with EADDRNOTAVAIL.
7. The fix is not a bigger port range. The fix is connection reuse: a pool that keeps connections open instead of opening a new one per query.
8. Here is what listening sockets look like on a real machine.

$ ss -tlnp
State  Local Address:Port   Peer     Process
LISTEN 127.0.0.1:5432       0.0.0.0:* postgres
LISTEN 0.0.0.0:8080         0.0.0.0:* java
LISTEN [::]:443             [::]:*    nginx

$ netstat -an -p tcp | head -4
Proto  Local Address       Foreign Address      (state)
tcp4   192.168.0.24.54318  20.207.73.82.443     SYN_SENT
tcp4   127.0.0.1.5432      *.*                  LISTEN

9. That SYN_SENT line is the reader’s fault frozen in place. The connection was attempted, the port was allocated, and no reply ever changed the state.

PLAIN34.2.4 what is really happening inside#

1. When you call connect() without choosing a port, the kernel picks one for you. This is called an ephemeral or dynamic port.
2. Old systems picked the next number in sequence. That was predictable, so modern systems randomize the choice to make blind attacks harder.
3. The kernel then checks that the resulting four-tuple is not already in use. If it is, it picks again.
4. When a connection closes, the side that closed first keeps the tuple in a waiting state called TIME_WAIT.
5. The reason is simple and important. Packets from the old connection may still be wandering the network. If the same tuple were reused immediately, a straggler could be accepted as part of the new connection.
6. The second reason is that the last acknowledgement might be lost, and the other side may resend its final packet. Something must be there to answer it.
7. The privileged port rule is a convention from early UNIX. On a machine where only an administrator could bind port 513, receiving a connection from a low port was weak evidence that a trusted program on a trusted machine sent it.
8. That trust model is long dead, because anyone can run their own machine and bind whatever they like. The restriction survives anyway, and now it mainly stops an ordinary user from hijacking the web server’s port.
9. SO_REUSEADDR is the socket option that says “let me bind this port even though an old connection is still cooling off”. Without it, restarting a server often fails with “address already in use” for up to a minute.

TECHNICAL34.2.5 the engineer’s version#

1. Ports are 16-bit unsigned fields in the TCP and UDP headers, so 0 to 65,535. Port 0 is reserved; binding to it asks the kernel to choose.
2. RFC 6335, August 2011, also known as BCP 165, defines the registry and the three ranges, jointly maintained by IANA and the IETF.

3. The privileged range on UNIX is enforced in the kernel. On Linux the check is against CAP_NET_BIND_SERVICE, and since kernel 4.11 in 2017 the threshold itself is tunable with net.ipv4.ip_unprivileged_port_start, default 1024. Windows has never had this restriction at all.
4. The actual ephemeral ranges in use do not match the IANA suggestion, and this surprises people. Real defaults:

5. Check them with sysctl net.ipv4.ip_local_port_range on Linux, sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last on macOS, and netsh int ipv4 show dynamicport tcp on Windows.
6. RFC 6056, January 2011, specifies port randomization algorithms, motivated by off-path attacks against TCP and against DNS after Dan Kaminsky’s 2008 disclosure.
7. TIME_WAIT is defined in RFC 793, September 1981, as twice the maximum segment lifetime, with an MSL of 2 minutes, giving 4 minutes. Linux ignores that and hardcodes 60 seconds in TCP_TIMEWAIT_LEN, a compile-time constant that no sysctl exposes.
8. SO_REUSEADDR permits binding a local address that is in TIME_WAIT. SO_REUSEPORT, added in Linux 3.9 in April 2013, is different: it lets several sockets bind the identical address and port so the kernel can spread incoming connections across worker processes. On BSD the two options have older and slightly different semantics.
9. net.ipv4.tcp_tw_reuse allows reuse of a TIME_WAIT entry for a new outgoing connection when timestamps confirm safety. Its dangerous cousin tcp_tw_recycle broke clients behind NAT and was removed from Linux in version 4.12, July 2017. If you find advice recommending it, the advice is out of date.
10. Diagnostic commands worth memorizing:

lsof -nP -iTCP -sTCP:LISTEN        # macOS: who is listening
lsof -nP -i :8080                  # who owns port 8080
ss -tan state time-wait | wc -l    # Linux: TIME_WAIT count
netstat -an -p tcp | grep 443      # macOS: all port 443 sockets
sysctl net.inet.ip.portrange.first # macOS ephemeral start

WORDS34.2.6 remember these#

1. Port — a number picking the program — a 16-bit field in the TCP or UDP header identifying an endpoint within a host.
2. Ephemeral port — a temporary number borrowed for one outgoing call — a dynamically allocated source port from the local dynamic range.
3. Four-tuple — the four values that make a connection unique — source address, source port, destination address, destination port.
4. TIME_WAIT — the cooling-off period after a connection ends — a TCP state held by the closing side for twice the maximum segment lifetime.
5. Privileged port — a low number only an administrator may open — ports below 1024, requiring root or CAP_NET_BIND_SERVICE on UNIX systems.
6. Port exhaustion — running out of temporary numbers — failure of connect() with EADDRNOTAVAIL when no free four-tuple exists.

34.3 Firewalls, properly#

PLAIN34.3.1 in simple words#

1. A firewall is a set of rules that decides which packets may pass and which are stopped. That is the entire idea.
2. It sits somewhere on the path: in your laptop, in your home router, at the edge of a company, or in front of a cloud server.
3. What a firewall is not: it is not antivirus, it does not read your files, and it does not know whether a program is good or evil.
4. It also cannot protect you from something you invited in. If you connect out to a hostile server, the firewall was not asked.
5. The simplest kind looks at each packet alone and asks: where is it from, where is it going, which port, which protocol.
6. A smarter kind remembers connections. When you start a conversation, it notes that fact, and it lets the replies back in without a separate rule.
7. That memory is called state, and it is the single biggest idea in firewalls.
8. Two directions matter. Ingress filtering controls what comes in. Egress filtering controls what goes out. Most people only think about the first.
9. Egress filtering is what stops a compromised machine from phoning home. It is also, occasionally, what stops you reaching a site for no reason you can see.

PLAIN34.3.2 a picture in your head#

1. Picture a security desk at the entrance to a building.
2. The stateless guard has a printed list: “let in anyone wearing a blue badge, turn away everyone else”. Each person is judged alone, with no memory.
3. If you send a courier out with a parcel and the courier returns with a reply, the stateless guard has no idea. He checks the printed list again.
4. So with a stateless guard you need a second rule saying “let returning couriers in”, and that rule is a hole, because you cannot tell a real returning courier from someone pretending.
5. The stateful guard keeps a notebook. When your courier leaves, he writes down who left, where they went and when.
6. When a reply arrives that matches a line in the notebook, it is let in automatically. Nothing else is.
7. He crosses out lines after a while, so the notebook does not grow forever.

Where this comparison breaks:

8. The guard’s notebook holds a hundred lines. A firewall’s table holds hundreds of thousands, and when it fills, new connections are silently dropped. That failure looks exactly like a network outage and is one of the nastiest faults in operations.
9. And the guard can see faces. A firewall in front of an HTTPS site cannot see the contents at all, only the outside of the envelope.

PLAIN34.3.3 a worked example#

1. The same policy, “allow SSH and HTTPS in, allow everything out, drop the rest”, written four ways on real systems.

# Linux, iptables
iptables -P INPUT DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack \
         --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22  -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Linux, nftables
nft add table inet filter
nft add chain inet filter input \
    '{ type filter hook input priority 0; policy drop; }'
nft add rule inet filter input ct state established,related accept
nft add rule inet filter input iif lo accept
nft add rule inet filter input tcp dport { 22, 443 } accept

# macOS and BSD, /etc/pf.conf
set skip on lo0
block drop in all
pass out all keep state
pass in proto tcp to any port { 22, 443 } keep state

# AWS security group, one ingress rule
aws ec2 authorize-security-group-ingress \
  --group-id sg-0abc123 --protocol tcp \
  --port 443 --cidr 0.0.0.0/0

2. Read the common shape. Every one of them: default deny inbound, allow loopback, allow replies to things we started, then a short list of open ports.
3. In the cloud version there is no “allow established” line, because AWS security groups are stateful by design and you cannot make them otherwise.
4. Look at the connection tracking table itself on a Linux box.

$ sudo conntrack -L | head -3
tcp 6 431999 ESTABLISHED src=10.0.1.5 dst=20.207.73.82
    sport=54318 dport=443 [ASSURED] mark=0 use=1
tcp 6 118 TIME_WAIT src=10.0.1.5 dst=10.0.2.9
    sport=41022 dport=5432 [ASSURED] mark=0 use=1

5. The number after the protocol is the seconds left on that entry. 431,999 seconds is just under five days, which is the default timeout for an established TCP flow on Linux, and it is far too long for a busy box.

PLAIN34.3.4 what is really happening inside#

1. A stateless filter is a loop. For each packet, walk the rule list from the top, compare fields, and act on the first rule that matches.
2. The fields compared are the same handful every time: protocol number, source address, destination address, source port, destination port, and TCP flags.
3. A stateful filter does that only for the first packet of a flow. If the packet is allowed, it writes an entry in a table.
4. The entry holds the four-tuple, the protocol, the current TCP state, byte counters, and a timer.
5. Every later packet is looked up in that table first, usually with a hash, in constant time. Only unmatched packets fall through to the rules.
6. That is why stateful firewalls are fast for long connections and expensive for floods of new ones. A SYN flood attacks the table, not the rules.
7. For UDP, which has no connections, the firewall invents one. It creates an entry when it sees an outgoing datagram and expires it after a short timeout, usually 30 seconds. This is why your video call needs regular traffic to keep working.
8. An application-layer firewall goes further and parses the payload. It knows what an HTTP request looks like, so it can act on a method, a path or a header.
9. It cannot do that if the traffic is encrypted, unless it also terminates the encryption, which means installing its own certificate authority on every client. That is a large decision, not a checkbox.

TECHNICAL34.3.5 the engineer’s version#

1. The vocabulary, ordered by how much the device understands:

2. History, with names and dates. Jeff Mogul’s screend at Digital Equipment Corporation in 1989 was an early packet filter. DEC SEAL in 1991, built by Marcus Ranum, is widely called the first commercial firewall. Ranum’s TIS Firewall Toolkit followed in 1993.
3. Stateful inspection was commercialized by Check Point in FireWall-1, 1994, and patented by Gil Shwed, United States patent 5,606,668, filed December 1993 and granted February 1997.
4. William Cheswick and Steven Bellovin published “Firewalls and Internet Security: Repelling the Wily Hacker” in 1994, which is where much of the modern vocabulary comes from.
5. Palo Alto Networks was founded in 2005 by Nir Zuk, and the term next-generation firewall was popularized by Gartner around 2009. It means stateful inspection plus application identification, user identity and intrusion prevention in one device. It is a marketing category, not a standard.
6. ModSecurity, the first widely used open web application firewall, was released by Ivan Ristic in 2002. The OWASP Core Rule Set is the common rule pack.
7. Linux lineage: ipfwadm in kernel 2.0, 1996; ipchains in 2.2, January 1999; iptables with the netfilter framework by Rusty Russell in 2.4, January 2001; nftables merged in kernel 3.13, January 2014. Debian 10 in 2019 made nftables the default backend.
8. BSD lineage: pf was written by Daniel Hartmeier for OpenBSD 3.0 in December 2001, after a licence dispute over IPFilter. Apple adopted pf in Mac OS X 10.7 Lion, July 2011. macOS also has a separate per-application firewall, controlled by socketfilterfw, present since Mac OS X 10.5 Leopard in 2007.
9. Windows: Internet Connection Firewall shipped in Windows XP in 2001, was enabled by default and renamed Windows Firewall in XP Service Pack 2, August 2004, gained outbound rules in Vista in 2007, and was renamed Microsoft Defender Firewall around 2019.
10. Connection tracking limits are real operational numbers. nf_conntrack_max is sized from RAM at boot and is often between 65,536 and 262,144. nf_conntrack_tcp_timeout_established defaults to 432,000 seconds. When the table fills, the kernel logs “nf_conntrack: table full, dropping packet” and new connections vanish without explanation.
11. Cloud: an AWS security group is stateful, allows only allow rules, and is evaluated as the union of all groups attached to an interface. A network ACL is stateless, has numbered allow and deny rules evaluated in order, and requires you to open the ephemeral range 1024-65535 for return traffic. Both fail silently, which matters enormously in section 34.4.
12. Observation: sudo pfctl -sr and pfctl -si on macOS, iptables -L -n -v and nft list ruleset on Linux, netsh advfirewall firewall show rule name=all on Windows, conntrack -L and cat /proc/sys/net/netfilter/nf_conntrack_count.

WORDS34.3.6 remember these#

1. Firewall — a rule-based gate for packets — a policy enforcement point filtering traffic on header fields, flow state or application content.
2. Stateless filter — judges each packet alone — a filter with no flow table, requiring explicit rules for return traffic.
3. Stateful inspection — remembers conversations — filtering using a connection tracking table keyed on the flow tuple and protocol state.
4. Connection tracking table — the firewall’s notebook — the in-memory table of active flows, with states, timers and counters.
5. Ingress and egress — inbound and outbound filtering — policy applied to traffic entering or leaving an interface or network boundary.
6. Web application firewall — a filter that reads HTTP — a layer 7 filter matching request patterns against attack signatures.

34.4 DROP versus REJECT#

PLAIN34.4.1 in simple words#

1. When a firewall decides not to let a packet through, it has two ways to behave, and the difference decides how your day goes.
2. REJECT means: refuse, and say so. The firewall sends a short message back saying “no”.
3. DROP means: refuse, and say nothing. The packet is deleted. No reply of any kind is generated.
4. To the client, REJECT is a closed door with a sign on it. DROP is a wall where you expected a door.
5. With REJECT, your program fails almost instantly, usually in well under a second, with a clear error like “connection refused”.
6. With DROP, your program has no idea anything happened. It assumes the request got lost, waits, sends it again, waits longer, sends it again.
7. So it hangs. Fifteen seconds, thirty seconds, seventy-five seconds, depending on the system and the program, and then it gives up with “timed out”.
8. Both outcomes are a refusal. Only one of them tells you so.
9. This is the exact shape of the reader’s own fault, and we will come back to it at the end of this section.

PLAIN34.4.2 a picture in your head#

1. You knock on a door in a large building looking for an office.
2. In the REJECT case, someone opens the door and says “wrong floor”. You know immediately. Total time: two seconds. You go and look elsewhere.
3. In the DROP case, nobody answers. You wait. You knock again, louder. You wait longer. You knock a third time.
4. Eventually you leave. And here is the key point: you still do not know whether the office is empty, whether the person is ignoring you, or whether you were at the wrong building all along.
5. That is the whole cost of silence. The refusal carries no information.
6. Now think about why a building might do this on purpose. Someone walking the corridor knocking on every door learns nothing from silence. If every wrong door said “wrong floor”, they would map the building in a minute.

Where this comparison breaks:

7. A person knocking gives up in a minute. A computer keeps a strict schedule of retries with doubling gaps, and the total is predictable to the second.
8. And in a building the silence comes from one identifiable door. On a network, the silence could come from any of a dozen devices on the path, and you cannot tell which, because none of them left a trace.

PLAIN34.4.3 a worked example#

1. The same request against three different policies, with real timings.

Case 1: REJECT with a TCP reset
$ time curl -v https://example.test
*   Trying 203.0.113.9:443...
* connect to 203.0.113.9 port 443 failed: Connection refused
real  0m0.031s

Case 2: REJECT with ICMP administratively prohibited
$ time curl -v https://example.test
*   Trying 203.0.113.9:443...
* connect to 203.0.113.9 port 443 failed: No route to host
real  0m0.042s

Case 3: DROP, nothing sent back
$ time curl -v https://example.test
*   Trying 203.0.113.9:443...
* Connection timed out after 15001 milliseconds
real  0m15.004s

2. Case 3 is the reader’s session, character for character. curl -v printed Trying 20.207.73.82:443... and then produced nothing for 15 seconds.
3. Now the packets on the wire in case 3. The client alone is talking.

 0.000 192.168.0.24.54318 > 20.207.73.82.443: Flags [S]
 1.001 192.168.0.24.54318 > 20.207.73.82.443: Flags [S]
 3.005 192.168.0.24.54318 > 20.207.73.82.443: Flags [S]
 7.010 192.168.0.24.54318 > 20.207.73.82.443: Flags [S]
15.020 192.168.0.24.54318 > 20.207.73.82.443: Flags [S]
(no packets in the other direction, ever)

4. The gaps double: one second, two, four, eight. That doubling is the standard retransmission backoff. Nothing in the other direction is the signature.
5. Compare with the REJECT capture, which is two packets and over.

 0.000 192.168.0.24.54318 > 203.0.113.9.443: Flags [S]
 0.028 203.0.113.9.443 > 192.168.0.24.54318: Flags [R.]

PLAIN34.4.4 what is really happening inside#

1. A REJECT is an active act. The firewall generates a brand-new packet and sends it back to the source address.
2. For TCP it usually sends a reset, which is a TCP packet with the RST flag set. The client’s kernel sees it and immediately fails the connect() call with “connection refused”.
3. Alternatively it sends an ICMP error, a small control message. The most honest one is “communication administratively prohibited”, which literally means “a policy stopped this”.
4. The client’s kernel maps that ICMP error to an error code and the program reports something like “no route to host” or “host unreachable”.
5. A DROP does nothing at all. The packet is freed. No counter is incremented in anything the client can see. No log line reaches the client.
6. So the client’s TCP stack sits in a state called SYN_SENT and follows its retransmission schedule, doubling the wait each time.
7. There is a second, subtler reason silence is not always deliberate. A device under heavy load may rate-limit the errors it generates, so a genuine REJECT can be swallowed and look like a DROP.
8. And a black hole, where a router has no route and the error is filtered on the way back, produces the same silence with nobody having intended it.
9. That is why the honest reading of silence is: something on this path is not delivering, and it did not say why. Not: someone blocked me.

TECHNICAL34.4.5 the engineer’s version#

1. The client-visible symptom table. Learn this and you can diagnose from the error message alone.

2. The ICMP codes that carry a refusal are all type 3, destination unreachable: code 0 network unreachable, code 1 host unreachable, code 3 port unreachable, code 9 network administratively prohibited, code 10 host administratively prohibited, code 13 communication administratively prohibited. In ICMPv6 the equivalent is type 1 code 1.
3. iptables -j REJECT defaults to --reject-with icmp-port-unreachable, which is type 3 code 3. --reject-with tcp-reset sends an RST instead, and is the politest option for TCP because it is indistinguishable from a closed port. --reject-with icmp-admin-prohibited is the most informative and the least used.
4. In pf the two spellings are block drop and block return, and block return-rst or block return-icmp name the reply explicitly.
5. nmap names the three outcomes directly, and this is the cleanest vocabulary there is: open means a SYN-ACK came back, closed means an RST came back, filtered means nothing came back. A filtered result is a DROP by definition.
6. Timeouts differ by system, and the reader’s 15 seconds came from curl, not from the kernel:

7. Why operators choose DROP, stated fairly. It denies a scanner confirmation that a host exists. It forces the scanner to wait for a timeout on every probe, turning a one-second sweep into a multi-hour job. It generates no traffic, so it cannot be used to reflect or amplify an attack at a third party. And it does not reveal that a filtering policy exists at all.
8. Why DROP is expensive for everyone else. Every failure mode collapses into one symptom. A dropped SYN, a routing black hole, an asymmetric return path, a full conntrack table, a dead server behind a load balancer and a national filter all produce the identical client experience.
9. Cloud platforms make this worse by having no choice. AWS security groups and network ACLs both drop silently. There is no reject option. So a one-line security group mistake presents to the developer as an unexplained 15-second hang, which they will spend an hour blaming on DNS.
10. Now the tie back, stated precisely, separating proof from suggestion.

11. What is proven: packets left the machine towards 20.207.73.82:443 and nothing at all came back for 15 seconds, and the same request over a different access network succeeded. That is the DROP signature and it rules out the destination being down.
12. What is not proven: which device dropped, whether it was on the forward or the return path, whether it was policy or fault, and whether it was permanent. A silent drop leaves no evidence of its author, which is exactly what its author intended.

WORDS34.4.6 remember these#

1. DROP — discard and say nothing — a filtering action that frees the packet without generating any response.
2. REJECT — refuse and say so — a filtering action that generates a TCP RST or an ICMP destination-unreachable message.
3. Administratively prohibited — a policy said no — ICMP type 3 code 13, or ICMPv6 type 1 code 1.
4. Filtered — the scanner’s word for silence — an nmap port state meaning no response was received to any probe.
5. Black hole — traffic that vanishes with no error — a path that discards packets and returns no ICMP, whether by policy or by fault.

34.5 Middleboxes, and the end of end-to-end#

PLAIN34.5.1 in simple words#

1. The simple story of the internet says two computers talk directly, and the routers in between only forward.
2. That story is no longer true anywhere. Between you and any server sits a row of machines that change, inspect or block what passes.
3. These machines have a collective name: middleboxes.
4. Some rewrite addresses so many homes can share one public address. That is NAT, and the reader’s own trace showed private ISP addresses at hops 2 to 6.
5. Some quietly capture your web requests and answer from a local copy, without you configuring anything. Those are transparent proxies.
6. Some look inside packets to identify what application you are using, and act on it. That is deep packet inspection.
7. Some break your encryption on purpose, read the contents, and re-encrypt with their own certificate. That is TLS interception, common inside companies.
8. Some hijack your very first web request and show you a login page. That is a captive portal, the thing at every airport and hotel.
9. The pattern is always the same: something in the middle knows more about your traffic than the simple model allows.

PLAIN34.5.2 a picture in your head#

1. You post a letter to a friend and imagine it travelling sealed from your hand to theirs.
2. In reality the sorting office takes your envelope, puts it in a new envelope with the office’s own return address, and forwards it. That is NAT.
3. Another office keeps photocopies of popular letters and, if someone asks for one, sends the copy instead of forwarding the request. That is a transparent cache.
4. A third reads the address, the size, the thickness and the postmark to guess what the letter contains, without opening it. That is deep packet inspection on encrypted traffic.
5. A fourth actually opens the envelope, reads it, copies it into a fresh envelope, and reseals it with its own wax stamp. Your friend sees a sealed letter and may not notice the stamp changed. That is TLS interception.
6. A fifth holds every letter until you sign a form at the counter. That is a captive portal.

Where this comparison breaks:

7. In the post, opening a sealed letter is illegal and visible. On a network, TLS interception is legal, routine on company laptops, and technically invisible unless you look at the certificate chain.
8. And a letter passes maybe three offices. A packet from an Indian flat to a server in the United States passes twenty or more devices, several of which have inspection features switched on.

PLAIN34.5.3 a worked example#

1. Here is the real path from the reader’s session, annotated with which kind of box each hop is likely to be. Everything marked “likely” is inference, not proof.

hop  address              likely role
 1   192.168.0.1          home router: NAT, DHCP, DNS, firewall
 2   172.31.0.17          ISP access, private address
 3   137.97.29.249        ISP public edge
 4   172.26.22.235        ISP core, private address
 5-6 172.16.18.33 etc     ISP core, load balanced links
 7   104.44.196.187       Microsoft edge, Delhi
 8-9 104.44.55.163 etc    Microsoft, Mumbai
10-12 104.44.31.62 etc    Microsoft, Pune
13+  * * *                no replies

2. Two NATs are visible in that list, not one. The home router translates 192.168.0.24 to whatever address it holds, and the presence of RFC 1918 private addresses in the ISP core is consistent with carrier-grade NAT translating again.
3. So the source address that Microsoft’s edge saw was not the reader’s home router address either. Two rewrites, neither of them visible from the laptop.
4. To detect TLS interception on any machine, look at who signed the certificate. A public site should chain to a public certificate authority.

$ openssl s_client -connect example.com:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer
issuer=C=US, O=DigiCert Inc, CN=DigiCert TLS RSA SHA256 2020 CA1

# On an intercepted network you would instead see something like:
issuer=C=IN, O=ExampleCorp IT, CN=ExampleCorp Root CA

5. If the issuer is your employer, every HTTPS page you load on that machine is readable by your employer’s proxy. This is not a hack. It is a purchased product, and the certificate was installed by the device management tool.

PLAIN34.5.4 what is really happening inside#

1. A transparent proxy is put in the path by routing, not by configuration. The network is arranged so packets to port 80 or 443 arrive at the proxy instead of the real destination.
2. The proxy then opens its own connection onward and copies bytes between the two. Your operating system never knew.
3. Deep packet inspection on unencrypted traffic simply reads the payload. On encrypted traffic it cannot, so it reads what is still visible: the destination address, the port, the packet sizes and timing, and above all the server name sent in the clear at the start of a TLS handshake.
4. That server name field is the single most used censorship and filtering hook in the world today. It is plain text, it names the site, and it arrives before any encryption is in force.
5. TLS interception works by installing an extra root certificate on your machine. The proxy then generates a fresh certificate for every site you visit, signed by that root, and your browser trusts it because the root is in the trust store.
6. A captive portal usually does two things. It answers every DNS query with its own address, and it answers every plain HTTP request with a redirect to the login page.
7. That is why HTTPS sites fail with a certificate warning on a hotel network instead of showing the portal. The portal cannot forge a valid certificate, so the browser correctly refuses.
8. Modern operating systems work around this by deliberately making one plain HTTP request to a known address after joining a network, and checking whether the expected answer comes back.
9. If the answer is wrong, the system concludes a portal is present and opens the little login window automatically.

TECHNICAL34.5.5 the engineer’s version#

1. RFC 3234, February 2002, “Middleboxes: Taxonomy and Issues”, is the document that named and catalogued them. It lists twenty-two kinds and warns about exactly the failure modes we now live with.
2. The end-to-end argument comes from the 1984 paper “End-to-End Arguments in System Design” by Jerome Saltzer, David Reed and David Clark. Its claim is that functions belong at the endpoints unless the network can do them materially better.
3. Middleboxes cause ossification: the network becomes unable to accept new protocols because boxes drop what they do not recognize. Attempts to deploy new TCP options routinely fail for this reason.
4. This is a large part of why QUIC, standardized in RFC 9000 in May 2021, runs over UDP and encrypts almost its entire header. It is designed so middleboxes cannot parse it and therefore cannot ossify it.
5. Server Name Indication, the field carrying the site name in the clear, is defined in RFC 6066. Encrypted Client Hello is the proposed fix; it was still working through the IETF at the time of writing, and Cloudflare enabled it broadly in 2023. Treat it as emerging, not settled.
6. TLS interception at scale is a product category. Its risks are documented: a 2017 study by Durumeric and others, “The Security Impact of HTTPS Interception”, found many interception products weakened the connection they proxied.
7. Two famous incidents are worth knowing. Lenovo shipped consumer laptops with Superfish adware carrying a private root certificate, discovered in February
   2015. Kazakhstan required citizens to install a state root certificate in July 2019, and browser vendors blocked it.
8. Captive portal standards are recent. RFC 8910 and RFC 8908, both September 2020, define a DHCP and Router Advertisement option pointing at a machine readable portal API, so the operating system can be told rather than having to guess.
9. The probes each system uses are an implementation detail but worth knowing: Apple requests a tiny page from a Apple-operated host and expects the word “Success”. Android requests a URL expecting HTTP status 204 with an empty body. Windows uses a similar text probe.
10. Filtering techniques, and how each one looks from the client, which is how you tell them apart:

11. Applying that table honestly to the reader’s session: DNS returned a real address, so DNS blocking is ruled out. Nothing came back at all, and the failure was at the SYN, before any TLS ClientHello existed, so an SNI-based reset is ruled out too. What remains is address-level or port-level discarding, or a plain path fault. The evidence cannot separate those two.

WORDS34.5.6 remember these#

1. Middlebox — a device in the path that does more than forward — any on-path element that inspects, modifies or filters traffic.
2. Transparent proxy — an intercepting cache you did not configure — a proxy inserted by routing rather than by client settings.
3. Deep packet inspection — looking inside, not just at the label — examination of packet payloads or metadata to classify or control traffic.
4. TLS interception — legal, installed man-in-the-middle — decryption and re-encryption using a locally trusted root certificate.
5. Captive portal — the hotel login page — a network that intercepts the first request and redirects to an authentication page.
6. Ossification — the network refusing anything new — the inability to deploy new protocol behaviour because middleboxes drop unfamiliar traffic.

34.6 VPNs, what they do and what they do not#

PLAIN34.6.1 in simple words#

1. A VPN, a virtual private network, does one mechanical thing: it wraps your traffic inside another connection to a server somewhere else, and lets that server send it onward.
2. Your packets go out looking like ordinary encrypted traffic to one address. Inside them are your real packets to their real destinations.
3. The VPN server unwraps them, sends them on with its own address, receives the replies, wraps them, and sends them back to you.
4. Two things change. First, your local network and your internet provider can no longer see where you are going. They only see the VPN server.
5. Second, the websites you visit see the VPN server’s address, not yours, and your path to them now starts wherever that server is.
6. There are two main uses. One joins two whole offices into one network. The other lets a single laptop join a network it is not physically on.
7. The important honest point comes early: a VPN moves your trust. It does not remove it. Whoever runs the VPN can see what your provider used to see.
8. It is not anonymity. It does not stop tracking. It does not make you safe. It changes who is in a position to watch.

PLAIN34.6.2 a picture in your head#

1. Imagine you write letters, but you do not want your local post office to know who you write to.
2. So you put your addressed letter inside a second envelope and post it to a forwarding service in another city. The local post office only sees “letter to the forwarding service”.
3. The service opens the outer envelope, posts your real letter with its own return address, receives the reply, and posts it back to you inside another outer envelope.
4. Your local post office learns nothing about your correspondents. Your correspondents learn nothing about your address.
5. The forwarding service, of course, learns everything. It sees every real address and, if the inner letter is not itself sealed, every word.
6. That is the trade in one paragraph. You have not become invisible. You have chosen a different party to be visible to.

Where this comparison breaks:

7. The inner letter is usually already sealed. Almost all web traffic is HTTPS, so the VPN provider sees which sites you visit and how much you transfer, but not the contents of pages.
8. And the forwarding service adds distance. If you are in India and the service is in Germany, every letter now travels to Germany first, which is why a VPN often makes things slower even when it works perfectly.

PLAIN34.6.3 a worked example#

1. Here is the reader’s failing path, and the same request with a full-tunnel VPN to a server in Singapore.

Without a VPN:
laptop -> home router -> ISP core -> ISP edge ->
Microsoft Delhi -> Mumbai -> Pune -> nothing

With a full-tunnel VPN:
laptop -> home router -> ISP core -> ISP edge ->
VPN server in Singapore -> Microsoft Singapore edge ->
github.com  (works)

2. Everything after the VPN server is a different path across a different set of networks. If the device that dropped the packets sat on the old path and not the new one, the problem simply does not occur.
3. That is why “turn on a VPN” so often appears to fix an unexplained network fault. It is not magic. It is a different route.
4. Now look at what the tunnel looks like on the machine.

$ ifconfig utun4
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1420
       inet 10.7.0.6 --> 10.7.0.6 netmask 0xffffff00

$ netstat -rn | head -6
Destination    Gateway         Flags  Netif
0/1            10.7.0.5        UGSc   utun4
default        192.168.0.1     UGScg  en0
128.0/1        10.7.0.5        UGSc   utun4
203.0.113.77   192.168.0.1     UGHS   en0

5. Read those routes carefully, because they contain the cleverest trick in VPN software. The original default route through 192.168.0.1 is still there, untouched.
6. Instead, two routes were added: 0.0.0.0/1 and 128.0.0.0/1. Together they cover the entire address space, and both are more specific than the default route, so both win.
7. The single exception is the fourth line: a direct route to the VPN server’s own public address through the real gateway. Without that, the tunnel’s own packets would try to go through the tunnel, which cannot work.
8. Note the MTU of 1420 rather than 1500. The wrapping costs bytes, so the inner packets must be smaller.

PLAIN34.6.4 what is really happening inside#

1. The VPN client creates a virtual network interface. On macOS these are called utun, on Linux tun or wg0, on Windows a virtual adapter.
2. To the operating system it looks like an ordinary network card, so ordinary routing rules can send traffic to it.
3. Anything the kernel routes into that interface is handed to the VPN software as a raw IP packet.
4. The software encrypts that packet, wraps it in a normal UDP or TCP packet addressed to the VPN server, and sends it out of the real interface.
5. So the same physical link now carries an outer packet, whose payload is a whole inner packet. This is encapsulation, and it is the same idea as putting one envelope inside another.
6. At the far end, the server decrypts, extracts the inner packet, and forwards it into the internet from its own address, keeping a translation record so it can return the reply to you.
7. Split tunnelling means only some routes point at the virtual interface. Work traffic goes through the tunnel, everything else goes out normally.
8. Full tunnelling means every route points at the virtual interface, so all traffic goes through the tunnel.
9. Split is faster and cheaper. Full is safer for a hostile local network, because nothing at all leaks to the local link.

TECHNICAL34.6.5 the engineer’s version#

1. Deployment shapes: site-to-site joins two networks with a permanent gateway-to-gateway tunnel, usually IPsec, and users are unaware of it. Remote access connects one device into a network, and the user runs a client.
2. The protocol comparison, with the facts that decide real choices:

3. IPsec is a family, not one protocol. RFC 4301, December 2005, gives the architecture. ESP, RFC 4303, does the encryption. IKEv2, RFC 7296, October 2014, negotiates keys. NAT traversal wraps ESP in UDP port 4500 per RFC 3948. MOBIKE, RFC 4555, lets a tunnel survive an address change, which is why IKEv2 is the usual choice on phones.
4. OpenVPN was first released on 13 May 2001 by James Yonan. It runs in user space over OpenSSL or mbedTLS, defaults to UDP port 1194, and can be forced over TCP port 443 to look like ordinary web traffic, at the cost of TCP inside TCP, which behaves badly under loss.
5. WireGuard was written by Jason A. Donenfeld, first released publicly in 2016 and merged into the Linux kernel in version 5.6, released 29 March 2020. It is roughly 4,000 lines of kernel code, against hundreds of thousands for a full IPsec plus OpenVPN stack. Small code means fewer places for bugs and an auditable design.
6. WireGuard’s speed comes from three choices. It runs in the kernel. It has no cipher negotiation at all: the algorithms are fixed at Curve25519, ChaCha20-Poly1305, BLAKE2s and HKDF, using the Noise protocol framework. And it is stateless in the sense that there is no session setup handshake per connection, only a periodic key exchange.
7. The trade-off is real and worth stating: with no negotiation, upgrading the cryptography means replacing the protocol version everywhere at once. The designers consider that a feature. Some enterprise architects consider it a problem. Both positions are defensible.
8. PPTP should never be used. RFC 2637, July 1999, is informational, and its authentication, MS-CHAPv2, was broken publicly by Moxie Marlinspike and David Hulton at Defcon in July 2012, reducing the effective work to a single DES key. Apple removed PPTP support in macOS Sierra 10.12 and iOS 10 in 2016.
9. L2TP alone, RFC 2661, August 1999, provides no encryption whatsoever. It is always paired with IPsec, per RFC 3193, and the pairing is often misconfigured.
10. About the reader’s utun interfaces. utun is macOS’s user tunnel kernel control interface, and a utun device is created by any process that asks the kernel for one. macOS itself creates several for its own features, so seeing utun0 through utun3 on a stock machine with no VPN is normal. Their presence is not evidence that a VPN was running.
11. To tell a real VPN from a system interface: a live tunnel has a routable address, at least one route pointing at it, and traffic counters that move. Check with ifconfig utunN, netstat -rn, scutil --nc list for configured VPN services, and netstat -ibn for byte counters.
12. What a commercial VPN genuinely protects against: a hostile or badly-run local network, such as a shared cafe or hotel WiFi; your access provider’s visibility into which sites you reach; and geographic restrictions based on the apparent country of your address.
13. What it does not do, stated without softening. The provider now sees exactly what your provider used to see, and you have less legal recourse against them. It is not anonymity, because your account, your payment and your behaviour identify you. It does not stop cookies, browser fingerprinting, logins or advertising identifiers, which are all above the network layer. It does not protect a compromised device. And a “no logs” claim is a promise, not a mechanism.
14. Applied to the reader’s fault: a full-tunnel VPN would probably have made the symptom disappear, because the path after the VPN server is entirely different. That is a workaround. It adds latency, costs money, moves trust to a third party, and leaves the broken path broken for everybody else on that provider. The fix is to report the fault with the evidence from Chapter 29.

WORDS34.6.6 remember these#

1. VPN — a private path built over a public one — an encrypted tunnel encapsulating IP traffic between a client or site and a gateway.
2. Encapsulation — a packet inside a packet — carrying a complete inner datagram as the payload of an outer one.
3. Split tunnelling — only some traffic goes through — a route configuration sending selected prefixes over the tunnel interface.
4. Full tunnelling — everything goes through — installing 0.0.0.0/1 and 128.0.0.0/1 so all traffic prefers the tunnel.
5. utun — the macOS virtual tunnel device — a kernel control interface providing point-to-point tunnel interfaces to user-space processes.
6. MOBIKE — a tunnel that survives changing networks — IKEv2 mobility and multihoming, RFC 4555.

34.7 Proxies, forward and reverse#

PLAIN34.7.1 in simple words#

1. A proxy is a machine that makes a request on your behalf, or answers one on somebody else’s behalf. Two jobs, two names, endlessly confused.
2. A forward proxy sits next to the clients. You send it your request, it fetches the page, it gives you the result.
3. A reverse proxy sits next to the servers. The public sends requests to it, and it decides which real server behind it should handle each one.
4. The memory rule, worth writing down: a forward proxy hides the client from the server. A reverse proxy hides the server from the client.
5. Forward proxies exist for filtering, caching, and control. Schools and companies use them.
6. Reverse proxies exist for spreading load, terminating encryption, caching, and routing by path or hostname. Almost every website has one.
7. You have used a reverse proxy every day of your life without knowing. You have probably only used a forward proxy at work or at school.
8. A proxy is not a VPN. A proxy handles particular connections for particular programs. A VPN handles all traffic for the whole machine.

PLAIN34.7.2 a picture in your head#

1. A forward proxy is a purchasing department. You are not allowed to buy directly. You submit a request, they buy the item, they hand it to you.
2. They can refuse certain purchases, keep a record, and reuse a bulk item they already have in stock instead of buying it again.
3. The supplier only ever deals with the purchasing department, and never learns which employee wanted the item.
4. A reverse proxy is a receptionist at a large company. Every visitor comes to the desk. The receptionist decides which department handles the visitor and walks them there.
5. Visitors never learn how many departments exist, where they sit, or that one of them was replaced last night.

Where this comparison breaks:

6. A receptionist handles one visitor at a time. A reverse proxy handles tens of thousands of connections at once and can send one visitor’s request to two departments and merge the answers.
7. And the purchasing department knows what you bought. A forward proxy handling HTTPS traffic mostly does not, because it only opens a blind tunnel, unless it is also intercepting your encryption.

PLAIN34.7.3 a worked example#

1. The two shapes, drawn.

FORWARD PROXY (near the client)

 you -> proxy -> internet -> the site
        ^ you configured this
        ^ the site sees the proxy's address

REVERSE PROXY (near the server)

 you -> internet -> reverse proxy -> app server 1
                                  -> app server 2
                    ^ you never configured this
                    ^ you see one address and one name

2. Now a forward proxy in action for a plain HTTP request. The request line carries the whole address, which it never does otherwise.

GET http://example.com/page.html HTTP/1.1
Host: example.com
Proxy-Connection: keep-alive

3. For an encrypted request the proxy cannot do that, so the client asks for a tunnel instead, using a special method.

CONNECT example.com:443 HTTP/1.1
Host: example.com

HTTP/1.1 200 Connection established

(from here the proxy copies bytes blindly in both
 directions; the TLS handshake happens end to end)

4. The proxy therefore learns the hostname and port from the CONNECT line, and the number of bytes and the timing, but not the page contents.
5. Configuring one from the shell uses environment variables, which are a convention that most tools follow and a few ignore.

export http_proxy=http://proxy.corp.example:3128
export https_proxy=http://proxy.corp.example:3128
export no_proxy=localhost,127.0.0.1,.corp.example
curl -v https://example.com/     # now goes via CONNECT
curl --proxy socks5h://127.0.0.1:1080 https://example.com/

6. The h in socks5h matters: it means the proxy resolves the hostname, so your DNS queries do not leak to your local resolver.

PLAIN34.7.4 what is really happening inside#

1. With a forward proxy, your program opens a TCP connection to the proxy, not to the destination. There are two separate connections and the proxy joins them.
2. For plain HTTP the proxy fully understands the request and can cache, rewrite or block it.
3. For HTTPS the proxy is asked to become a pipe. After it answers “200 Connection established” it stops understanding anything and just copies bytes.
4. A SOCKS proxy is simpler and lower down. It does not understand HTTP at all. It just carries any TCP connection, and optionally UDP, to wherever you say.
5. That is why SOCKS works for SSH, databases, game protocols and anything else, while an HTTP proxy only works for HTTP and CONNECT tunnels.
6. With a reverse proxy, the public name and address belong to the proxy. It terminates the TCP connection and the encryption, reads the request, and decides.
7. It can then route on the hostname, on the path, on a header, or on a cookie, and it opens its own connection to a chosen backend server.
8. Because it terminates encryption, it can add caching, compression, request limiting and attack filtering in one place, for every backend behind it.
9. The cost is that the backend no longer sees the real client address, so the proxy must pass it on in a header, and the backend must be configured to trust that header only from the proxy.

TECHNICAL34.7.5 the engineer’s version#

1. The distinction in one table:

2. The CONNECT method is defined in RFC 9110, June 2022, section 9.3.6, and was earlier described in RFC 2817, May 2000. A proxy that permits CONNECT to arbitrary ports is an open relay and will be abused, so real proxies restrict it to 443 and a short list.
3. SOCKS version 4 was written by David Koblas around 1992. SOCKS5 is RFC 1928, March 1996, and adds authentication, IPv6, UDP association and, critically, remote name resolution via the domain-name address type.
4. Proxy auto-configuration files are a JavaScript function named FindProxyForURL(url, host) returning a string such as DIRECT or PROXY host:port. Netscape introduced them in 1996. Web Proxy Auto-Discovery locates the file via DHCP option 252 or a DNS name, and has been a recurring security weakness because whoever answers first wins.
5. Proxy versus VPN, at the packet level. A proxy operates at layer 5 to 7: your application makes a new TCP connection to the proxy, and the proxy makes a separate one onward. Only applications that know about the proxy use it. ICMP, DNS and everything else bypass it. A VPN operates at layer 3: whole IP packets are encapsulated, so every protocol and every program is covered whether it knows or not.
6. Reverse proxy software, with real origins. nginx was written by Igor Sysoev and first released publicly on 4 October 2004, to solve the C10K problem of ten thousand concurrent connections. HAProxy was written by Willy Tarreau, first released in 2001, and is still the reference for layer 4 and layer 7 load balancing. Envoy was open-sourced by Lyft on 20 September 2016 and became the data plane of service meshes such as Istio. Caddy, by Matt Holt, first released in 2015, was the first popular server to obtain and renew TLS certificates automatically.
7. Squid, the classic forward caching proxy, dates from 1996 and descends from the Harvest project.
8. Where reverse proxies sit in a real deployment, outermost first: a CDN edge, then a cloud load balancer, then an ingress reverse proxy such as nginx or Envoy, then the application. Four layers of proxy is normal, and each one can add its own timeout, its own header rewriting and its own failure mode.
9. Client address propagation uses X-Forwarded-For, a convention from Squid, or the standardized Forwarded header from RFC 7239, June 2014. Trusting these headers from anything other than your own proxy is a well-known vulnerability, because a client can simply send them.
10. Observation: curl -v --proxy ... shows the CONNECT exchange, nginx -T prints the full resolved configuration, and echo "show stat" | socat /var/run/haproxy.sock stdio dumps HAProxy counters.

WORDS34.7.6 remember these#

1. Forward proxy — fetches things for clients — a proxy configured by the client that hides clients from origin servers.
2. Reverse proxy — answers for servers — a proxy owned by the site that terminates client connections and dispatches to backends.
3. CONNECT — ask a proxy for a blind pipe — the HTTP method that establishes a TCP tunnel through a proxy, used for HTTPS.
4. SOCKS — a general-purpose connection relay — a layer 5 proxy protocol carrying arbitrary TCP and UDP, defined in RFC 1928.
5. PAC file — a script that picks the proxy — a JavaScript FindProxyForURL function evaluated by the client per request.
6. X-Forwarded-For — the original client address, passed along — a convention header, standardized as Forwarded in RFC 7239.

34.8 Tunnelling in general#

PLAIN34.8.1 in simple words#

1. Tunnelling means putting one packet inside another packet.
2. The outer packet is addressed to a machine that knows how to unwrap it. The inner packet is addressed to where you actually wanted to go.
3. The network in between reads only the outer packet. It never looks at the inner one and does not need to understand it.
4. That is the whole trick. Everything called a tunnel is a version of it.
5. There are three honest reasons to tunnel. To carry something the network refuses to carry. To hide what you are carrying. To make two far-apart networks behave like one.
6. Because the outer packet needs its own header, a tunnel always makes packets bigger, so every tunnel eats a little of your usable space.
7. A tunnel is not automatically private. Wrapping and encrypting are different jobs. Some tunnels encrypt, many do not.
8. SSH gives you three ready-made tunnels for free. They are among the most useful tools a developer owns and almost nobody is taught them.
9. There are also services whose only job is to give a machine behind a home router a public address, by having that machine dial out first.

PLAIN34.8.2 a picture in your head#

1. You want to send a letter into a building that does not accept post from outside.
2. So you seal your letter, write your colleague’s internal room number on it, and put it inside a bigger envelope addressed to the reception desk.
3. The postal service only reads the outer envelope. It delivers to reception.
4. Reception opens the outer envelope, sees an internal room number, and passes the inner letter along the internal mail system.
5. Reception is the tunnel endpoint. The outer envelope is the encapsulation. The inner letter is your real packet.
6. The outer envelope also weighs something, so a fixed weight limit now holds slightly less of your actual letter.

Where this comparison breaks:

7. The post office does not silently destroy an envelope that is one gram too heavy. Networks do exactly that when a packet is too big and the message saying so is filtered. You get silence, not a bounce.
8. And reception can read your inner letter unless you sealed it. Wrapping alone gives you no privacy at all. Only encryption does.

PLAIN34.8.3 a worked example#

1. SSH has three forwarding modes. Learn the letters and you own them for life.
2. -L is local forwarding. It opens a port on your machine and sends anything arriving there out through the SSH server.

# reach a database that only the bastion host can see
ssh -N -L 5433:db.internal:5432 user@bastion.example.com
# now, in another shell, on your own laptop:
psql -h 127.0.0.1 -p 5433 -U app appdb

3. Read -L 5433:db.internal:5432 as: listen on my port 5433, and from the SSH server open a connection to db.internal port 5432.
4. -R is remote forwarding. It is the mirror image. It opens a port on the SSH server and sends anything arriving there back to you.

# let a colleague see the site running on your laptop
ssh -N -R 9000:localhost:3000 user@public.example.com
# they open port 9000 on public.example.com and reach
# port 3000 on your laptop

5. By default that remote port only listens on the server’s own loopback. To let other people reach it, the server needs GatewayPorts yes in its sshd_config. That is a deliberate safety default.
6. -D is dynamic forwarding. It turns your SSH connection into a SOCKS proxy, so any program can send any connection through it.

ssh -N -D 1080 user@bastion.example.com
curl --proxy socks5h://127.0.0.1:1080 https://intranet.example/

7. -N means “do not run a command on the far end, just forward”. -f puts it in the background. Together they are the usual pair.
8. The three modes drawn, because the arrows are the whole point.

-L LOCAL FORWARD   (you open the door on your side)

  your app -> 127.0.0.1:5433 -[ssh]-> bastion -> db:5432

-R REMOTE FORWARD  (you open the door on their side)

  visitor -> public:9000 -[ssh]-> your laptop -> :3000

-D DYNAMIC         (a SOCKS proxy, any destination)

  any app -> 127.0.0.1:1080 -[ssh]-> bastion -> anywhere

9. Now the outside-in version. ngrok and Cloudflare Tunnel do the -R trick as a product, with a public name and a certificate.

ngrok http 3000
cloudflared tunnel --url http://localhost:3000

10. Both print a public hostname. Both work from behind a home router with no port forwarding, because the agent dialled out and the router allowed that outbound connection, exactly as it allows any web request.

PLAIN34.8.4 what is really happening inside#

1. SSH forwarding is not really packet-in-packet. It is connection-in-connection, and the difference matters.
2. Your ssh client opens a normal listening socket on your machine, on the port you asked for.
3. When a program connects to that port, the client opens a new logical channel inside the one existing SSH connection and says “please connect to this host and port”.
4. The SSH server makes a brand new TCP connection from itself to the target, and then copies bytes between that connection and the channel.
5. So the target sees a connection coming from the SSH server, not from you. Its logs show the bastion’s address. Its firewall rules must allow the bastion.
6. Many channels share one SSH connection, so many forwards cost one TCP connection and one handshake.
7. A real network tunnel such as VXLAN or GRE works one level lower. It takes a whole packet, headers and all, and puts it in the payload of a new packet.
8. The receiving end strips the outer header and injects the inner packet as if it had just arrived on a wire. Nothing above it knows a tunnel happened.
9. That is why a layer 3 tunnel carries every protocol, and an SSH forward carries only the TCP connections you named.
10. In both cases the outer header takes space. If the original packet was already full size, the wrapped packet is now too big for the path, and it must be fragmented, refused, or dropped.

TECHNICAL34.8.5 the engineer’s version#

1. Encapsulations you will meet, with their real identifiers and overhead over IPv4.

2. GRE, Generic Routing Encapsulation, is RFC 2784, March 2000, updated by RFC 2890, and replaced the earlier RFC 1701 of October 1994. It has no encryption and no authentication at all. It is a plain wrapper, still used between routers and by DDoS scrubbing providers to return cleaned traffic.
3. VXLAN, Virtual eXtensible LAN, is RFC 7348, August 2014. It puts a whole Ethernet frame inside UDP. Its 24-bit VXLAN Network Identifier gives 16,777,216 segments, against 4,094 usable VLANs, which is the reason it exists. Cloud overlays and Kubernetes network plugins use it constantly.
4. Geneve, RFC 8926, November 2020, is the newer extensible version, and is what the AWS Gateway Load Balancer speaks.
5. MTU arithmetic, the part that bites. On a 1500-byte Ethernet path, a VXLAN tunnel leaves 1450 bytes for the inner frame. WireGuard’s wg-quick sets the tunnel interface MTU to 1420 by default for the same reason.
6. When encapsulation shrinks the path and the ICMP “fragmentation needed” message is filtered, Path MTU Discovery fails silently and you get the classic signature: the handshake works, small responses arrive, and the first large response hangs forever. The fix is usually TCP MSS clamping, for example iptables -t mangle -A FORWARD -p tcp --syn -j TCPMSS --clamp-mss-to-pmtu.
7. SSH port forwarding is defined in RFC 4254, January 2006, the SSH Connection Protocol, as channels of type direct-tcpip for -L and forwarded-tcpip for -R. SOCKS5 dynamic forwarding is RFC 1928, March 1996. OpenSSH itself was forked from Tatu Ylonen’s SSH 1.2.12 by the OpenBSD project and first released in December 1999.
8. Do not tunnel TCP inside TCP if you can avoid it. Two retransmission timers stack, and a lost packet triggers both, which is why ssh -w and TCP-based VPNs collapse on lossy links. UDP-based tunnels do not have this problem.
9. Outbound-dialled tunnel services, with real dates. ngrok was created by Alan Shreve and released in 2013. Cloudflare launched Argo Tunnel in 2018 and renamed it Cloudflare Tunnel in 2021; its agent is cloudflared and it connects outbound over QUIC or HTTP/2. Tailscale added Funnel in 2022.
10. The security consequence is blunt and worth stating once. A public tunnel to localhost exposes that service to the entire internet, including automated scanners that find new hostnames within minutes. Put authentication in front of it before you start it, not after.
11. Observation commands: ssh -v prints each channel as it opens, ip link show type vxlan and bridge fdb show inspect a VXLAN interface on Linux, tcpdump -ni any udp port 4789 shows the encapsulated frames, and ip route get 10.0.0.5 tells you whether a packet will take the tunnel.

WORDS34.8.6 remember these#

1. Tunnel — a packet carried inside another packet — encapsulation of one protocol’s payload within another protocol for transit.
2. Encapsulation — the wrapping itself — prepending an outer header so intermediate routers forward on the outer address only.
3. Local forward — a door on your side — ssh -L, a listening socket on the client relayed to a target reached from the server.
4. Remote forward — a door on their side — ssh -R, a listening socket on the server relayed back to a target reached from the client.
5. Dynamic forward — a proxy, not a fixed pair — ssh -D, a local SOCKS5 listener whose destinations are chosen per connection.
6. VXLAN — Ethernet inside UDP — RFC 7348 overlay with a 24-bit VNI and 50 bytes of overhead, UDP port 4789.
7. GRE — a plain wrapper with no encryption — RFC 2784, IP protocol 47, 24 bytes of overhead.
8. MSS clamping — telling both ends to use smaller pieces — rewriting the TCP maximum segment size option to fit the tunnel’s path MTU.

34.9 CDNs and anycast in depth#

PLAIN34.9.1 in simple words#

1. A CDN, a content delivery network, is a large set of computers placed close to users, holding copies of somebody else’s files.
2. The point is distance. A file served from a machine 40 kilometres away arrives far sooner than the same file from 13,000 kilometres away, and no amount of money changes that.
3. The site that owns the content is called the origin. The CDN machines are called edges or points of presence.
4. A CDN can copy anything that is the same for everybody: images, style sheets, scripts, fonts, video pieces, downloads.
5. A CDN cannot copy things that are different for each person, such as your account page or your shopping basket, unless the site takes special care.
6. There are two ways to fill a CDN. Push, where you upload files to it in advance. Pull, where it fetches a file from the origin the first time anyone asks, and keeps it.
7. Nearly everything today is pull, because it needs no work from you.
8. Anycast is a separate idea that CDNs also use. It means one address exists in many places at once, and the network delivers you to whichever copy is closest by its own reckoning.
9. Anycast and DNS steering both send different users to different places, and people mix them up constantly. One works in the routing system, the other in the naming system. We will separate them properly at the end.

PLAIN34.9.2 a picture in your head#

1. Think of a popular book and a national library system.
2. The origin is the one library that owns the original manuscript.
3. Rather than making every reader travel to that one building, the system prints copies and puts them in every town branch.
4. The first reader in your town asks for the book, the branch does not have it, so the branch orders one copy from the central library and keeps it on the shelf.
5. Every reader after that is served in two minutes from the local shelf. That is a cache hit.
6. The branch is told how long the copy stays valid. After that it must check with the central library before lending it again.
7. Now the anycast part. Imagine every branch has the same street address painted on the door, and the road signs in each town point to the nearest one. You always drive to “1 Library Road” and always arrive somewhere local.

Where this comparison breaks:

8. Books do not change under you. Web content does, and a stale copy on a shelf can be actively wrong, which is why purging and versioned file names exist.
9. Road signs do not change while you are driving. Internet routes do, and if they change mid-journey an anycast connection can arrive at a branch that has never heard of you and be turned away.
10. And a library branch does not run your code. A modern edge does, which is the part of this comparison that has aged the worst.

PLAIN34.9.3 a worked example#

1. One request for one image, walked end to end, with a cold cache.

 you, Mumbai                          origin, Virginia
     |                                       ^
     | 1. DNS: name -> nearest edge          |
     v                                       |
 [ EDGE POP, Mumbai ]                        |
     | 2. TLS finishes here, ~30 ms away     |
     | 3. build cache key, look it up        |
     |      HIT  -> send bytes, done         |
     |      MISS -> ask the shield           |
     v                                       |
 [ SHIELD POP, one per origin ] --- 4. ------+
     5. shield stores it, sends it to edge
     6. edge stores it, sends it to you

2. Step 3 needs a cache key. By default it is built like this.

key = scheme + host + path + query string
    = https + img.example.com + /logo.avif + ?v=7

3. Step 4 only happens on a miss. With a shield in front of the origin, 200 edges missing the same file cause one origin request, not 200.
4. What the origin sends back, and what each header does.

HTTP/1.1 200 OK
Content-Type: image/avif
Cache-Control: public, s-maxage=600, max-age=60
ETag: "9f2a1c"
Vary: Accept-Encoding

5. max-age=60 tells your browser to keep it for 60 seconds. s-maxage=600 tells shared caches such as the CDN to keep it for 600 seconds. Shared caches prefer s-maxage when both are present.
6. Vary: Accept-Encoding adds that request header to the cache key, so the compressed and uncompressed versions are stored separately.
7. What the edge sends you, with two extra headers of its own.

HTTP/1.1 200 OK
Age: 143
X-Cache: HIT
ETag: "9f2a1c"

8. Age: 143 means this copy has been in caches for 143 seconds, so it has 457 seconds of its 600 left. X-Cache: HIT is a convention, not a standard; every CDN spells it differently.
9. Now the timing, using round-trip times we will justify in section 34.10.

10. That table is the entire business case for CDNs in three rows.

PLAIN34.9.4 what is really happening inside#

1. The name you type does not resolve to the origin. It resolves, one way or another, to a CDN address near you.
2. There are only two mechanisms for that, and this is the crux of the whole section.
3. Mechanism one, DNS steering. The authoritative name server looks at who is asking and returns a different address to different askers. Different users get different addresses.
4. Mechanism two, anycast. Every user gets the same address, and the routing system itself carries each user to a different physical machine.
5. Most large CDNs use both: anycast for the edge addresses, and DNS to hand out which anycast address family or which service you land in.
6. Once you reach the edge, the edge terminates your TCP connection and your encryption. That is why the handshake feels fast: it finished 30 ms away, not 200 ms away.
7. The edge computes the cache key, looks in memory, then on local flash. A hit is answered immediately.
8. A miss is not sent straight to the origin. It goes to a parent or shield node, and identical concurrent misses are collapsed into one upstream request so the origin is never stampeded.
9. The connection from edge to origin is usually already open, already warmed up, and often runs over the CDN’s own private network rather than the public internet.
10. Stale content is handled by time, not by magic. When the timer expires the edge revalidates with a conditional request, and a 304 Not Modified reply costs one small round trip instead of a whole file.
11. If you need something gone before its timer expires, you purge it, which is a message pushed out to every edge holding it.
12. Edge compute means the edge can also run small programs of yours before or after the cache lookup, so decisions that used to need the origin now happen 30 ms from the user.

TECHNICAL34.9.5 the engineer’s version#

1. Cacheability rules, from RFC 9111, June 2022, which replaced RFC 7234. A shared cache may store a response if the method is cacheable, the status is understood, and no directive forbids it.
2. The directives that decide it, and what they really mean.

3. stale-while-revalidate and stale-if-error come from RFC 5861, May 2010. The first serves the old copy while fetching a new one in the background. The second serves the old copy when the origin is down, and it has saved more outages than any dashboard.
4. Validators: ETag with If-None-Match, and Last-Modified with If-Modified-Since, producing 304 Not Modified with no body.
5. Cache key control is where real hit rates are won. Strip tracking query parameters, normalize the order of the remaining ones, and never send Vary: User-Agent, which multiplies your object count by thousands.
6. Purging, in increasing order of danger: by exact URL, by surrogate key or tag, by prefix, and purge-everything. Fastly calls the tag header Surrogate-Key, Cloudflare calls it Cache-Tag, Akamai uses Edge-Control and its own cache tags. Purge-everything on a busy site sends every edge to the origin at once and can take the origin down.
7. The better pattern is to never purge. Put a content hash in the file name, serve it with max-age=31536000, immutable, and change the name when the content changes. One year is the maximum value RFC 9111 encourages.
8. Edge compute, with real dates. Cloudflare Workers were announced on 29 September 2017, running V8 isolates rather than containers. AWS Lambda@Edge became generally available in July 2017 and CloudFront Functions in May 2021. Fastly’s Compute product, built on WebAssembly, reached general availability in 2020.
9. Deployment shapes differ, and the industry genuinely disagrees about which is better. These are company-published figures as of 2026, which is to say marketing claims rather than audited counts, and they change constantly.

10. The argument: many small POPs sit closer to users; few large POPs hold more objects each, so they have higher hit rates and better shielding. Both camps are right about their own metric.
11. History: Akamai was founded in 1998 by Tom Leighton and Daniel Lewin at MIT, out of work on consistent hashing, and launched service in 1999. Cloudflare was founded in 2009 by Matthew Prince, Lee Holloway and Michelle Zatlyn and launched publicly at TechCrunch Disrupt in September 2010. Fastly was founded in 2011 by Artur Bergman.
12. Now anycast, properly. Anycast was described in RFC 1546, November 1993, by Craig Partridge, Trevor Mendez and Walter Milliken. Operational guidance is RFC 4786, December 2006, which is BCP 126.
13. The mechanism is entirely BGP. The same prefix, say a /24, is announced from many autonomous systems or many locations of one autonomous system. Every router in the world independently picks one best path to that prefix using its usual decision process: local preference, then AS path length, then origin, then MED, then eBGP over iBGP, then IGP cost.
14. Therefore “nearest” in anycast means nearest in BGP policy terms, not in kilometres. A site two countries away with a shorter AS path will win over a site in your city that your ISP reaches through three transit providers.
15. Why UDP and DNS love anycast: a DNS query is one packet out, one packet back, with no state at the server. If the next query lands at a different site, nothing is lost. The 13 root server identities are served this way from more than 1,900 instances worldwide, a figure that changes monthly.
16. Why TCP needs care: every packet of a flow must reach the same site, because only that site holds the transmission control block. A packet arriving at a site with no matching socket gets a RST and the connection dies.
17. In practice it works because routes are stable for minutes to hours, and because equal-cost multipath hashing inside a network is per-flow, keyed on the five tuple, so all packets of one connection follow one path.
18. A session sticks to one site for exactly as long as the routing decision at every hop stays the same. It is not pinned, tracked or guaranteed. It is merely stable, which is a weaker and more honest word.
19. When routing changes mid-connection, the flow is delivered to a different POP, which resets it. The user sees a broken download or a stalled upload. Short connections almost never notice. Long ones do.
20. Mitigations used in production: keep announcements stable and drain sites gracefully rather than withdrawing routes instantly, use consistent hashing in the load balancers behind the anycast address, and for QUIC use the connection ID, which lets a new front end recognize an existing connection and route it correctly, as specified in RFC 9000, May 2021.
21. A common hybrid is anycast for the first contact and a unicast address for the long transfer, which gives fast site selection and stable long flows.
22. Anycast inherits BGP’s weaknesses. A route hijack sends your users to somebody else’s machine. Real cases: Pakistan Telecom’s accidental global hijack of YouTube on 24 February 2008, and the deliberate hijack of Amazon Route 53 address space on 24 April 2018 used to steal cryptocurrency from MyEtherWallet users. RPKI route origin validation exists to reduce this.
23. The final separation, which is the most confused pair in networking.

24. Two more differences that matter in an incident. DNS steering can be overridden by a client that ignores TTLs or pins an address, and there are many such clients. Anycast cannot be overridden by the client at all.
25. And DNS steering sees the resolver, not the user. The reader’s own session used 1.1.1.1, Cloudflare’s public resolver, rather than the ISP’s. With DNS steering that can put you in the wrong region, unless the resolver forwards a truncated client subnet using EDNS Client Subnet, RFC 7871, May
    2016. Cloudflare’s resolver deliberately does not send client subnet by default, for privacy; Google’s public resolver does.
26. In the reader’s case the answer was still local: github.com resolved to 20.207.73.82, an address in a Microsoft-owned range serving India, and the traceroute entered Microsoft’s network at a Delhi site named ae66-0.del01-96cbe-1b.ntwk.msn.net. The evidence supports local steering working correctly. It does not prove which mechanism produced it.
27. Observation: dig +short github.com from two networks shows steering, dig CHAOS TXT id.server @1.1.1.1 or hostname.bind reveals which anycast instance answered you, and curl -sI plus a look at Age, X-Cache or CF-Cache-Status tells you whether you got a cached copy.

WORDS34.9.6 remember these#

1. CDN — machines near you holding copies of a site’s files — a distributed caching and delivery network fronting an origin.
2. Origin — the real server that owns the content — the authoritative upstream a CDN fetches from on a miss.
3. Pull versus push — fetch on first request versus upload in advance — origin pull caching versus pre-positioned distribution.
4. Cache key — the label a cached copy is filed under — normally scheme, host, path and query, extended by Vary headers.
5. Origin shield — one CDN node that speaks to the origin — a mid-tier cache collapsing edge misses into a single upstream request.
6. Purge — delete a copy before it expires — an invalidation pushed to every edge, by URL, tag or prefix.
7. Edge compute — your code running in the CDN — isolate or WebAssembly runtime executing per request at the point of presence.
8. Anycast — one address in many places — the same prefix announced by BGP from multiple sites, each router choosing its best path.
9. DNS steering — a different answer per asker — authoritative DNS returning addresses chosen by resolver location, latency or health.
10. EDNS Client Subnet — telling the server roughly where the user is — RFC 7871 option carrying a truncated client prefix to the authoritative server.

34.10 Latency versus bandwidth#

PLAIN34.10.1 in simple words#

1. Latency is the waiting time before anything arrives. Bandwidth is how much arrives per second once it starts.
2. They are different quantities with different units, and they have almost nothing to do with each other.
3. Latency is measured in milliseconds. Bandwidth is measured in megabits per second. You cannot convert one into the other.
4. Your internet plan sells you bandwidth. Your experience of a slow website is almost always latency.
5. Latency has a hard floor set by physics. Nothing travels faster than light, and light in glass is slower than light in air.
6. Bandwidth has no such floor. You can always lay another cable, or send more colours of light down the same one.
7. So bandwidth is an engineering and money problem, and latency, past a point, is a geography problem.
8. The single most useful sentence in this chapter: you can buy more bandwidth, you cannot buy a shorter distance.
9. A page that makes 60 small requests to a server 200 milliseconds away is slow no matter how fast your connection is, because it spends its life waiting, not transferring.

PLAIN34.10.2 a picture in your head#

1. Think of a road between two cities.
2. Latency is how long the road is, divided by the speed limit. It is the time for one car to make the trip.
3. Bandwidth is how many lanes the road has. It is how many cars can be moving at once.
4. Adding lanes does not shorten the road. A ten-lane motorway and a two-lane road of the same length take one car the same time to drive.
5. If you need to move ten thousand cars, lanes are what you want. If you need to move one car and get an answer back, length is what you want.
6. That is the whole difference, and most complaints about “slow internet” are complaints about the length of the road.

Where this comparison breaks:

7. Cars do not wait for permission. Data does. A sender may only put a limited amount on the road before it must wait for confirmation from the far end, so the length of the road actually limits how many lanes you can use.
8. On a real motorway, adding cars makes everyone slower. On a fibre link, the cable’s capacity is fixed, and what actually slows down is the queue at the entrance, which is a different mechanism with a name: bufferbloat.
9. And the road is not straight. Cables follow coasts, avoid mountains and share trenches, so the real distance is commonly 1.3 to 1.6 times the map distance.

PLAIN34.10.3 a worked example#

1. Start with physics. Light in vacuum travels at 299,792 kilometres per second.
2. Light in optical fibre travels slower, because glass has a refractive index of about 1.47. That gives roughly 204,000 kilometres per second.
3. Round it to 200,000 kilometres per second, which is 200 kilometres per millisecond, and you get a rule of thumb worth memorizing.
4. Ideal round trip in milliseconds is roughly the one-way distance in kilometres divided by 100. The 100 rather than 200 is because a round trip covers the distance twice.
5. Mumbai to Ashburn, Virginia, where a large part of the United States east coast cloud sits, is about 12,850 kilometres in a straight line.
6. So the ideal round trip is about 12,850 divided by 100, which is about 128 milliseconds. That is with a perfectly straight cable and zero equipment.
7. Real measurements from an Indian home connection to that region are around 185 to 200 milliseconds. The extra 60 or so milliseconds is cable routing, plus every router, queue and last-mile hop on the way.
8. No purchase, no protocol and no optimization will ever take that number below 128 milliseconds, short of digging a straighter trench.
9. Now the second calculation, the one that explains slow pages. Suppose a 100 megabit per second line and a 200 millisecond round trip.
10. How much data can be in flight at once? That is the bandwidth-delay product.

BDP = bandwidth x round-trip time
    = 100,000,000 bits/s x 0.200 s
    = 20,000,000 bits
    = 2,500,000 bytes
    = about 2.5 megabytes

11. To use the full 100 megabits per second, the sender must be allowed to have 2.5 megabytes unacknowledged at all times.
12. The original TCP window field is 16 bits, so without the window scaling option the largest window is 65,535 bytes.
13. That caps throughput at 65,535 bytes divided by 0.2 seconds, which is about 328 kilobytes per second, or about 2.6 megabits per second.
14. On a 100 megabit line. That is 2.6 percent of what you pay for, and the cause is entirely the round trip.
15. Finally, why more bandwidth does not help a page. Take a 100 kilobyte response over that same 200 millisecond link.

16. Ten times the bandwidth bought 8 percent. Halving the round trip to 100 milliseconds would have given 480 milliseconds, a 45 percent saving.

PLAIN34.10.4 what is really happening inside#

1. A round trip is not one thing. It is four things added together, and only one of them is physics.
2. Propagation delay: the time for the signal to travel the distance. Fixed by the speed of light in the medium. You cannot improve it.
3. Serialization delay: the time to push the bits of a packet onto the wire. A 1500-byte packet takes 1.2 milliseconds at 10 megabits per second and 12 microseconds at 1 gigabit per second.
4. Queuing delay: time spent waiting in a buffer behind other packets. This is the one that varies wildly and the one that badly sized buffers make worse.
5. Processing delay: the time each router takes to look up and forward. On modern hardware this is microseconds.
6. On a long path, propagation dominates. On a congested last mile, queuing dominates. Both are latency, and neither is fixed by more bandwidth.
7. Now the feedback loop. TCP does not send everything at once. It starts small and grows, because it does not know what the path can take.
8. It starts with an initial window of about ten packets, roughly 14 kilobytes, and it doubles that each round trip while nothing is lost.
9. So a 100 kilobyte file takes about three round trips of growth, whatever the line speed is. At 200 milliseconds each, that is 600 milliseconds of waiting, not transferring.
10. Every extra round trip in your protocol costs the full latency, every time. DNS is one. The TCP handshake is one. Modern encryption is one more. A redirect is another full set.
11. That is why page load time is latency-bound. A typical page is not one transfer. It is dozens of small transfers, each one paying the round trip again, and many of them cannot start until an earlier one finished.
12. Reducing the number of round trips is therefore the single highest-value performance work there is, and it is usually free.

TECHNICAL34.10.5 the engineer’s version#

1. Typical round-trip times from a home broadband connection in India, against the theoretical floor. Measured values are approximate, vary by provider and by time of day, and are given as ranges for that reason.

2. Distances are great-circle from Mumbai. Ideal round trip assumes 204,000 kilometres per second in fibre, which is the speed of light divided by a refractive index of 1.4675, a normal figure for standard single-mode fibre.
3. Note Tokyo and Oregon. Both are further by cable than by map, because the cable routes run through Singapore and around the Pacific rather than over central Asia. The ratio of real to ideal is the honest measure of a route.
4. Bandwidth-delay product, formally: BDP in bytes equals bandwidth in bits per second times round-trip time in seconds, divided by eight. It is the amount of data that must be in flight to keep the pipe full.
5. TCP window scaling is RFC 7323, September 2014, which replaced RFC 1323 of May 1992. The scale factor shifts the 16-bit window left by up to 14 bits, giving a maximum window of 1 gigabyte. It is negotiated in the SYN, so both ends must offer it, and a middlebox that strips the option silently caps your throughput.
6. Initial congestion window of 10 segments is RFC 6928, April 2013. Before it, the standard was 2 to 4 segments, and a small page took twice as many round trips.
7. Loss matters more than bandwidth on long paths. The Mathis equation approximates a single TCP flow’s rate as MSS divided by RTT, times one over the square root of the loss probability.
8. Worked: 1,460-byte segments, 200 millisecond round trip, 0.1 percent loss. That gives about 1.85 megabits per second for one connection, on a line of any speed. This is why long-distance transfers use many parallel connections or a loss-tolerant protocol.
9. Queuing delay and bufferbloat: the term was popularized by Jim Gettys in 2010 and 2011. Oversized, unmanaged buffers in home routers add hundreds of milliseconds of latency under load. The modern fixes are CoDel, RFC 8289, and fq_codel, RFC 8290, both January 2018, plus BBR congestion control from Google in 2016.
10. Protocol round-trip costs, cold start, one round trip written as 1 RTT.

11. Page weight for scale: the HTTP Archive’s public data has put the median page in the mid-2020s at roughly 2.5 megabytes across about 70 requests. Approximate, and it drifts upward every year.
12. Measurement tools, and what each actually measures. ping gives ICMP round trip, which some networks deprioritize. mtr and traceroute give per-hop round trip, which is the time to that hop and back, not a segment time. iperf3 measures achievable throughput, not latency. curl -w measures the phases of a real request, which is what section 34.13 uses.
13. One caution from the reader’s own session. Round-trip time to a hop tells you nothing about whether traffic passes through it. The reader’s trace reached ae106-0.rwa04.pnq20.ntwk.msn.net at hop 12 and then went silent, and the silence was not a latency problem at all.

WORDS34.10.6 remember these#

1. Latency — the wait before anything arrives — one-way or round-trip delay, measured in milliseconds.
2. Round-trip time — there and back — RTT, the interval from sending a packet to receiving its acknowledgement.
3. Bandwidth — how much per second — capacity in bits per second, distinct from achieved throughput.
4. Throughput — what you actually got — measured delivery rate, always less than or equal to bandwidth.
5. Propagation delay — travel time over the distance — distance divided by the signal velocity, about 200,000 km/s in fibre.
6. Serialization delay — time to push bits onto the wire — packet size divided by link rate.
7. Bandwidth-delay product — how much can be in flight — bandwidth times RTT, the window size needed to fill a path.
8. Window scaling — permission to have more in flight — RFC 7323 TCP option raising the maximum window to 1 GiB.
9. Bufferbloat — queues so long they add delay — excessive unmanaged buffering, addressed by CoDel and fq_codel.

34.11 IPv6 properly#

PLAIN34.11.1 in simple words#

1. Every device on the internet needs an address. The old kind, IPv4, is four numbers such as 20.207.73.82, and there are about 4.3 billion of them.
2. The world has far more than 4.3 billion connected devices, so we ran out. That was known to be coming in the early 1990s.
3. IPv6 is the replacement. Its addresses are four times longer, written in hexadecimal and separated by colons, like 2001:db8::1.
4. The number of IPv6 addresses is about 340 undecillion, a 3 followed by 38 digits. It is not a bigger pool. It is a pool of a completely different kind.
5. Because addresses are no longer scarce, every device can have its own public address, and the address-sharing trick called NAT becomes unnecessary.
6. IPv6 is not a newer version of IPv4 that IPv4 machines can talk to. It is a separate, parallel internet. A machine with only IPv6 cannot reach a machine with only IPv4.
7. That single fact explains why adoption took thirty years: everyone had to run both, which is more work, not less, until almost everyone has moved.
8. Most machines therefore run dual stack: both address families at once, choosing per connection.
9. The reader’s own session printed IPv6: (none). We will say exactly what that means, and exactly what it did and did not cause.

PLAIN34.11.2 a picture in your head#

1. IPv4 is a country that issued eight-digit phone numbers, then grew, and ran out of numbers.
2. Its answer was to give one real number to a whole building and put a switchboard in the lobby. Everyone inside dials out through the switchboard, and nobody outside can dial a specific flat directly. That is NAT.
3. IPv6 is a new numbering plan with numbers so long that every room, every appliance and every future appliance can have its own.
4. So the switchboard is no longer needed. Every flat can be dialled directly.
5. But the two numbering plans are not interchangeable. A phone that can only dial old numbers cannot reach a new number at all, and there is no prefix that converts one to the other.
6. So for a long transition every building keeps both systems, and every phone tries the new number first, falling back to the old one.

Where this comparison breaks:

7. Direct dialling is not the same as being open to callers. Removing NAT does not remove the front door. IPv6 home routers still block unsolicited incoming connections by default; the address is reachable, the firewall still decides.
8. And phone numbers are handed out by an office. IPv6 addresses are very often built by the device itself from a prefix the router announces, with no central office involved at all.

PLAIN34.11.3 a worked example#

1. An IPv6 address is 128 bits, written as eight groups of four hexadecimal digits separated by colons.
2. There are exactly two compression rules, and they are always applied in this order.
3. Rule one: in each group, drop leading zeros. Never drop trailing zeros.
4. Rule two: replace one run of consecutive all-zero groups with ::. You may do this only once in an address, because otherwise it would be ambiguous.

full     2001:0db8:0000:0000:0008:0800:200c:417a
rule 1   2001:db8:0:0:8:800:200c:417a
rule 2   2001:db8::8:800:200c:417a

5. A second example, the loopback address, which is what 127.0.0.1 is in the new world.

full     0000:0000:0000:0000:0000:0000:0000:0001
rule 1   0:0:0:0:0:0:0:1
rule 2   ::1

6. A third, the unspecified address, which is what 0.0.0.0 is in the new world. All 128 bits are zero, so the whole thing compresses to ::.
7. A fourth, showing the “only once” rule. This address has two separate zero runs.

full     2001:0db8:0000:0000:0001:0000:0000:0001
right    2001:db8::1:0:0:1
also ok  2001:db8:0:0:1::1
wrong    2001:db8::1::1        (ambiguous, illegal)

8. The preferred form, from RFC 5952, says compress the longest run, use lowercase letters, and do not use :: for a single zero group. So 2001:db8::1:0:0:1 is the correct spelling of that one.
9. In a web address you must put the address in square brackets, because the colon already means “port”.

https://[2606:4700:4700::1111]/          port 443
https://[2606:4700:4700::1111]:8443/     port 8443

10. Prefix lengths work exactly as in IPv4. /64 is the normal size of one local network, and it contains 18.4 quintillion addresses. Your home is usually delegated a /56, which is 256 such networks.

PLAIN34.11.4 what is really happening inside#

1. When an IPv6 machine joins a network, it does not wait to be given an address. It makes one, checks it, and starts using it.
2. First it builds a link-local address beginning fe80::. Every IPv6 interface always has one, and it works even with no router present.
3. Then it checks nobody else has that address, by asking the network directly. This is duplicate address detection.
4. Then it sends a router solicitation, a short message meaning “is there a router here, and what prefix do we use?”.
5. The router replies with a router advertisement carrying the prefix, for example 2401:4900:1f3f:abcd::/64, and some flags.
6. The machine appends its own 64-bit interface part to that prefix and now has a global address. This is SLAAC, stateless address autoconfiguration, and no server was involved.
7. Because that interface part used to be derived from the hardware address, it followed you between networks and could track you. Modern systems therefore generate random, periodically changing addresses instead.
8. A machine normally ends up with several addresses at once: one link-local, one stable global, one or more temporary globals, and possibly one from DHCPv6. This is normal and not a fault.
9. ARP does not exist in IPv6. Its job, turning an address into a hardware address on the local wire, is done by neighbour discovery, which uses ICMPv6 messages sent to a multicast group rather than a broadcast.
10. That is a real improvement. An ARP broadcast interrupts every device on the network. A neighbour solicitation is sent to a solicited-node multicast group that typically only the target listens to.
11. A dual-stack machine looking up a name asks for both an A record, which is IPv4, and an AAAA record, which is IPv6, at the same time.
12. If both come back, it starts connecting over IPv6, and if that has not succeeded within a short delay it starts an IPv4 attempt in parallel and uses whichever finishes first. That is Happy Eyeballs.
13. The user never sees which family won. That is the entire point of the design, and it is why broken IPv6 in 2012 caused visible page hangs and broken IPv6 today usually does not.

TECHNICAL34.11.5 the engineer’s version#

1. IPv6 was first specified in RFC 1883, December 1995, by Steve Deering and Bob Hinden, revised as RFC 2460 in December 1998, and is now RFC 8200, July 2017, an Internet Standard, STD 86.
2. Address architecture is RFC 4291, February 2006. Canonical text representation is RFC 5952, August 2010. The documentation prefix 2001:db8::/32 is RFC 3849 and is the only prefix you should use in examples.
3. Address types you must recognize on sight.

4. There is no broadcast address in IPv6 at all. Everything broadcast used to do is done with multicast groups such as ff02::1, all nodes on the link, and ff02::2, all routers on the link.
5. Unique local addresses are RFC 4193, October 2005. In practice only fd00::/8 is used, with a randomly generated 40-bit global identifier, so that two merged networks do not collide. They are not routable on the public internet.
6. Link-local addresses need a zone index when you use them, because fe80::1 is ambiguous across interfaces. Write fe80::1%en0 on macOS or fe80::1%eth0 on Linux.
7. SLAAC is RFC 4862. Router advertisement is ICMPv6 type 134, router solicitation is type 133. The A flag on a prefix option permits autoconfiguration; the M flag says use DHCPv6 for addresses; the O flag says use DHCPv6 only for other information such as DNS servers.
8. Interface identifiers were originally modified EUI-64, built by splitting the 48-bit MAC address, inserting ff:fe in the middle and flipping the seventh bit. Because that leaks the hardware address, RFC 8981, February 2021, which replaced RFC 4941, defines temporary addresses, and RFC 7217 defines stable opaque identifiers that differ per network.
9. DHCPv6 is RFC 8415, November 2018. It provides stateful assignment and, more importantly for home networks, prefix delegation, by which a router requests a /56 or /48 for the networks behind it. A long-standing practical problem is that Android has never implemented DHCPv6 address assignment, so any network that offers only DHCPv6 leaves Android phones without IPv6.
10. Neighbour discovery is RFC 4861, September 2007. Neighbour solicitation is ICMPv6 type 135, advertisement is 136. The solicited-node multicast address is ff02::1:ff followed by the low 24 bits of the target address. It also provides duplicate address detection and neighbour unreachability detection, which ARP never had.
11. Because so much of IPv6 depends on ICMPv6, blanket-blocking ICMP breaks IPv6 completely. RFC 4890 lists which types must be permitted. This is a common self-inflicted outage.
12. Happy Eyeballs is RFC 6555, April 2012, by Dan Wing and Andrew Yourtchenko, replaced by RFC 8305, December 2017, by David Schinazi and Tommy Pauly. The recommended resolution delay is 50 milliseconds and the recommended connection attempt delay is 250 milliseconds, with a 2 second minimum fallback. A version 3 is being written in the IETF HAPPY working group; revision 04 is dated 2 July 2026 and it is not an RFC yet.
13. Transition and exhaustion dates, which explain the pace better than any argument.

14. The honest reasons adoption took thirty years, in order of weight. IPv6 is not backward compatible, so the first mover pays and gains nothing. NAT and CIDR, RFC 1519 in 1993, relieved the pressure that was supposed to force the move. Carrier-grade NAT let ISPs grow without addresses. Every firewall, monitoring tool, log parser and internal script had to be redone. And a resale market in IPv4 addresses, at roughly 30 to 50 US dollars each in recent years, made staying put affordable.
15. Where it did happen, it happened fast and for a reason: mobile networks built after 2012 had no address supply, so they went IPv6-first with translation at the edge. Reliance Jio in India, launched in 2016, is the largest example, which is why India’s IPv6 capability is around 72 percent, near the top of the world table, while its fixed broadband lags.
16. Now the reader’s own line, IPv6: (none). What it means precisely: the machine had no usable global IPv6 address and therefore no IPv6 default route. Almost certainly the router advertised no IPv6 prefix, which is normal on IPv4-only home broadband behind carrier-grade NAT. The trace’s private hops 172.31.0.17, 172.26.22.235 and 172.16.18.33 are consistent with exactly that kind of network.
17. What it did not cause: the failure. github.com publishes no AAAA record. Checked again on 13 August 2026, dig AAAA github.com returns nothing while dig A github.com returns an address. So even a perfect dual-stack machine would have used IPv4 for that request.
18. What it did remove: a second chance. On a dual-stack host talking to a dual-stack site, Happy Eyeballs races both families, and a silent drop on one is often invisible because the other completes. With no IPv6 there was exactly one path, and when that path went silent there was nothing to fall back to.
19. It also simplified the diagnosis, which is a genuine benefit. There was one address, one path, one failure, and curl -v naming 20.207.73.82 was the complete story.
20. Observation commands: ip -6 addr and ip -6 route on Linux, ifconfig and netstat -rn -f inet6 on macOS, ip -6 neigh or ndp -an for the neighbour cache, ping6 -c3 ff02::1%en0 to see every IPv6 host on the link, dig AAAA example.com for records, and curl -6 or curl -4 to force a family.

WORDS34.11.6 remember these#

1. IPv6 — the long-address internet — 128-bit addressing defined by RFC 8200, not backward compatible with IPv4.
2. Dual stack — both at once — a host running IPv4 and IPv6 simultaneously and selecting per connection.
3. Link-local — an address that works with no router — fe80::/10, valid only on one link, requiring a zone index.
4. Unique local — private addressing for IPv6 — fc00::/7, in practice fd00::/8 with a random global identifier, RFC 4193.
5. SLAAC — the device builds its own address — stateless autoconfiguration from a router-advertised prefix, RFC 4862.
6. Router advertisement — the router announcing the prefix — ICMPv6 type 134, carrying prefix, flags and lifetimes.
7. Neighbour discovery — IPv6’s replacement for ARP — ICMPv6 types 133 to 137, using solicited-node multicast instead of broadcast.
8. AAAA record — the IPv6 version of an A record — a DNS record mapping a name to a 128-bit address.
9. Happy Eyeballs — try both families and take the winner — the connection racing algorithm of RFC 8305.
10. Prefix delegation — your router being given a block — DHCPv6-PD handing a site a /56 or /48 to subnet internally.

34.12 Multicast, broadcast and mDNS#

PLAIN34.12.1 in simple words#

1. There are three ways to address a message on a network, and every beginner meets all three without being told they are different.
2. Unicast is one to one. You address one machine, and only that machine receives it. This is almost all traffic.
3. Broadcast is one to everybody on this local network. Every device receives it and must look at it, even if it does not care.
4. Multicast is one to a named group. Only devices that joined the group receive it. Everyone else ignores it, ideally without even being interrupted.
5. Broadcast never crosses a router. It stops at the edge of your local network, and that is by design, or the internet would drown.
6. Multicast can in principle cross routers, and inside company and provider networks it does. Across the public internet it essentially never does.
7. Broadcast is how your machine gets an address when it joins a network, and how it finds the hardware address of its neighbours.
8. Multicast is how devices announce what they are: printers, speakers, televisions, game consoles, other laptops.
9. mDNS is ordinary DNS, done over multicast, with no server anywhere. It is why names ending in .local work, and why your phone finds a printer you never configured.

PLAIN34.12.2 a picture in your head#

1. Picture a classroom full of people.
2. Unicast is walking over and speaking to one person quietly.
3. Broadcast is standing at the front and shouting to the whole room. Everybody must stop and listen long enough to decide whether it concerns them.
4. Multicast is saying “this is for the chess club only”. You still say it out loud, but only the chess club is expected to pay attention.
5. mDNS is somebody shouting “does anyone here have a printer?”, a printer shouting back “I do, and here is where to find me”, and everyone else in the room writing that down for later because they heard it anyway.
6. That last detail is real and clever. Answers are shouted, not whispered, so every device updates its notes from one exchange.

Where this comparison breaks:

7. A shout does not leave the room, and neither does broadcast. But many cheap network switches treat multicast exactly like a shout and send it to every port anyway, which defeats the point. Better switches watch who joined which group and only forward accordingly.
8. And a classroom has one room. A home with a guest network, or an office with several VLANs, is several rooms with a closed door between them, which is why discovery mysteriously stops working the moment somebody adds a second network.

PLAIN34.12.3 a worked example#

1. Finding a printer on a home network, with no configuration at all.

1 laptop asks    PTR  _ipp._tcp.local
2 printer says   PTR  Office Printer._ipp._tcp.local
3 laptop asks    SRV and TXT for that name
4 printer says   SRV  port 631, host prn9010.local
5 printer says   TXT  rp=ipp/print, pdl=image/urf
6 laptop asks    A    for prn9010.local
7 printer says   A    192.168.0.31
8 laptop prints  to 192.168.0.31 port 631

2. Every one of those questions and answers went to the multicast group 224.0.0.251 on UDP port 5353. No DNS server was involved at any point.
3. You can watch all of it happen. On macOS the tool is built in.

dns-sd -B _services._dns-sd._udp local.   # what kinds exist
dns-sd -B _ipp._tcp                       # printers
dns-sd -B _airplay._tcp                   # AirPlay devices
dns-sd -L "Office Printer" _ipp._tcp      # full details
ping prn9010.local                        # name to address

4. On Linux the equivalent comes from Avahi.

avahi-browse -art
avahi-resolve -n prn9010.local

5. Service names follow a fixed shape: an instance name, an underscore service type, an underscore protocol, then local.

6. So “it just works” is not magic. It is four record types, one multicast group and one well-known port.

PLAIN34.12.4 what is really happening inside#

1. Broadcast at the hardware level means sending to the Ethernet address ff:ff:ff:ff:ff:ff. Every network card accepts a frame with that destination and passes it up.
2. That is why ARP and the first steps of DHCP use broadcast: at that moment the sender does not know who to ask, or does not have an address yet.
3. Multicast at the hardware level uses a different set of Ethernet addresses, derived from the group address. A card is told which of them to accept, and it silently discards the rest without waking the processor.
4. That is the real saving. Broadcast costs every device time. Multicast costs only the members, provided the switch and the network card do their part.
5. Devices announce group membership so switches and routers know where to send. In IPv4 this is IGMP; in IPv6 it is MLD. Switches that listen in on those messages are said to do snooping, and that is what stops multicast becoming broadcast.
6. mDNS uses this. Every participating device joins one group, listens on one port, and answers questions about itself only.
7. Answers are sent to the group, not to the asker, so every listener can cache them. A busy network therefore needs far fewer queries than you would expect.
8. Names are claimed, not assigned. A device probes its desired name three times, and if someone else answers, it renames itself, typically by adding a number. That is where MacBook-2.local comes from.
9. Records carry a flag meaning “replace what you have”, so corrections propagate. A device leaving politely sends its records with a lifetime of zero, which is a goodbye.
10. The packets are deliberately built so they do not travel. Their time-to-live is set so a router will not forward them, and the group is a link-local group. mDNS is not designed to cross networks, and does not.
11. That is the single biggest source of “it worked at home and not at the office”. Two subnets, or a guest network with client isolation turned on, and discovery stops instantly with no error message.
12. A full-tunnel VPN can do the same thing, by capturing name lookups or the default route. On the reader’s macOS machine, utun interfaces were present, and that class of interface is exactly where such interference would appear.

TECHNICAL34.12.5 the engineer’s version#

1. The addresses worth memorizing.

2. IPv4 multicast is 224.0.0.0/4. Within it, 224.0.0.0/24 is link-local control traffic that routers never forward, and 239.0.0.0/8 is administratively scoped for private use, defined in RFC 2365, July 1998.
3. IPv6 has no broadcast at all. ff02::1 is all nodes on the link and ff02::2 is all routers, and scope is encoded in the address itself.
4. Ethernet mapping: an IPv4 group maps to a MAC address of 01:00:5e plus the low 23 bits of the group. Because 32 different groups share one MAC address, a card can accept traffic it did not want, and software must filter again.
5. Group membership signalling: IGMPv1 is RFC 1112, August 1989, IGMPv2 is RFC 2236, November 1997, IGMPv3 is RFC 3376, October 2002, which added source-specific multicast. For IPv6 the equivalent is MLDv2, RFC 3810, June
   2004. IGMP snooping guidance is RFC 4541, May 2006, and is informational, which is why switch behaviour varies so much.
6. Discovery protocols on a normal home network.

7. mDNS is RFC 6762 and DNS Service Discovery is RFC 6763, both February 2013, both by Stuart Cheshire and Marc Krochmal of Apple. Apple shipped the implementation as Rendezvous in Mac OS X 10.2 in August 2002 and renamed it Bonjour in 2005 after a trademark dispute. Avahi is the Linux equivalent, first released in 2005. Windows has shipped a native mDNS responder since Windows 10.
8. .local is reserved by RFC 6762 for mDNS. Using .local as an internal Active Directory domain name, which was once common advice, produces a permanent low-level conflict on any network with Apple devices.
9. Service enumeration uses the meta-query _services._dns-sd._udp.local, which returns a PTR record per service type present. Then PTR gives instances, SRV gives host and port, TXT gives key-value parameters, and A or AAAA gives the address. Four record types, in that order, every time.
10. SSDP is part of UPnP, which Microsoft introduced in 1999. It uses HTTP-like text over UDP with M-SEARCH and NOTIFY verbs. It has been widely abused for reflection and amplification attacks, which is why UDP 1900 should never be reachable from the internet.
11. WS-Discovery on UDP 3702 is what modern Windows and modern network printers use after Microsoft deprecated the old NetBIOS browsing and SMBv1.
12. The Wi-Fi cost, which surprises people. Multicast and broadcast frames are normally sent at the lowest basic rate so every client can hear them, and they are not acknowledged. On a busy access point, chatty discovery traffic therefore consumes disproportionate airtime. Enterprise access points respond by converting multicast to unicast, proxying mDNS, or dropping it, and each of those changes behaviour in a different way.
13. Multicast routing across the public internet was specified, deployed experimentally as the MBone from 1992, and never became general service. PIM Sparse Mode is RFC 7761, March 2016, and is used inside provider networks for IPTV and inside exchanges for market data, but you cannot assume any multicast reaches the internet.
14. Observation: tcpdump -ni en0 udp port 5353 shows the traffic, dns-sd -B browses, avahi-browse -art dumps everything, netstat -gn or ip maddr show lists joined groups, and ping -c3 224.0.0.1 reveals which hosts on the link answer at all.

WORDS34.12.6 remember these#

1. Unicast — one to one — a packet addressed to a single interface.
2. Broadcast — one to all on this link — IPv4 only, 255.255.255.255 or the subnet’s all-ones address, never forwarded by routers.
3. Multicast — one to a group — delivery to hosts that joined a group address in 224.0.0.0/4 or ff00::/8.
4. IGMP — how a host joins a group — Internet Group Management Protocol, RFC 3376 for version 3; MLD is the IPv6 equivalent.
5. Snooping — the switch learning who joined — IGMP or MLD snooping, so multicast is not flooded to every port.
6. mDNS — DNS with no server — RFC 6762 multicast DNS on UDP 5353 serving the .local namespace.
7. DNS-SD — how services are named and found — RFC 6763 conventions using PTR, SRV and TXT records over unicast or multicast DNS.
8. Bonjour — Apple’s name for it — the Apple implementation of mDNS and DNS-SD, shipped as Rendezvous in 2002.
9. SSDP — the older discovery protocol — UPnP’s search mechanism on UDP 1900, a frequent amplification-attack vector.

34.13 Network performance for developers#

PLAIN34.13.1 in simple words#

1. When a page feels slow, “the network is slow” is not a diagnosis. It is a place to start looking.
2. Every web request is made of five separate waits, and they have completely different causes and fixes.
3. DNS: turning the name into an address. Slow if the resolver is far away or the record is not cached.
4. Connect: the handshake that opens the connection. Costs one round trip, always, no exceptions.
5. TLS: agreeing on encryption. Costs one more round trip on modern versions, two on older ones.
6. Time to first byte: you asked, and you are waiting for the server to think. This includes one more round trip plus the server’s own work.
7. Transfer: the bytes arriving. This is the only part that bandwidth actually improves.
8. Most slow requests are the first four. Most “optimizations” people try improve only the fifth.
9. The good news is that you can measure all five with one command, on any machine, in one line, and stop guessing forever.

PLAIN34.13.2 a picture in your head#

1. Think of telephoning a shop in another city to ask whether they have an item.
2. Looking up the number is DNS.
3. Dialling and waiting for someone to pick up is the connection handshake.
4. Agreeing that you will both speak in a private code, and checking the other side really is that shop, is TLS.
5. Asking your question and waiting while they walk to the back room is time to first byte.
6. Them reading the answer aloud is the transfer.
7. If the shop is far away, every single exchange in that conversation takes longer, because every sentence travels the distance twice.
8. So the way to make the call quick is to ask fewer questions, not to speak faster.

Where this comparison breaks:

9. A phone call is one conversation. Loading one page opens dozens, and older protocols could only ask one question per line at a time, which is exactly why browsers used to open six lines to the same shop.
10. And you can keep a phone line open for the next question. Programs often fail to, hanging up and redialling for every item, which is the single most common self-inflicted performance bug.

PLAIN34.13.3 a worked example#

1. curl can print the exact time each phase finished. Put the format in a file, because it is easier to read.

cat > fmt.txt <<'EOF'
   dns  %{time_namelookup}s
  conn  %{time_connect}s
   tls  %{time_appconnect}s
  send  %{time_pretransfer}s
  ttfb  %{time_starttransfer}s
 total  %{time_total}s  code %{http_code}
EOF
curl -sS -o /dev/null -w @fmt.txt https://github.com/

2. A healthy result from an Indian connection to a nearby edge looks like this.

   dns  0.028s
  conn  0.081s
   tls  0.190s
  send  0.191s
  ttfb  0.352s
 total  0.404s  code 200

3. Those numbers are cumulative, all measured from the start. The useful quantities are the differences.

4. Read it. The round trip to this server is about 53 milliseconds, because that is what a TCP handshake costs. TLS cost roughly two of those. Actual data transfer was 52 milliseconds out of 404.
5. So 87 percent of that request was setup and waiting. Buying a faster line improves the 52 milliseconds and nothing else.
6. Now the same command during the reader’s own outage. The shape of the failure is the diagnosis.

   dns  0.026s
  conn  0.000s
   tls  0.000s
  ttfb  0.000s
 total  15.001s  code 000

7. DNS succeeded, so the name resolved to 20.207.73.82 correctly. Then every later phase is zero, because none of them was ever reached.
8. A zero in a later phase does not mean it was fast. It means it never happened. code 000 means no HTTP response existed at all.
9. That is the signature of a silent drop: the connection phase never completed and nothing came back, not a refusal, not an error. Compare it with a refused connection, which fails in a few milliseconds with code 000 and a different curl exit code, and with a slow server, which shows a large ttfb and a normal code 200.
10. Three different faults, three different shapes, one command.

PLAIN34.13.4 what is really happening inside#

1. The first fix is to stop paying the setup cost repeatedly. If you close the connection after each request, the next request pays DNS, TCP and TLS again.
2. Keeping the connection open is called keep-alive, and it has been the default in HTTP since version 1.1. But a program that creates a new client object per request defeats it, and this is very common in application code.
3. The second fix is to stop queuing. In HTTP/1.1, one connection carries one request at a time. Browsers worked around it by opening about six connections per host, which multiplies handshakes and confuses congestion control.
4. HTTP/2 solves this properly by carrying many requests inside one connection as interleaved streams. One handshake, one congestion window, dozens of requests in flight.
5. Header compression helps too. The same cookies and headers repeat on every request; HTTP/2 sends them once and refers back to them afterwards.
6. But HTTP/2 has one honest weakness. All those streams still ride one TCP connection, so a single lost packet stalls every stream until it is resent. That is head-of-line blocking, moved down a layer rather than removed.
7. HTTP/3 fixes that by running over QUIC on UDP, where each stream is recovered independently, and a loss affecting one stream does not stall the others.
8. The third fix is compression. Text compresses enormously, so a compressed response transfers in a fraction of the time. It costs a little processor time on both ends and nothing else.
9. Compression only shortens the transfer phase. It does not remove a single round trip, which is why it helps large responses and barely touches small ones.
10. The fourth fix is doing setup earlier. A browser can be told to resolve a name, or open a connection, before it knows it will need it, so the cost is paid during time the user was going to spend anyway.
11. The last and largest fix is asking for less. A request you do not make costs nothing and cannot fail.

TECHNICAL34.13.5 the engineer’s version#

1. The curl write-out variables, all in seconds from the start of the transfer.

2. Useful companions: %{http_version}, %{num_connects}, %{num_redirects}, %{time_redirect}, %{size_download}, %{speed_download} and %{remote_ip}. curl -w also accepts %{json} in version 7.70.0 and later, which prints every variable as one JSON object.
3. A one-line version worth keeping in your shell history: curl -so /dev/null -w 'dns %{time_namelookup} tls %{time_appconnect} ttfb %{time_starttransfer} tot %{time_total}\n' https://example.com/.
4. Connection reuse. HTTP/1.1, RFC 2068 in January 1997, made persistent connections the default. In practice the wins come from client configuration: reuse one HTTP client object, set a connection pool size, and keep idle connections alive for longer than your request interval.
5. HTTP/2 is RFC 7540, May 2015, restated as RFC 9113, June 2022. It requires ALPN negotiation during the TLS handshake, uses HPACK header compression, RFC 7541, and multiplexes streams over one connection. Server push was part of it, was found not to help, and Chrome removed support in version 106 in October 2022.
6. HTTP/3 is RFC 9114, June 2022, over QUIC, RFC 9000, May 2021, with QPACK header compression, RFC 9204. It removes transport-level head-of-line blocking, survives a change of network address through connection identifiers, and includes the TLS 1.3 handshake in its own setup.
7. Compression, with real behaviour. gzip is RFC 1952 and is universal. Brotli is RFC 7932, July 2016, and typically produces files 15 to 20 percent smaller than gzip on JavaScript and CSS at its highest setting, at a much higher compression cost, so it suits static assets compressed once. Zstandard is RFC 8878, February 2021, and its value is speed at a similar ratio, which suits dynamic responses; browser support arrived in Chrome 123 in March 2024 and Firefox 126 in May 2024.
8. Never compress content that is already compressed, such as images, video or archives. You spend processor time to add a few bytes.
9. TLS costs, precisely. TLS 1.2, RFC 5246, needs two round trips. TLS 1.3, RFC 8446, August 2018, needs one, and a resumed session can send data in the first flight, which is 0-RTT, at the price of replay exposure for non-idempotent requests. Session resumption tickets are what make the second visit cheap.
10. Browser hints that buy round trips back, all standard HTML today: dns-prefetch resolves a name early, preconnect completes DNS, TCP and TLS early, preload fetches a known-critical asset early, and Early Hints, status code 103 from RFC 8297, December 2017, lets the server send those hints before the real response is ready.
11. Server-side measurement should be exported, not guessed. The Server-Timing response header, a W3C specification, carries named durations from the server into the browser’s own performance timeline.
12. Rough real-world weights for a typical page on a 200 millisecond path, approximate and workload-dependent.

13. Measurement discipline. Test from where the users are, not from the datacentre. Test with a cold cache and a warm cache separately. Test the 99th percentile, because the mean hides the failures. And prefer real user monitoring over one laptop’s opinion.

WORDS34.13.6 remember these#

1. Time to first byte — how long before anything arrives — the interval from request sent to first response byte, including one round trip.
2. Keep-alive — do not hang up — HTTP persistent connections, default since HTTP/1.1.
3. Multiplexing — many requests, one connection — interleaved streams within a single HTTP/2 or HTTP/3 connection.
4. Head-of-line blocking — one stuck item stalls the rest — a lost segment delaying all streams sharing a TCP connection.
5. ALPN — deciding which HTTP version during the handshake — Application-Layer Protocol Negotiation, a TLS extension, RFC 7301.
6. Brotli — a stronger text compressor — RFC 7932 encoding, advertised as br in Accept-Encoding.
7. Preconnect — pay the setup before you need it — a resource hint that performs DNS, TCP and TLS for a host in advance.
8. 0-RTT — sending data in the first packet — TLS 1.3 early data on a resumed session, with replay risk.

34.14 Cloud networking preview#

PLAIN34.14.1 in simple words#

1. Everything in this chapter also exists inside a cloud account, with different names and a control panel instead of a cable.
2. A VPC, a virtual private cloud, is your own private network inside the provider’s network. Nobody else’s machines are in it.
3. A subnet is a slice of that network, usually tied to one physical zone.
4. A route table decides where traffic from a subnet goes. A subnet is called public or private purely because of what its route table says.
5. A security group is a firewall attached to a machine, and it remembers connections it allowed.
6. A network ACL is a firewall attached to a subnet, and it remembers nothing, so you must write both directions yourself.
7. A load balancer is a managed reverse proxy: one address in front of many machines.
8. Private link lets you reach a service without your traffic ever touching the public internet.
9. Chapter 43 builds all of this properly. This section is only so the names are not new when you get there.

PLAIN34.14.2 a picture in your head#

1. A VPC is a floor of an office building that you have rented.
2. Subnets are the rooms on that floor, each in a different fire zone.
3. The route table is the sign at each room door saying which corridor to use to leave.
4. A security group is a lock on each desk. A network ACL is a lock on the room door.

Where this comparison breaks:

5. A door lock does not care whether you are leaving or entering as a reply to something. A security group does, and a network ACL does not, and that single difference causes most cloud firewall confusion.

PLAIN34.14.3 a worked example#

1. A minimal two-tier layout, written as a plan.

VPC            10.0.0.0/16
 public subnet 10.0.1.0/24  -> route 0.0.0.0/0 to IGW
 private subnet 10.0.2.0/24 -> route 0.0.0.0/0 to NAT
 load balancer in public subnet, port 443
 app servers in private subnet, port 8080
 database in private subnet, port 5432

2. The security groups, written as sentences rather than tables.

sg-lb   allow in  443 from 0.0.0.0/0
sg-app  allow in  8080 from sg-lb
sg-db   allow in  5432 from sg-app

3. Note what is missing. No outbound rules were needed, because security groups are stateful, so the reply to an allowed request is allowed automatically.
4. Note also that rules refer to other groups, not to addresses. That is the feature worth learning first, because it survives machines being replaced.

PLAIN34.14.4 what is really happening inside#

1. None of this is a physical network. It is a software overlay running on the provider’s real network, and the encapsulation is exactly the tunnelling from section 34.8.
2. Your packets are wrapped, carried across shared hardware, and unwrapped, and the wrapping is what keeps your VPC separate from everyone else’s.
3. A security group is enforced at the virtual network interface of the machine itself, before the packet reaches the guest, which is why the machine’s own firewall never sees a blocked packet.
4. A network ACL is enforced at the subnet boundary, so traffic between two machines in the same subnet never meets it.
5. Because security groups only allow and never deny, they cannot express “allow everyone except this address”. Network ACLs can, which is the main reason they still exist.
6. A private link endpoint puts an address from your own subnet in front of somebody else’s service, so the traffic stays inside the provider’s network and never needs an internet gateway at all.

TECHNICAL34.14.5 the engineer’s version#

1. Amazon launched Virtual Private Cloud in August 2009, and made a default VPC automatic for new accounts in 2013. Azure calls it a virtual network, and Google calls it a VPC network, which is global rather than regional.
2. The comparison that matters most in interviews and in incidents.

3. AWS reserves five addresses in every subnet: the network address, the VPC router, the DNS address at the second host address, one reserved for future use, and the broadcast address. So a /24 gives 251 usable addresses, not
   254. VPC CIDR blocks may be /16 to /28.
4. Load balancer types, with dates. The Classic Load Balancer arrived in 2009, the Application Load Balancer, which is layer 7 and understands HTTP, in August 2016, the Network Load Balancer, layer 4, in September 2017, and the Gateway Load Balancer, which uses Geneve encapsulation, in November 2020.
5. AWS PrivateLink was announced in November 2017. Gateway endpoints for object storage predate it. The distinction to remember is that an interface endpoint consumes an address in your subnet and a gateway endpoint is a route table entry.
6. Connectivity between networks: peering is non-transitive, so three peered VPCs do not form a mesh; Transit Gateway, announced November 2018, exists precisely to solve that.
7. Egress costs money and ingress usually does not, which is a commercial fact with architectural consequences, and it is why cross-zone chatter shows up on bills.
8. Observation: aws ec2 describe-security-groups, aws ec2 describe-route-tables, and VPC Flow Logs, which record accepted and rejected flows and are the cloud equivalent of the packet capture you would otherwise take.
9. Chapter 43 covers the whole of this properly, including NAT gateways, IPv6- only subnets, and how a request reaches a container.

WORDS34.14.6 remember these#

1. VPC — your own private network in the cloud — a logically isolated virtual network with your own address range.
2. Subnet — a slice of that network in one zone — a CIDR range bound to an availability zone with its own route table association.
3. Route table — the sign saying which way out — a set of destination-to-target rules applied to a subnet.
4. Security group — a stateful firewall on the machine — allow-only rules evaluated at the elastic network interface.
5. Network ACL — a stateless firewall on the subnet — numbered allow and deny rules evaluated in order, in both directions.
6. Load balancer — one public address, many servers — a managed layer 4 or layer 7 reverse proxy with health checks.
7. Private link — reach a service without the internet — an endpoint placing a provider service on an address inside your own subnet.

34.98 Common wrong ideas#

1. Wrong: 127.0.0.1 and 0.0.0.0 are two ways of saying the same thing. Right: 127.0.0.1 means this machine only, and listening on 0.0.0.0 means every interface the machine has, which is the opposite of private.
2. Wrong: a firewall makes a machine secure. Right: a firewall only decides which packets are allowed to arrive. The service listening behind it must still be safe on its own, because allowed traffic is not checked.
3. Wrong: DROP and REJECT are the same, since both block the packet. Right: REJECT sends a refusal and fails in milliseconds, DROP says nothing and fails at your timeout. The reader’s 15 seconds of silence was the DROP signature.
4. Wrong: * * * at the end of a traceroute proves the traffic is blocked there. Right: many routers deprioritize or never send ICMP time-exceeded replies, so silence at the end of a trace is normal and proves nothing.
5. Wrong: a VPN makes you anonymous. Right: it moves the point where your traffic becomes visible, from your ISP to your VPN provider, and it changes nothing about cookies, logins or browser fingerprints.
6. Wrong: a proxy handling your HTTPS traffic can read it. Right: after a CONNECT it copies bytes blindly and sees the hostname, sizes and timing only, unless it has been made a trusted certificate authority on your machine.
7. Wrong: a CDN makes everything faster. Right: it makes cacheable things faster and a first miss slightly slower, and it does nothing at all for a response that is different for every user.
8. Wrong: anycast is a kind of DNS load balancing. Right: anycast is one address announced from many places by BGP, decided by routers. DNS steering is different addresses given to different askers, decided by a name server.
9. Wrong: a slow website is fixed by buying more bandwidth. Right: page load is usually latency-bound, so it is fixed by removing round trips, moving content closer, and reusing connections.
10. Wrong: IPv6 is IPv4 with more numbers, so machines can talk to each other across the two. Right: they are separate protocols with no interoperability, which is why dual stack and Happy Eyeballs exist.

34.99 Chapter summary in 20 lines#

1. 127.0.0.1 is an address meaning this machine, localhost is a name for it, and 0.0.0.0 is a placeholder that means all interfaces when listening and unknown when sending.
2. A service bound to loopback is invisible from every other machine, and that is the address behaving correctly, not a firewall.
3. Ports are 16-bit numbers, split by convention into well known below 1024, registered to 49151, and ephemeral above that, and a busy machine exhausts the ephemeral range long before it runs out of anything else.
4. A firewall decides which packets are allowed. It does not inspect what an allowed packet then does, and it is not a substitute for a safe service.
5. Stateless filtering judges each packet alone; stateful filtering remembers connections, which is why you write one rule instead of two.
6. REJECT refuses immediately and produces an instant error. DROP says nothing and produces a timeout, which is why the reader’s curl sat silent for 15 seconds against 20.207.73.82 and then gave up.
7. The same request succeeding over mobile data proved the server was up and the fault lay on one path, while the trace ending in * * * after ae106-0.rwa04.pnq20.ntwk.msn.net proved nothing at all on its own.
8. Middleboxes sit between every client and server, and each one can see a different amount: addresses, ports, hostnames from SNI, or full content if it terminates encryption.
9. A VPN encapsulates whole IP packets at layer 3, so every program is covered, and macOS shows those tunnels as utun interfaces.
10. A proxy works at layer 5 to 7 for the connections that were configured to use it, so a forward proxy hides clients and a reverse proxy hides servers.
11. Tunnelling is putting one packet inside another, and it always costs overhead, which is why broken path MTU discovery produces the classic small-pages-work, large-pages-hang fault.
12. SSH gives three tunnels: -L opens a port on your side, -R opens a port on theirs, and -D gives a SOCKS proxy for anything.
13. Services such as ngrok and Cloudflare Tunnel expose a local port publicly by dialling outward first, which is why they work behind a home router with no port forwarding.
14. A CDN caches what is identical for everybody, uses a cache key built from scheme, host, path and query, and turns a 600 millisecond origin fetch into a 45 millisecond edge hit.
15. Anycast is one address announced from many sites by BGP; it suits DNS and UDP perfectly, works for TCP because routes are stable rather than pinned, and breaks a live connection when routing changes mid-flow.
16. DNS steering is the other mechanism entirely: different answers to different askers, limited by TTLs and by the fact that the server sees the resolver, such as the reader’s 1.1.1.1, and not the user.
17. Latency is bounded by the speed of light in fibre, about 200,000 kilometres per second, so India to United States East cannot go below about 128 milliseconds and measures 185 to 200 in practice.
18. Bandwidth-delay product is bandwidth times round-trip time, so a 100 megabit link at 200 milliseconds needs 2.5 megabytes in flight, and page load time is dominated by round trips rather than by capacity.
19. IPv6 addresses are 128 bits with two compression rules, use SLAAC and neighbour discovery instead of DHCP and ARP, and the reader’s IPv6: (none) cost them nothing directly because github.com publishes no AAAA record, though it removed the second path Happy Eyeballs would have raced.
20. Broadcast reaches everyone on a link, multicast reaches a group, mDNS on 224.0.0.251 port 5353 makes .local discovery work with no server, and curl -w measures DNS, connect, TLS, first byte and transfer separately so that “the network is slow” becomes an actual number.