Networking From Zero - Signals, Ethernet, MAC, ARP and DHCP

23.0 What this chapter gives you#

1. You will be able to say what a network actually is, and name the four jobs every network must do no matter how it is built.
2. You will be able to explain how a bit becomes a voltage on copper, and why a plain high-low signal is not good enough.
3. You will be able to name the line codes that fix it, from Manchester in 1973 to PAM-4 today, and say what each one buys.
4. You will be able to read a cable category number and state its real bandwidth and distance limit instead of guessing.
5. You will be able to tell bandwidth, throughput, latency and jitter apart, and compute a bandwidth-delay product on paper.
6. You will be able to draw an Ethernet frame field by field with correct byte lengths, and explain where the number 1500 came from.
7. You will be able to read a MAC address, find its vendor, and say why your phone lies about it on purpose.
8. You will be able to describe how a switch learns, why a loop is a disaster, and what spanning tree does about it.
9. You will be able to walk through ARP and DHCP message by message, using your own router at 192.168.0.1 as the example.
10. You will be able to explain everything that happens on the wire before a single IP packet is sent, which is what the next six chapters build on.

This chapter opens Part F. Part F uses one real event as its running example: a real network outage on a real home broadband line in India, on a macOS laptop, where curl -v https://github.com printed Trying 20.207.73.82:443... and then sat in complete silence for fifteen seconds. We will take that event apart layer by layer across the whole of Part F. This chapter covers everything underneath the IP layer: the signal, the cable, the frame, the address, and the two small protocols that get a machine onto a network at all. IP addressing, NAT, DNS, routing, traceroute, TCP and TLS each get a full chapter of their own later in Part F. Where they come up here, we name them and move on.

23.1 What a network is: two machines and a shared medium#

PLAIN23.1.1 in simple words#

1. A network is two or more machines that can send each other messages.
2. To send a message you need something in between. A wire, a fibre, or air.
3. That in-between thing is called the medium. It is the road the message travels on.
4. If only two machines share the medium, life is easy. Anything you send can only arrive at one place.
5. As soon as three or more machines share it, four new problems appear at once.
6. Problem one is addressing. If the message can reach everyone, it must say who it is for.
7. Problem two is framing. The medium carries a stream of signals. Something must mark where one message stops and the next starts.
8. Problem three is error detection. Wires pick up noise. The receiver must be able to tell that what arrived is not what was sent.
9. Problem four is sharing. If two machines talk at the same time on one shared medium, both messages are ruined. Somebody has to take turns.
10. Every network technology ever built is an answer to those same four problems. The answers differ. The problems do not.

PLAIN23.1.2 a picture in your head#

1. Picture a long corridor in a hostel with many rooms opening onto it.
2. If you shout from your doorway, everyone in the corridor hears you. The corridor is the shared medium.
3. So you begin with a name: “Anita, the parcel is at the desk.” That is addressing. Everyone hears it. Only Anita acts on it.
4. You say the whole sentence in one go, then stop. The pause tells listeners the message has ended. That is framing.
5. If a bike goes past outside and drowns out three words, Anita asks you to repeat. That is error detection.
6. If two people shout at once, nobody can make out either. So people listen first, and wait if they hear a voice. That is sharing the medium.
7. Now picture a different arrangement: a private phone line from your room to Anita’s room only. No names needed. No shouting. No waiting.
8. That is the difference between a broadcast medium and a point-to-point link.

Where this comparison breaks: a corridor is slow and forgiving. Humans hear a clash and naturally back off. On a wire, two signals overlapping do not sound wrong, they simply produce a third voltage that looks like valid data. The machine cannot notice by ear. It has to be told to check, with a checksum, or to detect the clash electrically within a fixed number of microseconds. Also, a corridor has no fixed speed. A wire does, and that fixed speed is what makes all the timing rules later in this chapter possible.

PLAIN23.1.3 a worked example#

1. Take a home with a laptop, a phone, a smart TV and a router.
2. The router is at 192.168.0.1. That is the real address of the reader’s router, and we will use it throughout this chapter.
3. All four devices are on one logical shared network, even though the laptop and phone reach it over radio and the TV over a cable.
4. The laptop wants to send something to the router. It cannot just push the bytes out and hope.
5. It must attach the router’s local hardware address, so the network knows the message is for the router and not the TV. That is addressing.
6. It must wrap the bytes in a start marker and an end marker so the router knows where the message begins and ends. That is framing.
7. It must append a check number computed from the bytes, so the router can confirm nothing was corrupted. That is error detection.
8. Over radio, it must wait until the air is quiet before transmitting, because the phone might be talking. That is sharing.
9. Over the cable to the TV, none of the waiting is needed, because a modern switched cable link is private in both directions.
10. Same four problems. Two different sets of answers, chosen by the medium.

PLAIN23.1.4 what is really happening inside#

1. Networks are built in layers, and each layer solves one of those problems and hands the rest to the layer above.
2. The bottom layer is the physical layer. Its only job is to turn a 1 or a 0 into something the medium can carry, and back again.
3. Above it sits the data link layer. This is where framing, local addressing, error detection and medium sharing live.
4. Above that sits the network layer, which is where IP lives, and which we deliberately leave for the next chapter.
5. A topology is the shape of the connections. There are four classic shapes.
6. A bus is one long shared cable with every machine tapped onto it. Cheap, and one cut kills everything. This was early Ethernet.
7. A star has every machine wired to one central box. One cut kills one machine. This is what every home and office uses now.
8. A ring passes messages around a loop, each machine to the next. Token Ring and FDDI worked this way. Fibre backbones still use rings for resilience.
9. A mesh connects many machines to many others, so there are several paths between any two. The internet core is a mesh. So is a modern WiFi mesh kit.
10. In practice a home network is a star of stars: devices star into the router, and the router is one point in the ISP’s tree.

TECHNICAL23.1.5 the engineer’s version#

1. The reference model in textbooks is the OSI seven-layer model, published as ISO/IEC 7498-1, with the first version issued in 1984.
2. Real systems follow the four-layer TCP/IP model described in RFC 1122 (1989): link, internet, transport, application.
3. This chapter is entirely layer 1 and layer 2 in OSI terms, and entirely the link layer in TCP/IP terms.
4. The four link-layer duties have formal names: physical signalling, delineation, frame check sequence, and medium access control.
5. Medium access control (MAC) is the sublayer that decides who transmits when. Its name is also why hardware addresses are called MAC addresses.
6. Access methods in use: CSMA/CD (classic Ethernet), CSMA/CA (802.11 WiFi), token passing (802.5 Token Ring, FDDI), TDMA and OFDMA (cellular, WiFi 6).
7. The topology table below gives the practical trade-off.

8. A collision domain is the set of ports where two simultaneous transmissions would interfere. A broadcast domain is the set of ports a broadcast frame reaches. Section 23.8 pulls these apart properly.
9. The distinction between a standard and a convention matters here. IEEE 802.3 is a standard. Calling the central box a “switch” rather than a “bridge” is a convention; the standard still says bridge.

WORDS23.1.6 remember these#

1. Medium — the stuff a message travels through — the physical channel: copper, fibre or free space.
2. Point-to-point — a private line between two machines — a link with exactly two endpoints and no contention.
3. Broadcast medium — a shared space everyone hears — a multiple-access channel requiring a MAC protocol.
4. Framing — marking where a message starts and stops — delineation of a bit stream into protocol data units.
5. Topology — the shape of the wiring — the graph of nodes and links, physical or logical.
6. Collision — two machines talking at once — overlapping transmissions on a shared medium producing an undecodable signal.
7. Layer — one job in the stack — an abstraction level with a defined service interface to the layer above.

23.2 Bits on a wire, physically#

PLAIN23.2.1 in simple words#

1. A wire cannot carry a number. It can only carry a voltage, which is an electrical push.
2. So we agree on a code. High voltage means 1. Low voltage means 0. That is the simplest possible scheme.
3. This simple scheme has a name: NRZ, short for non-return-to-zero. The voltage just stays where it is for the whole bit.
4. NRZ has a serious problem, and the problem is time.
5. The receiver has to know exactly when each bit starts and ends. Otherwise it cannot tell one long 1 from four short 1s.
6. It could use its own clock, but no two clocks in the world run at exactly the same speed. They drift apart.
7. So the receiver needs the signal itself to tell it the time. It does this by watching for the moments the voltage changes.
8. Every change is a tick it can use to correct its clock. This is called clock recovery.
9. Send a hundred zeros in a row and there are no changes for a hundred bit times. The receiver’s clock drifts, and it loses count. That is the failure.
10. There is a second problem. A long run of the same voltage builds up a steady electrical offset in the transformers and capacitors along the path.
11. That offset is called DC bias, and it slowly ruins the receiver’s idea of what counts as high and what counts as low.
12. So we need a code where the voltage keeps changing no matter what the data says, and where highs and lows come out roughly equal over time. That balance is called DC balance.

PLAIN23.2.2 a picture in your head#

1. Think of a drummer keeping time for a dancer who cannot see the drummer.
2. The dancer knows the tempo roughly, but not perfectly. Over a long silence, the dancer slips.
3. If the drummer hits the drum on every single beat, the dancer never slips, because every beat is confirmed.
4. Plain NRZ is a drummer who only hits the drum when the music changes. During a long steady passage there is nothing to hear.
5. Manchester coding is a drummer who hits the drum in the middle of every single beat, always, and uses the direction of the hit to carry the message.
6. It costs twice the effort, but the dancer never loses the beat.
7. The clever later codes are drummers who found a way to guarantee a hit at least every few beats, without hitting on every beat.

Where this comparison breaks: a dancer who slips can recover by watching other dancers. A receiver has nothing else to watch. Also, a drum hit is either there or not, while a real signal on a long cable arrives smeared, rounded and mixed with echoes of earlier bits. The receiver is not detecting a clean edge, it is running an equalizer that reconstructs a plausible edge from a blurred one. At 10 Gbit/s over 100 m of copper, the raw received waveform looks nothing like the transmitted one until that equalizer has done its work.

PLAIN23.2.3 a worked example#

1. We will encode one byte, the ASCII letter K, which is hex 0x4B and binary 01001011.
2. We use the IEEE 802.3 Manchester convention: a 0 is high then low, a 1 is low then high. Each bit becomes two half-bits.
3. Below, - is high voltage and _ is low. Each bit gets four characters, so two per half-bit.

bit    0   1   0   0   1   0   1   1
NRZ  ____----________----____--------
MAN  --____----__--____----____--__--

4. Look at the NRZ line. Bits 3 and 4 are both 0, so eight characters pass with no change at all. Bits 7 and 8 are both 1, same problem.
5. Now look at the Manchester line. There is a transition in the middle of every single bit, guaranteed, whatever the data is.
6. Count the highs and lows in the Manchester line. There are sixteen of each. Perfectly balanced. That is DC balance, guaranteed by construction.
7. The price is visible too. Manchester needs 32 half-bits to send 8 bits. It uses twice the signal changes per bit, so twice the bandwidth.
8. That is exactly why 10BASE-T, which used Manchester, needed 20 MHz of cable bandwidth to carry 10 Mbit/s.

PLAIN23.2.4 what is really happening inside#

1. A twisted pair is two copper wires twisted around each other along the whole length of the cable.
2. The transmitter does not put a voltage on one wire and use earth as the other. It drives the two wires in opposite directions.
3. If one goes up by one volt, the other goes down by one volt. The receiver only looks at the difference between them. This is differential signalling.
4. Now suppose a motor nearby throws out electrical noise. That noise hits both wires almost equally, pushing both up by the same amount.
5. The difference between them is unchanged. The noise cancels. Noise that affects both wires equally is called common mode, and differential receivers ignore it.
6. The twisting is what makes the noise hit both wires equally. Over each half twist, the wire nearer the noise source swaps places with the other one.
7. Averaged over the length, both wires have been equally near the noise. So the induced noise on the two is equal, and cancels.
8. Twisting also stops the pairs inside one cable from disturbing each other. That interference has a name, crosstalk, and each pair is twisted at a different rate so their fields do not line up.
9. Higher cable categories are, in large part, tighter and more precisely varied twist rates, better copper, and sometimes metal foil around each pair.
10. Foil or braid around the pairs is called shielding. It blocks outside interference, but only works if the shield is properly grounded at the ends.

TECHNICAL23.2.5 the engineer’s version#

1. NRZ maps one bit to one symbol with two voltage levels. Zero coding overhead, no guaranteed transitions, no DC balance.
2. Manchester (IEEE 802.3 convention: 0 is high-to-low at mid-bit, 1 is low-to-high) is 50 percent efficient. It doubles the baud rate. Used by 10BASE-T, and still used by consumer infrared, RFID and NFC.
3. There are two Manchester conventions. G. E. Thomas defined the opposite mapping in 1949; inverting a Manchester signal converts one convention into the other. Always state which one you mean.
4. 4B/5B maps each 4 data bits to a 5-bit symbol chosen so no symbol has more than three consecutive zeros. 80 percent efficient. Used by FDDI and by 100BASE-TX, where it feeds an MLT-3 line driver.
5. MLT-3 is a three-level code that cycles low, mid, high, mid on each 1 and holds on each 0. It cuts the fundamental frequency of 100BASE-TX from 125 MHz to about 31.25 MHz, which is why Cat5 could carry Fast Ethernet.
6. 8b/10b was described by Al Widmer and Peter Franaszek at IBM in 1983, US patent 4,486,739 granted December 1984. It maps 8 bits to 10, 80 percent efficient, guarantees run length and DC balance through a running disparity of at most plus or minus two, and supplies comma symbols for alignment. Used by 1000BASE-X, PCI Express 1.x and 2.x, SATA, USB 3.0, DisplayPort 1.x and Fibre Channel.
7. 64b/66b arrived with 10 Gigabit Ethernet. It adds a 2-bit sync header to 64 bits of scrambled payload. Overhead is about 3.1 percent instead of 20 percent. It does not guarantee DC balance or run length; it makes them statistically overwhelmingly likely via a self-synchronous scrambler.
8. PAM-4 sends two bits per symbol using four voltage levels instead of two. It halves the symbol rate for a given bit rate, at the cost of about 9.5 dB of signal-to-noise margin, so it is always paired with forward error correction. It is the basis of 50G, 100G, 200G and 400G electrical lanes.
9. PAM-16, sixteen levels, with 64B/65B framing and LDPC coding, is what 10GBASE-T uses over copper.

10. Twisted-pair cabling categories are defined by ANSI/TIA-568 in North America and ISO/IEC 11801 internationally. The numbers below are the tested frequency, not the data rate.

11. Cat 5 was deprecated in favour of Cat 5e in 2001. Cat 7 and Cat 7A are ISO/IEC classes and were never adopted by TIA, and they use non-RJ45 connectors such as GG45 and TERA; this is why they are rare in practice.
12. Cat 8 comes in Class I (Cat 8.1, U/FTP or F/UTP, RJ45 compatible) and Class II (Cat 8.2, F/FTP or S/FTP, GG45 or TERA). It supports 25GBASE-T and 40GBASE-T, standardized in IEEE 802.3bq in 2016, over a maximum channel of 30 m. It is a data centre top-of-rack cable, not a building cable.
13. Shielding notation is X/YTP where X is the overall shield and Y is the per pair shield: UTP, F/UTP (foil overall), S/FTP (braid overall, foil per pair), U/FTP (foil per pair only).
14. Every twisted-pair Ethernet variant from 10BASE-T through 10GBASE-T is specified to 100 m of channel, which is 90 m of solid horizontal cable plus 10 m of stranded patch cords.
15. The honest version: category ratings are measured on a fully certified channel with proper terminations. A Cat 6A cable punched down badly, or run next to a fluorescent ballast, may fail at 30 m. Cable certification testers such as the Fluke DSX series measure insertion loss, NEXT, return loss and delay skew, and they exist precisely because the label on the jacket is a claim, not a measurement.

WORDS23.2.6 remember these#

1. NRZ — voltage just stays high or low — non-return-to-zero line code, one bit per symbol, no guaranteed transitions.
2. Clock recovery — working out the beat from the signal — extracting a bit clock from data transitions with a phase-locked loop.
3. DC balance — equal amounts of high and low — zero mean of the transmitted waveform, required for transformer and capacitor coupling.
4. Manchester — a change in the middle of every bit — biphase-L code, 50 percent efficient, self-clocking and DC balanced.
5. Line code — the rule turning bits into signals — the mapping from data bits to channel symbols.
6. Differential signalling — drive two wires oppositely — balanced transmission where the receiver takes the difference and rejects common-mode noise.
7. Crosstalk — one pair leaking into another — near-end and far-end coupling between pairs, measured as NEXT and FEXT in dB.
8. Twisted pair — two wires wound together — balanced copper pair with a controlled twist rate for noise and crosstalk rejection.
9. PAM-4 — four voltage levels instead of two — pulse amplitude modulation carrying two bits per symbol, always used with FEC.

23.3 Bits on fibre and on air#

PLAIN23.3.1 in simple words#

1. Copper carries electricity. Fibre carries light. Air carries radio waves.
2. A fibre optic cable is a very thin strand of extremely pure glass. Thinner than a human hair.
3. A laser or an LED at one end flashes light into it. A light detector at the other end sees the flashes.
4. Light on means 1, light off means 0, in the simplest fibre systems.
5. The light does not leak out of the sides, even when the fibre is bent, and that is the whole trick.
6. It stays in because of total internal reflection. Light hitting the edge of the glass at a shallow enough angle bounces back in instead of escaping.
7. To make that happen, the fibre has two layers of glass. An inner core and an outer cladding with a slightly lower ability to bend light.
8. Fibre beats copper in three ways. It goes much further, it carries much more, and it does not care about electrical noise at all.
9. It has no metal in the signal path, so a lightning strike or a motor next door does nothing to it.
10. Radio is the third medium. It needs no cable at all, which is its whole appeal and also every one of its problems.

PLAIN23.3.2 a picture in your head#

1. Imagine a long straight corridor with mirrors on both walls.
2. Shine a torch down it at a shallow angle. The beam bounces from wall to wall and travels the whole length, still bright.
3. Now imagine the corridor is not straight but gently curved. The beam still bounces along, because the angles are still shallow.
4. Bend it sharply round a corner and some light hits the wall too steeply, passes through, and is lost. That is a bend loss in a real fibre.
5. Now imagine two kinds of corridor. A wide one, where beams entering at different angles take noticeably different path lengths.
6. Beams that started together arrive at different times, so a sharp flash at one end becomes a smeared blob at the other. That is multi-mode fibre.
7. And a very narrow corridor, so narrow that only one straight path fits. Every photon takes the same route and arrives together. That is single-mode.
8. Narrow costs more to build and needs a better torch. But the flash stays sharp over tens of kilometres.

Where this comparison breaks: light in a fibre is not a torch beam bouncing off mirrors. The refractive index change at the core boundary reflects the wave, and in single-mode fibre the core is only a few wavelengths wide, so ray optics stops being the right description entirely. There the light is a guided wave whose field extends slightly into the cladding. The corridor picture also has no equivalent of chromatic dispersion, where different colours of light travel at different speeds through the same glass and smear the pulse even in single-mode fibre.

PLAIN23.3.3 a worked example#

1. Compare three ways of getting 10 Gbit/s from one building to another.
2. Option one: 10GBASE-T over Cat 6A copper. Maximum 100 m. Cheap cable, cheap sockets, but the transceiver chip runs hot and burns several watts per port.
3. Option two: 10GBASE-SR over multi-mode fibre at 850 nm. Maximum 300 m on OM3 fibre, 400 m on OM4. Cheap laser, more expensive fibre.
4. Option three: 10GBASE-LR over single-mode fibre at 1310 nm. Maximum 10 km. More expensive laser, cheapest fibre per metre.
5. The buildings are 600 m apart. Copper is out at 100 m. Multi-mode is out at 400 m. Single-mode wins, and would still win at 6 km.
6. Notice that the fibre itself is not what sets the distance. The optics at each end do: the laser wavelength, its power, and the receiver’s sensitivity.
7. Those optics live in a small plug-in module called an SFP. The switch port is a slot. You choose the distance by choosing the module.

PLAIN23.3.4 what is really happening inside#

1. The transmitter is a laser diode, or for the cheapest short links an LED. It is switched on and off billions of times a second.
2. The receiver is a photodiode. Light landing on it frees electrons and makes a tiny current, which an amplifier turns back into a digital signal.
3. Three things degrade the signal along the way, and they are different from the copper problems.
4. Attenuation is the light simply getting dimmer, absorbed and scattered by the glass. Measured in decibels per kilometre.
5. Modal dispersion is the smearing caused by different paths in multi-mode fibre. It is the reason multi-mode has a short reach.
6. Chromatic dispersion is the smearing caused by different wavelengths in the pulse travelling at slightly different speeds. It limits long single-mode links.
7. Glass is not equally transparent at all colours. There are three windows where it is unusually clear, and every optical network uses them.
8. Those windows are around 850 nanometres, 1310 nanometres and 1550 nanometres. All are infrared. None are visible to the eye.
9. 1550 nm is the clearest of the three, which is why undersea cables use it.
10. Because different wavelengths do not interfere, you can send many separate channels down one fibre at once, each on its own colour. That is wavelength division multiplexing.
11. Radio works on the same idea of a carrier wave, but the medium is shared with everyone in range, and the signal fades, reflects and gets absorbed by walls and bodies.

TECHNICAL23.3.5 the engineer’s version#

1. Standard telecom fibre has a 125 micrometre cladding diameter regardless of type. Only the core differs.
2. Single-mode fibre has a core of about 8 to 10 micrometres. Multi-mode has 50 or 62.5 micrometres.
3. Single-mode is classified OS1 (indoor, tight buffered) and OS2 (outdoor, loose tube, lower loss). Multi-mode is OM1 through OM5.
4. Typical attenuation figures: about 3 dB/km at 850 nm on multi-mode, about 0.35 dB/km at 1310 nm and about 0.2 dB/km at 1550 nm on single-mode. Corning’s Vascade EX2500 submarine fibre is specified at a nominal 0.148 dB/km at 1550 nm.
5. Total internal reflection occurs when light in the core meets the cladding at an angle greater than the critical angle, set by the two refractive indices. The numerical aperture is the sine of the largest input angle that will be guided; typical multi-mode NA is 0.2 to 0.29.
6. Single-mode fibre is used for essentially all links longer than about 1 km, and multi-mode inside buildings and data halls.

7. The pluggable optic form factors are defined by multi-source agreements (MSAs) between vendors, not by IEEE. SFP dates from 2001, SFP+ from 2006, QSFP+ from 2009, QSFP28 from 2014, QSFP-DD and OSFP from 2016 onward.
8. This is a good example of a convention rather than a standard: an MSA is an industry agreement with no standards body behind it, which is why vendor lock-in on optics is so common.
9. Coarse WDM (CWDM) uses 8 to 18 channels spaced 20 nm apart. Dense WDM (DWDM) uses 40 to 96 or more channels spaced 0.8 nm or 0.4 nm apart on the ITU-T G.694.1 grid. A single modern DWDM fibre pair carries tens of terabits per second.
10. Light travels through glass at about 200,000 km/s, roughly two-thirds of its speed in vacuum, because the refractive index of silica is about 1.47. That gives about 5 microseconds of delay per kilometre of fibre, and it is a hard physical floor on latency.
11. Radio is IEEE 802.11 for WiFi and 3GPP specifications for cellular. Chapter 31 covers radio fully: modulation, channels, the 2.4, 5 and 6 GHz bands, CSMA/CA, MIMO, OFDMA and why WiFi behaves so differently from a cable. We say only this much here: the reader’s laptop reached the router over radio, and the fault in this book’s running example was not in that radio hop.
12. Observation commands: on a switch, show interfaces transceiver reports per-lane optical transmit and receive power in dBm, which is how you tell a dirty connector from a dead laser.

WORDS23.3.6 remember these#

1. Core — the middle of the fibre that carries light — the higher-refractive index region, 8 to 10 um single-mode, 50 or 62.5 um multi-mode.
2. Cladding — the outer glass that keeps light in — the lower-index layer, standard 125 um outer diameter.
3. Total internal reflection — light bouncing back instead of escaping — reflection above the critical angle at a refractive-index boundary.
4. Single-mode — one path only, goes far — small-core fibre supporting one propagation mode, no modal dispersion.
5. Multi-mode — many paths, goes less far — large-core fibre with modal dispersion limiting reach and bandwidth.
6. Attenuation — the signal getting weaker — optical power loss in dB per km.
7. Dispersion — the pulse smearing out — modal or chromatic pulse spreading that limits the bit rate over distance.
8. SFP — the plug-in module that does the light — small form-factor pluggable transceiver defined by an MSA, not by IEEE.
9. WDM — many colours down one fibre — wavelength division multiplexing on the ITU-T G.694.1 grid.

23.4 Bandwidth, throughput, latency and jitter#

PLAIN23.4.1 in simple words#

1. People say “my internet is slow” and mean four completely different things.
2. Bandwidth is the maximum rate the link can carry. It is a capacity. It is what you bought.
3. Throughput is the rate you actually get for a particular transfer, right now. It is always less than bandwidth, sometimes far less.
4. Latency is the waiting time before anything arrives. How long one message takes to get there.
5. Jitter is how much that waiting time wobbles from message to message.
6. A video call needs low latency and low jitter. It barely needs bandwidth.
7. A large download needs bandwidth. It does not care about latency at all.
8. A game needs low latency above everything.
9. These four are almost independent. A link can have enormous bandwidth and terrible latency, and a link can have tiny bandwidth and superb latency.
10. And there is a fifth state that is none of these: the link can be silent. Nothing comes back at all. That is what happened to the reader.

PLAIN23.4.2 a picture in your head#

1. The classic picture is a water pipe.
2. Bandwidth is how wide the pipe is. A wide pipe moves more water per second.
3. Latency is how long the pipe is. A long pipe means water takes longer to appear at the far end, even if the pipe is very wide.
4. Throughput is how much water actually comes out per second, which depends on the pipe, the tap, the pump, and anything half-blocking the middle.
5. Jitter is the pulsing you feel when the flow is not steady.
6. Widening the pipe does nothing for the time the first drop takes to appear. That is the single most useful thing this picture teaches.

Where this comparison breaks: it breaks in three important places, and engineers get caught by all three. First, water in a pipe is continuous, but data is sent in discrete packets, so a packet that is one byte too big for the pipe is not trimmed, it is dropped whole. Second, a water pipe does not lose water and then resend it, but a network does, and the resend costs a full round trip of latency that the pipe picture cannot represent. Third, and most importantly here, a blocked water pipe still tells you something: pressure builds, or water comes back. A silently dropping network gives you nothing at all. There is no equivalent of a packet vanishing without trace in the water picture, and that vanishing is precisely what the reader was looking at.

PLAIN23.4.3 a worked example#

1. The reader’s own case is the clearest example in this book.
2. The command was curl -v https://github.com. It printed Trying 20.207.73.82:443... and then nothing at all for fifteen seconds.
3. Consider what each of the four measures would have looked like if it were the problem.
4. Low bandwidth: the page would have loaded, slowly, byte by byte. It did not load at all.
5. High latency: the page would have loaded after a long pause. It never loaded.
6. Low throughput because of congestion: the transfer would have stalled and restarted. There was no transfer to stall.
7. High jitter: the page would have loaded unevenly. There was nothing to be uneven about.
8. What actually happened is that the connection attempt got no reply of any kind. No refusal, no error, no data. Silence.
9. That is not slowness. That is a packet going into the network and never producing any observable effect.
10. The same site loaded instantly on mobile data from the same phone. So the site was up, and the destination was reachable from India.
11. What that proves: the failure was specific to the path taken from the reader’s broadband line, not to the destination.
12. What it merely suggests: something on that path was discarding packets silently. It does not tell us which device, or why. Later chapters in Part F narrow that down.

PLAIN23.4.4 what is really happening inside#

1. Latency is made of four separate delays, and they add up.
2. Propagation delay is the time the signal takes to physically travel. In fibre, about 5 microseconds per kilometre. Nothing can beat this.
3. Transmission delay is the time to push all the bits of a message out of the interface. It equals message size divided by link rate.
4. Processing delay is the time each router takes to look at the packet and decide where to send it. Modern hardware does this in microseconds.
5. Queuing delay is the time the packet spends waiting in a buffer behind other packets. This is the one that varies, and it is the main source of jitter.
6. Now the key idea. A link with high bandwidth and high latency can hold a surprising amount of data that has left the sender but not yet arrived.
7. That quantity has a name: the bandwidth-delay product. It is the capacity of the pipe itself, measured in bytes in flight.
8. If the sender is only allowed a certain number of unacknowledged bytes at a time, and that number is smaller than the bandwidth-delay product, the link can never be filled, however fast it is.
9. This is why a 1 Gbit/s link from India to the United States can deliver 3 Mbit/s to a badly tuned application. The link is not slow. The window is too small.

TECHNICAL23.4.5 the engineer’s version#

1. Bandwidth in networking means capacity in bits per second. In signal processing it means a frequency range in hertz. The two are related through the Shannon-Hartley theorem but are not the same quantity. Say which you mean.
2. Round-trip time (RTT) is what ping measures. One-way latency is roughly half of it, but only on a symmetric path, and many paths are not symmetric.
3. Bandwidth-delay product, worked in full:

Link rate      = 100 Mbit/s = 100,000,000 bits/s
RTT            = 180 ms     = 0.180 s
BDP (bits)     = 100e6 * 0.180 = 18,000,000 bits
BDP (bytes)    = 18,000,000 / 8 = 2,250,000 bytes
               = about 2.15 MiB in flight

4. 180 ms is a realistic round trip from an Indian home line to a server on the US east coast. 2.15 MiB must be in flight at all times to keep that link full.
5. The original TCP receive window field is 16 bits, so a maximum of 65,535 bytes. That is 3 percent of what this link needs.
6. RFC 1323 (1992), now superseded by RFC 7323 (2014), added the window scale option, which multiplies the window by a power of two up to 2^14. Without it, long fat networks cannot be filled.
7. Maximum achievable throughput with a fixed window is window / RTT. With 65,535 bytes and 180 ms that is about 2.9 Mbit/s, on a 100 Mbit/s link.
8. TCP and its congestion control get a full chapter later in Part F. The point here is only that link speed alone never predicts throughput.

9. Jitter is formally the variation in one-way delay. RFC 3550, the RTP specification (2003), defines an interarrival jitter estimator that media applications use to size their playout buffer.
10. Bufferbloat is the condition where oversized, unmanaged buffers in a modem or router hold hundreds of milliseconds of packets, so latency explodes under load while bandwidth tests still look fine. Jim Gettys named and characterized it in 2010. The fixes are active queue management algorithms: CoDel (2012) and fq_codel, and the CAKE qdisc on Linux.
11. Tools: ping for RTT, iperf3 for throughput, mtr for per-hop loss and latency, and flent running the RRUL test for bufferbloat under load.
12. The honest version: ping measuring 20 ms does not mean your data has 20 ms of latency. ICMP echo is often handled on a different, slower or faster path inside a router than forwarded traffic, and is frequently rate-limited or deprioritized. This matters enormously for reading a traceroute, which is a later chapter in Part F.
13. For the reader’s case none of these numbers apply, because there was no measurable transfer. The correct technical description is a connection attempt that received neither a SYN-ACK, nor a TCP RST, nor an ICMP unreachable message, within the 15 second observation window.

WORDS23.4.6 remember these#

1. Bandwidth — how wide the pipe is — link capacity in bits per second, or a frequency range in hertz depending on context.
2. Throughput — what you actually get — achieved data rate for a given flow over a given interval, also called goodput when headers are excluded.
3. Latency — the waiting time before anything happens — one-way or round-trip delay, the sum of propagation, transmission, processing and queuing delays.
4. Jitter — how much the wait wobbles — variance in one-way delay, estimated per RFC 3550 in real-time media.
5. Propagation delay — the time light or electricity takes to travel — distance divided by signal velocity, about 5 us per km in fibre.
6. Bandwidth-delay product — how much data fits inside the link — link rate multiplied by RTT, the in-flight capacity in bytes.
7. Bufferbloat — big buffers making everything laggy — excess queuing delay from unmanaged FIFO buffers, fixed by AQM such as fq_codel.
8. Silent drop — nothing comes back at all — a discard with no ICMP or TCP notification, indistinguishable from a black hole from the sender’s side.

23.5 Ethernet#

PLAIN23.5.1 in simple words#

1. Ethernet is the set of rules for sending data over a local network cable. It is more than fifty years old and it won completely.
2. It was invented in 1973 at a research lab in California called Xerox PARC, by Robert Metcalfe and David Boggs.
3. The idea it copied came from Hawaii. A radio network called ALOHAnet let many machines share one radio channel by simply transmitting and retrying after a clash.
4. Metcalfe’s version used a cable instead of radio, and added one crucial rule: listen before you talk.
5. The name comes from “luminiferous ether”, the invisible substance nineteenth century physicists thought light travelled through. It was a joke about the cable being the medium everything swims in.
6. In the beginning, every machine was tapped onto one thick coaxial cable running through the building. One cable, shared by all.
7. Because it was shared, two machines could talk at once and ruin each other’s message. That is a collision.
8. The rule for handling it was: listen first, and if a collision happens anyway, stop, wait a random time, and try again.
9. That rule is CSMA/CD, which stands for carrier sense multiple access with collision detection.
10. Today, on almost every network you will ever touch, collisions cannot happen, because the shared cable is gone.

PLAIN23.5.2 a picture in your head#

1. Picture a small meeting where everyone shares one microphone that is always on.
2. Before speaking, you listen. If someone is talking, you wait. That is carrier sense.
3. Sometimes two people start in the same instant, because each heard silence. Both voices go out together and neither is understandable. That is a collision.
4. Both notice the clash while it is happening, stop immediately, and each waits a different random number of seconds before trying again. That is collision detection and backoff.
5. Now replace the meeting with a room where every person has a private two-way headset to a chairperson.
6. Anyone can speak at any time, in both directions at once, and the chairperson passes each message only to the person it is for.
7. Nobody ever clashes with anybody. That is a modern switched, full-duplex network.

Where this comparison breaks: humans hear a clash instantly. On a long cable, electricity takes time to travel, so a station at one end can be halfway through transmitting before it hears the clash from the far end. This is why classic Ethernet had a minimum frame size and a maximum cable length that were mathematically linked. If a frame were too short, the sender could finish before the collision signal got back, and would never know it had failed.

PLAIN23.5.3 a worked example#

1. Take the original 10 Mbit/s shared Ethernet and work out why the minimum frame is 64 bytes.
2. The rule is: a sender must still be transmitting when a collision from the far end reaches it.
3. Worst case, the signal must go all the way to the far end and the collision must come all the way back. That is a full round trip.
4. The standard fixes this round trip budget at 512 bit times, which is the time to send 512 bits.

512 bits at 10 Mbit/s = 51.2 microseconds
512 bits / 8          = 64 bytes

5. So a frame must be at least 64 bytes long, or the sender might finish too early. 64 bytes counts from the destination address through the check sequence.
6. That 51.2 microsecond budget also caps the cable. Signals travel through coax at roughly two thirds the speed of light, and repeaters add delay, so the maximum end-to-end network was 2500 m with at most four repeaters.
7. If a frame arrives shorter than 64 bytes, it is called a runt and is discarded. Historically a runt meant a collision. Today it means a faulty cable, a bad NIC or a duplex mismatch.
8. On a switched full-duplex link there is no shared medium and no collision, so none of this timing matters. The 64-byte minimum survives anyway, purely for compatibility.

PLAIN23.5.4 what is really happening inside#

1. A modern Ethernet link between a laptop and a switch is two independent one-way channels using different wire pairs.
2. Each end can transmit whenever it likes, because nobody else is on its transmit path. This is full duplex.
3. When the cable is plugged in, the two ends negotiate. They exchange short pulse bursts advertising every speed and duplex mode they support.
4. They then both pick the best mode that both support. This is auto-negotiation, and it is why plugging in a cable just works.
5. If negotiation fails or is disabled at one end, you can get a duplex mismatch: one side full duplex, the other half duplex.
6. That link will still pass traffic, but the half-duplex side will report collisions and late collisions and throughput will collapse under load. It is a classic slow-network fault.
7. The switch in the middle looks at the destination address of each frame and sends it out one port only. Section 23.8 explains exactly how it knows which.
8. Because each link is private and full duplex, the network has as many collision domains as it has ports, and each of those domains has exactly one possible talker in each direction. So there are no collisions.

TECHNICAL23.5.5 the engineer’s version#

1. The experimental Ethernet first ran on 22 May 1973 at Xerox PARC at 2.94 Mbit/s. That odd rate was the Alto computer’s system clock divided down.
2. Metcalfe and Boggs published “Ethernet: Distributed Packet Switching for Local Computer Networks” in Communications of the ACM in July 1976.
3. US patent 4,063,220, “Multipoint data communication system with collision detection”, was granted in 1977 to Xerox, naming Metcalfe, Boggs, Thacker and Lampson.
4. The DIX consortium of Digital Equipment Corporation, Intel and Xerox published the Ethernet Blue Book in 1980 at 10 Mbit/s. Version 2, the frame format still called Ethernet II, was published in November 1982.
5. Metcalfe founded 3Com in June 1979 to commercialize it.
6. IEEE published 802.3 as a draft in 1983 and as a full standard in 1985. The competing standards, Token Ring (IEEE 802.5, 1985) and FDDI (ANSI X3T9.5, 1987), were technically respectable and lost on cost.
7. The 802.3 speed family, with the amendment letter and year:

8. Media naming is systematic: the number is the speed in Mbit/s or Gbit/s, BASE means baseband (no carrier modulation of the whole channel), and the suffix names the medium. T is twisted pair, S is short-wave multi-mode fibre, L is long-wave single-mode, C is twinaxial copper, R is a 64b/66b coded lane.
9. 10BASE5 was 10 mm thick coax, up to 500 m per segment, tapped with a vampire tap that pierced the jacket. 10BASE2 was thinner coax with BNC T-pieces, 185 m per segment, and was universally called thinnet or cheapernet.
10. The connector story: coax with N-type or BNC, then the 8P8C modular plug that everyone calls RJ45 for 10BASE-T onward. RJ45 is technically the wrong name, since RJ45 is a telephone registered jack specification, but the usage is universal. That is a convention, not a standard.
11. Pin assignments come from TIA-568A and TIA-568B, which differ only in swapping the green and orange pairs. A cable with one end A and one end B is a crossover cable.
12. Crossover cables are no longer needed because Auto-MDI-X, standardized as part of 802.3ab in 1999 and now universal, detects and swaps the pairs electronically.
13. Interframe gap is 96 bit times: 9.6 us at 10 Mbit/s, 0.96 us at 100 Mbit/s, 96 ns at 1 Gbit/s. It exists so receivers can resynchronize between frames.
14. Half-duplex gigabit exists in the standard, using carrier extension to pad the slot time to 4096 bit times, but no one ever deployed it.
15. Observation: ethtool eth0 on Linux shows negotiated speed, duplex and auto-negotiation state; networksetup -listallhardwareports and ifconfig on macOS show the equivalent, and netstat -i shows error counters.

WORDS23.5.6 remember these#

1. Ethernet — the standard way to wire a local network — the IEEE 802.3 family of physical and MAC layer specifications.
2. CSMA/CD — listen, talk, stop if you clash — carrier sense multiple access with collision detection, mandatory on half-duplex shared media.
3. Collision — two stations transmitting at once — overlapping transmissions within one collision domain, detected by the transceiver.
4. Backoff — waiting a random time before retrying — truncated binary exponential backoff over a doubling contention window.
5. Full duplex — both directions at once — simultaneous independent transmit and receive on separate pairs or wavelengths.
6. Auto-negotiation — the cable sorting itself out — link code word exchange using fast link pulses to select speed, duplex and flow control.
7. Duplex mismatch — one side full, one side half — a misconfiguration causing late collisions and severe throughput loss.
8. Runt — a frame that is too short — a frame under 64 octets, discarded and counted as an error.
9. Interframe gap — the compulsory pause between frames — 96 bit times of idle before the next preamble.

23.6 The Ethernet frame, field by field#

PLAIN23.6.1 in simple words#

1. Everything sent on an Ethernet network is wrapped in a frame.
2. A frame is like an envelope. The data you care about is the letter inside. The envelope carries the addressing and the checks.
3. The envelope has a fixed layout, and every network card in the world agrees on it. That agreement is what makes any card talk to any other.
4. It begins with a wake-up pattern so the receiver can lock onto the timing.
5. Then the address it is going to, then the address it came from.
6. Then a two-byte code saying what kind of thing is inside. Is this an IP packet, or an ARP message, or something else.
7. Then the data itself, between 46 and 1500 bytes.
8. Then a four-byte check number computed from everything before it.
9. The receiver recomputes that check number. If it does not match, the frame is thrown away silently. Ethernet does not ask for a resend. Something higher up must notice.
10. The whole frame, not counting the wake-up pattern, is between 64 and 1518 bytes.

PLAIN23.6.2 a picture in your head#

1. Think of a postcard going through an old sorting office.
2. The first thing the machine needs is a run of identical marks along the edge so it can line the card up and match its rollers to the card’s speed. That is the preamble.
3. Then one different mark saying “the card starts here”. That is the start frame delimiter.
4. Then the delivery address, then the return address.
5. Then a small printed code saying what class of item this is: a letter, a parcel slip, a legal notice. That is the EtherType.
6. Then the message.
7. Then a checksum stamped at the end, computed from the whole card by the sending office. The receiving office recomputes it and bins the card if they disagree.
8. Postcards have a minimum size because the machine cannot grip anything smaller, and a maximum size because the slot is only so wide.

Where this comparison breaks: a sorting office that bins a damaged card usually tells someone. Ethernet tells nobody. A frame failing its check sequence is counted in a statistics register on the interface and then forgotten. Also, a postcard’s return address is written by the sender and can be a lie; so can an Ethernet source address, and that is the root of several attacks in section 23.10.

PLAIN23.6.3 a worked example#

1. Here is a real frame, an ARP request from the reader’s laptop asking who owns 192.168.0.1. MAC addresses are illustrative, since the session record does not include them.

Preamble  AA AA AA AA AA AA AA      (7 bytes, alternating)
SFD       AB                        (1 byte)
Dest MAC  FF FF FF FF FF FF         (6 bytes, broadcast)
Src MAC   3C 22 FB 8A 41 D7         (6 bytes, the laptop)
EtherType 08 06                     (2 bytes, ARP)
Payload   00 01 08 00 06 04 00 01   (28 bytes of ARP,
          3C 22 FB 8A 41 D7          padded to 46)
          C0 A8 00 6B
          00 00 00 00 00 00
          C0 A8 00 01
Padding   00 x 18                   (to reach 46 bytes)
FCS       9C 4E 1B 33               (4 bytes, CRC-32)

2. C0 A8 00 6B is 192.168.0.107 in hexadecimal. That is the laptop.
3. C0 A8 00 01 is 192.168.0.1 in hexadecimal. That is the reader’s router.
4. Count the frame from destination MAC to FCS: 6 + 6 + 2 + 46 + 4 = 64 bytes. Exactly the minimum.
5. The ARP message itself is only 28 bytes, so 18 bytes of zero padding were added to reach the 46-byte minimum payload.
6. The preamble and SFD are not counted in the 64. They are physical-layer framing, stripped before the frame is handed up.
7. A packet capture tool will normally show you this frame as 60 bytes, because it also strips the 4-byte FCS after checking it.

PLAIN23.6.4 what is really happening inside#

1. The preamble is 56 bits of alternating 1 and 0. On the wire this is a square wave, which is the easiest possible thing for a receiver’s clock circuit to lock onto.
2. The start frame delimiter breaks the pattern with two 1 bits in a row. That single break is what says “data starts on the next bit”.
3. The EtherType field does double duty, and this is a historical scar. Values of 1500 or below mean “this is a length in bytes”, the original IEEE 802.3 meaning. Values of 1536 or above mean “this is a type code”, the DIX Ethernet II meaning.
4. Because 1500 is below 1536, both interpretations can coexist unambiguously, which is why the maximum payload is 1500 and not something rounder.
5. The check sequence is a CRC-32, a cyclic redundancy check. It is not a simple sum. The frame is treated as one enormous binary number and divided by a fixed polynomial; the remainder is the check value.
6. CRC-32 catches all single-bit errors, all double-bit errors, all odd numbers of errors, and all burst errors up to 32 bits. Beyond that it catches all but about 1 in 4 billion.
7. The maximum payload of 1500 bytes is called the MTU, maximum transmission unit.
8. The number 1500 came from the 1980 DIX specification. Buffer memory was expensive in 1979, and a station holding the shared coax cable for too long starved everybody else. 1500 bytes at 10 Mbit/s is about 1.2 milliseconds of airtime, which was judged a fair maximum turn.
9. It has never changed, on any Ethernet speed, for compatibility. A 400 Gbit/s link still defaults to a 1500-byte MTU chosen for a 10 Mbit/s coax cable in
10. Jumbo frames raise the payload to around 9000 bytes. Nine thousand was chosen because CRC-32’s error detection strength starts to weaken beyond about 12000 bytes, and 9000 comfortably holds an 8192-byte block plus headers.
11. Jumbo frames are not in the IEEE standard. They are a vendor convention. Every device on the path must agree, or frames are silently dropped.

TECHNICAL23.6.5 the engineer’s version#

1. The complete frame layout, in transmission order:

2. Preamble is 10101010 repeated seven times. The SFD is 10101011. Together they are 64 bits, often described as an 8-byte preamble ending in two 1 bits.
3. Minimum frame is 64 octets measured from destination MAC through FCS inclusive. Preamble, SFD and IFG are not counted.
4. Maximum untagged frame is 1518 octets: 6 + 6 + 2 + 1500 + 4. With one 802.1Q tag it is 1522. With two stacked tags (802.1ad, “QinQ”) it is 1526.
5. With an 802.1Q tag present the minimum payload is 42 bytes rather than 46, so the 64-octet minimum still holds.
6. The FCS is a left-shifting CRC-32 with polynomial 0x04C11DB7, initial value 0xFFFFFFFF, post-complemented, giving a residue check value of 0x38FB2284.
7. Common EtherType values, all in hexadecimal, assigned by the IEEE Registration Authority:

8. MTU is the layer 3 payload limit, 1500 bytes for standard Ethernet. It is distinct from the maximum frame size, which includes the 18 bytes of header and FCS.
9. Fragmentation costs, and this is why MTU matters. If an IP packet is larger than the path MTU, it must be split. Every fragment carries a full IP header, so overhead rises. If any one fragment is lost, the whole original packet is lost. Firewalls and load balancers often handle fragments badly or drop them. Fragment reassembly consumes memory at the receiver and is a classic denial of service target.
10. IPv4 fragmentation is defined in RFC 791 (1981). IPv6 removed router fragmentation entirely in RFC 8200 (2017); only the source may fragment.
11. Path MTU discovery, RFC 1191 (1990) for IPv4 and RFC 8201 (2017) for IPv6, finds the smallest MTU on a path by sending packets with the Don’t Fragment bit set and listening for ICMP “fragmentation needed” replies.
12. The honest version: PMTUD is fragile in the real world, because many networks block all ICMP, so the “fragmentation needed” message never arrives and the connection hangs after the handshake. This is the classic PMTUD black hole, and RFC 4821 (2007) defines Packetization Layer PMTUD to work around it by probing at the TCP layer instead.
13. PPPoE, used on many DSL lines, adds 8 bytes of header and leaves an MTU of
    1492. VPNs and tunnels reduce it further, which is why tunnel interfaces often default to 1400 or lower.
14. The reader’s macOS machine showed several utun interfaces. Those are tunnel interfaces, and each carries its own smaller MTU. Check with ifconfig utun0 and look for the mtu value.
15. Jumbo frame support is queried with ip link show on Linux or ifconfig on macOS; on a switch it is often called “system jumbo mtu” or “maximum frame size” and applies per port or globally depending on the vendor. That is an implementation detail, not a standard.

WORDS23.6.6 remember these#

1. Frame — the envelope around your data on a local network — a layer 2 protocol data unit delimited by preamble and interframe gap.
2. Preamble — the wake-up pattern at the start — 56 bits of alternating 1 and 0 for receiver clock recovery.
3. EtherType — the code saying what is inside — a 2-byte protocol identifier when the value is 1536 or greater.
4. Payload — the actual data being carried — 46 to 1500 octets in a standard untagged frame.
5. FCS — the check number at the end — a 32-bit CRC over the frame, polynomial 0x04C11DB7.
6. MTU — the largest chunk of data one frame can carry — maximum transmission unit, 1500 octets for standard Ethernet.
7. Jumbo frame — a bigger than normal frame — a non-standard payload of about 9000 octets, requiring agreement across the whole path.
8. Fragmentation — cutting a packet up to fit — splitting an IP datagram across multiple frames, with per-fragment header cost and all-or-nothing loss.
9. Runt and giant — too small and too big — frames below 64 or above the configured maximum, both counted as errors.

23.7 MAC addresses#

PLAIN23.7.1 in simple words#

1. Every network interface has a hardware address burned into it. It is called a MAC address.
2. MAC stands for media access control. The name is an accident of which sublayer of the standard defines it.
3. It is 48 bits long, which is 6 bytes. It is written as six pairs of hexadecimal digits.
4. Example: 3C:22:FB:8A:41:D7. Some systems use hyphens or dots instead of colons. Same address.
5. It is meant to be globally unique, so that no two interfaces anywhere have the same one.
6. The first three bytes identify the manufacturer. The last three are the manufacturer’s own serial number for that interface.
7. So from a MAC address you can usually tell what kind of device it is, without asking the device.
8. An address is either for one device, for a group of devices, or for everyone on the network.
9. The everyone address is all ones: FF:FF:FF:FF:FF:FF. A frame sent there is delivered to every device on the local network.
10. Modern phones and laptops now often make up a fake MAC address on purpose, to stop shops and networks tracking you as you walk around.

PLAIN23.7.2 a picture in your head#

1. Think of a MAC address like the serial number stamped on the back of an appliance.
2. The first part identifies the factory. The second part identifies the individual unit that factory made.
3. Two appliances from the same factory share the first part and differ in the second.
4. Unlike a postal address, it says nothing about where the device is. It travels with the device.
5. A postal address changes when you move house. A serial number does not. This is exactly the difference between an IP address and a MAC address.
6. That difference is why both exist, and it is the reason ARP has to exist at all.

Where this comparison breaks: a serial number is permanent, but a MAC address can be changed in software on almost every operating system in about one command. Uniqueness is administrative, not physical. Manufacturers have shipped duplicate addresses by mistake, and cheap unbranded network chips have shipped whole batches with identical addresses.

PLAIN23.7.3 a worked example#

1. Take the address 3C:22:FB:8A:41:D7.
2. Split it: 3C:22:FB is the vendor part, 8A:41:D7 is the device part.
3. The vendor part is called the OUI, organizationally unique identifier. It is bought from the IEEE.
4. 3C:22:FB is registered to Apple. So this is very likely a Mac or an iPhone.
5. You can look up any OUI in the IEEE public registry, or with a local tool. On many Linux systems, /usr/share/ieee-data/oui.txt holds the whole list.
6. Now look at the first byte alone: 3C is 00111100 in binary.
7. The last bit of that first byte is 0. That means unicast: one device.
8. If it were 1, it would be multicast: a group. 01:00:5E:... is the IPv4 multicast group prefix, and 33:33:... is the IPv6 one.
9. The second-to-last bit of the first byte is 0. That means the address is globally assigned by a real manufacturer.
10. If it were 1, the address is locally administered, meaning somebody made it up. Randomized privacy addresses always have this bit set to 1.
11. So 3C ends in 00: globally unique, unicast, real Apple hardware.
12. Compare with 3E:22:FB:8A:41:D7. 3E is 00111110. Last bit 0, so still unicast, but second-to-last bit is 1, so this one was made up locally.

PLAIN23.7.4 what is really happening inside#

1. The network card holds its address in a small memory chip, and the driver reads it when the interface starts.
2. When the card receives a frame, it compares the destination address against a short list without waking the CPU.
3. The list is: its own address, the broadcast address, and any multicast groups it has been told to join.
4. If the destination matches nothing on that list, the card discards the frame in hardware. The operating system never sees it.
5. Multicast matching is often done with a hash filter rather than exact matching, so the card sometimes lets through groups it did not join. The driver filters those out in software.
6. There is a special mode where the card stops filtering entirely and passes up everything it hears. That is promiscuous mode, and section 23.12 covers it.
7. Address randomization works by generating a random 46-bit value, setting the locally administered bit to 1 and the multicast bit to 0, and using that as the interface address.
8. On phones, a different random address is typically generated per network name, so the same shop sees a consistent device but two different shops cannot link you.

TECHNICAL23.7.5 the engineer’s version#

1. The 48-bit format is IEEE EUI-48. There is also a 64-bit EUI-64 used by IPv6 interface identifiers, FireWire and ZigBee.
2. The IEEE Registration Authority sells three block sizes: MA-L (a 24-bit OUI, 16.7 million addresses), MA-M (28-bit prefix, 1 million addresses) and MA-S (36-bit prefix, 4096 addresses).
3. Bit ordering matters and confuses everyone. Ethernet transmits each octet least-significant bit first, so the I/G bit is literally the first bit on the wire, which is why the hardware can begin filtering immediately.
4. Bits in the first octet:

5. Broadcast is FF:FF:FF:FF:FF:FF, all 48 bits set, which is simply the all-groups multicast address.
6. IPv4 multicast maps to 01:00:5E plus the low 23 bits of the group address, per RFC 1112 (1989). IPv6 multicast maps to 33:33 plus the low 32 bits, per RFC 2464 (1998).
7. Ranges reserved for local use without IEEE registration are defined in IEEE 802c (2017), which carves the locally administered space into structured quadrants.
8. MAC randomization history, with dates, since this changes fast: Apple iOS 8 randomized addresses during WiFi scanning in 2014; Android 10 made randomized addresses the default for connections in 2019; Apple shipped Private WiFi Address in iOS 14 and iPadOS 14 in 2020, and macOS gained a rotating variant later. Windows 10 has had a per-network random hardware address option since
9. Since iOS 18 and macOS 15 (2024) Apple’s default is a rotating private address that changes periodically on networks the device does not consider trusted. Check current behaviour before relying on it; this area moves.
10. Consequences you must plan for: MAC-based DHCP reservations break, MAC filtering as a security control was always weak and is now useless, and network access control must use 802.1X with real credentials instead.
11. Commands to read your own address:

# macOS, all interfaces with hardware addresses
ifconfig | grep ether

# macOS, which port is which
networksetup -listallhardwareports

# macOS, the WiFi interface specifically
ifconfig en0 | grep ether

# Linux
ip link show
cat /sys/class/net/eth0/address

12. To see the vendor decoded for you, packet capture tools do it automatically: tcpdump -e prints Ethernet headers, and Wireshark shows Apple_8a:41:d7 style names using its own OUI database.
13. The honest version: MAC addresses do not travel across routers. A router strips the incoming frame and builds a new one with new source and destination MAC addresses for the next hop. The MAC address the reader’s router sees is the laptop’s. The MAC address GitHub’s server sees is its own upstream router’s. This is the single most important fact about MAC addresses and the reason section 23.10 exists.

WORDS23.7.6 remember these#

1. MAC address — the hardware address of a network interface — a 48-bit EUI-48 identifier used for layer 2 delivery.
2. OUI — the manufacturer part of the address — organizationally unique identifier, the first 24 bits, allocated by the IEEE.
3. Unicast — addressed to one device — I/G bit clear in the first octet.
4. Multicast — addressed to a group — I/G bit set, with reserved prefixes 01:00:5E for IPv4 and 33:33 for IPv6.
5. Broadcast — addressed to everyone on the link — FF:FF:FF:FF:FF:FF, all bits set.
6. Locally administered — an address somebody made up — U/L bit set, not guaranteed unique outside the local link.
7. MAC randomization — your device using a fake address for privacy — rotating locally administered addresses, default on iOS 14+ and Android 10+.
8. EUI-64 — the 64-bit version — extended unique identifier used by IPv6 SLAAC and other technologies.

23.8 Hubs, switches and how a switch learns#

PLAIN23.8.1 in simple words#

1. In the middle of a star-shaped network sits a box that everything plugs into.
2. There have been two kinds of box, and the difference between them is the biggest single improvement in local networking history.
3. A hub is the dumb one. Anything arriving on any port is copied out of every other port, immediately, with no thought.
4. So every device hears everything. The whole hub is one shared medium, and collisions happen exactly as on the old coax cable.
5. A switch is the smart one. It looks at where the frame is going and sends it out of one port only.
6. To do that it needs to know which device is on which port. Nobody tells it. It works this out by itself, by watching.
7. Every frame carries a source address. When a frame comes in on port 3 saying it is from address X, the switch writes down “X is on port 3”.
8. It builds a table this way, purely by listening. This is called learning.
9. If a frame arrives for an address the switch has not learned yet, it does the only safe thing: it sends it out of every port except the one it came in on. That is flooding.
10. Hubs have been obsolete since roughly 2005. You will only meet one in a museum or a very old cupboard.

PLAIN23.8.2 a picture in your head#

1. Picture a hotel with a single letterbox for the whole building, and a porter.
2. The hub porter takes every letter and photocopies it into every room’s pigeonhole. Everyone reads everything and throws away what is not theirs.
3. The switch porter starts on day one knowing nothing, with a blank notebook.
4. A letter comes out of room 12 with “from: Anita” on it. The porter writes “Anita, room 12” in the notebook.
5. Later a letter arrives addressed to Anita. The porter checks the notebook and walks it straight to room 12.
6. A letter arrives for Vikram, who is not in the notebook yet. The porter has no choice: a copy goes to every room. Somebody will be Vikram.
7. When Vikram replies, his letter has “from: Vikram” on it, and the porter writes down his room. From then on, letters to Vikram go straight there.
8. If a guest checks out and a new one moves in, the notebook goes stale. So each entry is rubbed out if it has not been seen for a while.

Where this comparison breaks: the porter walks at human speed and can think. A switch must make this decision in a few hundred nanoseconds, for millions of frames a second, simultaneously on every port. It does it with dedicated silicon and a content-addressable memory, not by looking things up in a list. Also, the porter would notice two guests claiming the same name. A switch does not object; it simply moves the entry, which is exactly how one class of attack works.

PLAIN23.8.3 a worked example#

1. Three devices plug into a four-port switch. All addresses shortened for readability.

Port 1: laptop     AA:AA
Port 2: phone      BB:BB
Port 3: TV         CC:CC
Port 4: router     DD:DD   (192.168.0.1)

2. Step 0. The switch has just been powered on. Its table is empty.

MAC table: (empty)

3. Step 1. The laptop sends a frame to the router: source AA:AA, destination DD:DD, arriving on port 1.
4. The switch learns the source: AA:AA is on port 1. It looks up DD:DD, finds nothing, and floods the frame out of ports 2, 3 and 4.

MAC table:
  AA:AA -> port 1

5. Step 2. The router replies: source DD:DD, destination AA:AA, arriving on port
6. The switch learns DD:DD is on port 4. It looks up AA:AA, finds port 1, and sends the frame only to port 1. The phone and TV never see it.

MAC table:
  AA:AA -> port 1
  DD:DD -> port 4

7. Step 3. The phone sends to the router: source BB:BB, destination DD:DD, on port 2.
8. The switch learns BB:BB on port 2, looks up DD:DD, finds port 4, forwards only there. No flooding at all this time.

MAC table:
  AA:AA -> port 1
  BB:BB -> port 2
  DD:DD -> port 4

9. The TV on port 3 has still never transmitted, so it is still unknown. Traffic for it will be flooded until it speaks once.
10. After three frames the switch has learned three of four devices and stopped flooding almost entirely. This is why switched networks scale and hubs do not.

PLAIN23.8.4 what is really happening inside#

1. The table is called the MAC address table, or the forwarding database, or the CAM table depending on who is speaking.
2. Each entry holds an address, a port number, a VLAN identifier and a timestamp.
3. Entries expire. The default ageing time is 300 seconds on most vendors. Every time a frame is seen from that address, the timer resets.
4. Ageing is what lets you unplug a laptop from one port and plug it into another. Actually, the switch relearns instantly on the first frame from the new port, and ageing only cleans up devices that have gone away entirely.
5. There are two ways to forward a frame, and they trade latency against safety.
6. Store-and-forward reads the entire frame into memory, checks the FCS, and only then forwards it. Corrupt frames never leave the switch.
7. Cut-through starts forwarding as soon as it has read the destination address, six bytes in. Far lower latency, but it will happily forward a frame that turns out to be corrupt.
8. A collision domain is a set of ports that can interfere with each other. On a switch, each port is its own collision domain, and on full duplex that domain has no contention at all.
9. A broadcast domain is the set of ports a broadcast frame reaches. A whole switch is one broadcast domain by default, and so is a whole stack of switches wired together.
10. Splitting a broadcast domain requires either a router or a VLAN, which is section 23.9.

TECHNICAL23.8.5 the engineer’s version#

1. The correct standards term for a switch is a MAC bridge, defined by IEEE 802.1D. “Switch” is universal industry usage, that is, a convention.
2. Learning is defined in 802.1D as the learning process, and flooding of unknown unicast is required behaviour, not an optimization.
3. Lookups use ternary content-addressable memory (TCAM) or hash-based exact-match tables. TCAM does a full parallel comparison in one clock cycle, at a substantial cost in silicon area and power.
4. Table sizes are a real limit. A small unmanaged switch may hold 1000 to 8000 entries. An access switch holds 16k to 32k. A data centre spine switch holds 256k or more.
5. When the table is full, new entries cannot be learned and all traffic to unknown addresses is flooded. MAC flooding is an attack that deliberately fills the table with bogus source addresses to turn a switch into a hub and enable eavesdropping. The defence is port security, limiting learned addresses per port.

6. Fragment-free is a compromise: wait for the first 64 bytes, which is where collision fragments would appear, then start forwarding.
7. Modern data centre switches quote port-to-port latency of 300 to 800 nanoseconds in cut-through mode. Store-and-forward latency scales with frame size, because the whole frame must arrive first; a 1518-byte frame at 1 Gbit/s takes about 12 us just to receive.
8. What a managed switch adds over an unmanaged one: VLANs (802.1Q), spanning tree (802.1D and successors), link aggregation (802.1AX, formerly 802.3ad), port mirroring for capture, quality of service queues (802.1p), 802.1X port authentication, LLDP neighbour discovery (802.1AB), SNMP monitoring, and per-port statistics and rate limiting.
9. Link aggregation is directly relevant to this book’s running example. The traceroute in the reader’s session showed interfaces named ae66-0, be23 and po22. Those prefixes mean aggregated Ethernet, bundle Ethernet and port-channel: three vendor words for several physical links bonded into one logical link. Those are naming conventions, not standards.
10. That bonding is also why hops 5, 6, 8, 9 and 11 in the reader’s traceroute showed more than one address for a single hop. Packets in the same trace took different member links of the same bundle. That is load balancing working correctly, not a fault.
11. Observation commands, vendor-dependent: show mac address-table on Cisco and Arista, show ethernet-switching table on Juniper, bridge fdb show on Linux, and ifconfig bridge0 on macOS for the built-in bridge.

WORDS23.8.6 remember these#

1. Hub — repeats everything to everyone — a multiport repeater, one collision domain, obsolete since the early 2000s.
2. Switch — sends each frame to one port — a MAC bridge per IEEE 802.1D, learning and forwarding per destination address.
3. MAC address table — the switch’s notebook — the filtering database mapping address and VLAN to egress port, with an ageing timer.
4. Learning — writing down who is where — populating the filtering database from observed source addresses.
5. Flooding — sending to everyone because you do not know — forwarding unknown unicast, broadcast and unregistered multicast to all ports in the VLAN.
6. Collision domain — where two talkers can clash — the set of ports sharing one contended medium; one per port on a switch.
7. Broadcast domain — how far a broadcast reaches — the set of ports a broadcast frame is flooded to, bounded by a router or a VLAN edge.
8. Store-and-forward — read it all, check it, then send — full-frame buffering with FCS validation before egress.
9. Cut-through — start sending after the address — forwarding begins after the destination MAC is read, propagating errored frames.
10. Managed switch — one you can configure — a bridge supporting 802.1Q, spanning tree, aggregation, mirroring and management protocols.

23.9 VLANs and spanning tree, briefly#

PLAIN23.9.1 in simple words#

1. One switch is one network. Every device on it can reach every other device, and every broadcast reaches everyone.
2. Often that is not what you want. An office may want staff computers, guest WiFi, security cameras and phones kept apart from each other.
3. You could buy four separate switches and four sets of cables. That is expensive and inflexible.
4. Instead you tell one switch to pretend to be four switches. Each pretend switch is called a VLAN, a virtual local area network.
5. Ports are assigned to VLANs. A port in VLAN 10 cannot reach a port in VLAN 20, even though they are in the same physical box.
6. To get between VLANs you need a router. That is the point: it gives you a place to apply rules.
7. Now the second idea. A network with two paths between the same two switches has a loop.
8. Loops are wonderful for reliability and catastrophic without protection, because a broadcast frame goes round the loop forever.
9. There is no field in an Ethernet frame that counts hops, so nothing ever stops it. It multiplies at every switch and the network dies in seconds.
10. Spanning tree is the protocol that finds loops and switches off just enough links to break them, while keeping those links ready as backups.

PLAIN23.9.2 a picture in your head#

1. Picture an office floor with one big open room.
2. VLANs are movable partition walls. The floor is unchanged, but people in one section cannot walk into another without going through a door with a guard on it. The guard is the router.
3. To move a person from one section to another you do not rewire anything. You change which section their desk is assigned to. That is reassigning a port to a VLAN.
4. Now picture the announcement system. A message sent in section A is repeated by every loudspeaker in section A only.
5. Now the loop. Imagine two corridors joining the same two rooms, and a rule that any announcement heard must be repeated into every other corridor.
6. Someone makes one announcement. It goes round both corridors, arrives back, gets repeated, and now there are two. Then four. Then eight.
7. Within seconds nobody can hear anything else. That is a broadcast storm.
8. Spanning tree is a supervisor who walks the building once, works out the corridors, and locks exactly one door so no circular route exists.

Where this comparison breaks: a locked door stays locked. Spanning tree keeps watching, and if the open corridor collapses it unlocks the spare door within seconds. Also, the supervisor is not a person: every switch runs the algorithm independently and they reach the same answer by exchanging small messages, with no central authority. That distributed agreement is the clever part.

PLAIN23.9.3 a worked example#

1. A small office, one 24-port switch, three VLANs.

2. Ports 1 to 23 are access ports. Each belongs to exactly one VLAN. The device plugged in knows nothing about VLANs at all.
3. Port 24 is a trunk port. It carries all three VLANs to the router over one cable.
4. For that to work, frames leaving port 24 must say which VLAN they belong to. So a 4-byte tag is inserted into each frame.
5. A camera on port 19 sends a frame. It is untagged, a plain 64-byte frame.
6. The switch forwards it towards port 24 and inserts a tag saying VLAN 30. The frame is now 68 bytes.
7. The router receives it, sees VLAN 30, and applies the camera policy: allowed to reach the recorder, not allowed to reach the internet.
8. A guest laptop on port 15 broadcasts. That broadcast reaches ports 13 to 18 and the trunk, and never touches ports 1 to 12. The staff never see it.
9. Without VLANs, one broadcast domain of 23 ports. With VLANs, three broadcast domains sharing one box.

PLAIN23.9.4 what is really happening inside#

1. The tag is four bytes inserted immediately after the source MAC address.
2. The first two bytes are always 0x8100. That is the EtherType that says “a VLAN tag follows”.
3. The next two bytes hold three things: three bits of priority, one bit that is almost always zero, and twelve bits of VLAN identifier.
4. Twelve bits gives 4096 values, of which 0 and 4095 are reserved, so 4094 usable VLANs.
5. Because the tag is inserted, the switch must recompute the frame check sequence. Adding a tag changes the frame, so the old CRC is no longer valid.
6. An access port strips the tag on the way out, so end devices never see it.
7. Now spanning tree. Every switch sends small messages called BPDUs out of every port every two seconds.
8. Each BPDU carries the sender’s identity and its cost to reach the switch it currently believes is the root of the tree.
9. Through this gossip, all switches agree on one root bridge: the switch with the numerically lowest bridge identifier.
10. Every other switch works out its own cheapest path to the root, keeps that port forwarding, and blocks any other port that would create a second path.
11. A blocked port still listens to BPDUs. If they stop arriving, the topology has changed, and the blocked port is brought back into service.

TECHNICAL23.9.5 the engineer’s version#

1. VLAN tagging is IEEE 802.1Q, first published in 1998 and now folded into 802.1Q-2022. The tag structure:

2. A tagged frame’s maximum size is 1522 octets instead of 1518. Older equipment that counted 1518 as an absolute maximum reported these as “baby giants”.
3. The native VLAN on a trunk is carried untagged, by convention VLAN 1. This is the basis of VLAN hopping via double tagging, so best practice is to set the native VLAN to an unused ID and tag everything.
4. 802.1ad (2005), “QinQ” or provider bridging, stacks a second tag with TPID 0x88A8 so a carrier can transport a customer’s whole VLAN space.
5. Spanning Tree Protocol was invented in 1985 at Digital Equipment Corporation by Radia Perlman, who described it in her paper “An Algorithm for Distributed Computation of a Spanning Tree in an Extended LAN” and in the poem “Algorhyme”. IEEE published it as 802.1D in 1990.
6. Classic STP port states are blocking, listening, learning, forwarding and disabled. Convergence takes 30 to 50 seconds, because listening and learning each last one forward delay of 15 seconds by default.
7. Default timers: hello 2 seconds, forward delay 15 seconds, max age 20 seconds.
8. Rapid Spanning Tree Protocol, IEEE 802.1w (2001), cut the states to discarding, learning and forwarding, and converges within three hello times or within milliseconds of a detected physical link failure. It was merged into 802.1D-2004.
9. Multiple Spanning Tree Protocol, IEEE 802.1s (2002), runs one tree per group of VLANs so different VLANs can use different physical links. It was merged into 802.1Q-2005.
10. Shortest Path Bridging, IEEE 802.1aq (2012), and TRILL, RFC 6325 (2011), replace the tree with a link-state protocol so all links can carry traffic. In practice most large data centres skipped both and moved to layer 3 fabrics with ECMP, or to VXLAN overlays per RFC 7348 (2014).
11. Why a loop is catastrophic, precisely: the Ethernet header has no TTL or hop count field. An IP packet has a TTL that a router decrements, so an IP routing loop self-terminates after at most 255 hops. A layer 2 loop has no such stop condition. Broadcast and unknown-unicast frames replicate at every switch on every pass, so traffic grows geometrically until link utilization reaches 100 percent, typically within one to two seconds.
12. There is a second failure alongside the storm: MAC table instability. The same source address arrives on two ports alternately, so every switch rewrites that entry thousands of times a second, and legitimate unicast traffic gets flooded as well.
13. Protective features, all vendor implementation details rather than standards: BPDU guard (shut a port that receives a BPDU when it should not), root guard, loop guard, and storm control thresholds.
14. Home routers usually run no spanning tree at all. Plugging one cheap unmanaged switch into another twice, by accident, will take down a home or small office network completely, and this is a genuinely common fault.

WORDS23.9.6 remember these#

1. VLAN — one switch pretending to be several — a virtual LAN, an isolated broadcast domain identified by a 12-bit VID under IEEE 802.1Q.
2. Access port — a port for an ordinary device — an untagged port belonging to exactly one VLAN.
3. Trunk port — a port carrying many VLANs at once — a tagged port carrying 802.1Q-tagged frames for multiple VIDs.
4. Tag — the four bytes saying which VLAN — TPID 0x8100 plus PCP, DEI and VID, inserted after the source MAC.
5. Loop — two paths between the same switches — a cycle in the layer 2 topology with no hop count to stop frames.
6. Broadcast storm — the network drowning in copies of one frame — geometric replication of flooded frames around a loop.
7. Spanning tree — the protocol that breaks loops safely — IEEE 802.1D and its successors, electing a root bridge and blocking redundant ports.
8. BPDU — the small message switches gossip with — bridge protocol data unit, sent every hello interval to build and maintain the tree.
9. Root bridge — the switch everything measures distance from — the bridge with the lowest bridge ID, elected by all participants.

23.10 ARP: from an IP address to a MAC address#

PLAIN23.10.1 in simple words#

1. Here is the problem this section solves, and it is the most important idea in the chapter.
2. Software works with IP addresses. The reader’s laptop wants to send something to 192.168.0.1, the router.
3. But the network card cannot send to an IP address. It can only send to a MAC address, the hardware address from section 23.7.
4. Nothing connects the two. An IP address does not contain a MAC address and a MAC address does not contain an IP address.
5. So the laptop has to ask. It shouts to the whole local network: “whoever has 192.168.0.1, tell me your hardware address”.
6. The router hears its own IP address in the question and answers directly: “192.168.0.1 is at this MAC address”.
7. That question-and-answer is ARP, the address resolution protocol.
8. The laptop writes the answer down in a small table so it does not have to ask again for every packet.
9. That table is the ARP cache, and entries in it expire after a few minutes.
10. ARP is only for the local network. To reach something far away, the laptop ARPs for the router and sends everything there. The router deals with the rest.

PLAIN23.10.2 a picture in your head#

1. You are in a large office and you have a note to hand to “the head of accounts”. You do not know who that is or where they sit.
2. You stand up and call out to the whole floor: “who is the head of accounts?”
3. Everyone hears. Everyone except one person ignores you.
4. One person calls back: “that is me, third desk by the window.”
5. You write that down on a sticky note on your monitor, and from then on you walk straight there.
6. A few weeks later the sticky note is out of date, because someone else took the job. So you throw the note away periodically and ask again.
7. Notice the danger. When you called out, anyone could have answered. Nobody checks credentials. The first confident reply wins.

Where this comparison breaks: in an office, a wrong answer would be noticed socially within a day. On a network, a false ARP reply is silently believed forever, or at least until the entry ages out, and the victim has no way at all to tell a true reply from a false one. ARP has no authentication of any kind. This is not a bug that was later fixed; it is the design, and the design is from 1982.

PLAIN23.10.3 a worked example#

1. This is the reader’s own network. The laptop is at 192.168.0.107 and the router is at 192.168.0.1. MAC addresses are illustrative, because the session record does not include them.

Step 1: the request, broadcast to everyone

Ethernet dest : FF:FF:FF:FF:FF:FF   (everyone)
Ethernet src  : 3C:22:FB:8A:41:D7   (the laptop)
EtherType     : 0x0806              (this is ARP)
  Opcode      : 1                   (request)
  Sender MAC  : 3C:22:FB:8A:41:D7
  Sender IP   : 192.168.0.107
  Target MAC  : 00:00:00:00:00:00   (unknown, that is
                                     the whole question)
  Target IP   : 192.168.0.1

Step 2: the reply, sent to one machine only

Ethernet dest : 3C:22:FB:8A:41:D7   (back to the laptop)
Ethernet src  : 5C:A6:E6:11:22:33   (the router)
EtherType     : 0x0806
  Opcode      : 2                   (reply)
  Sender MAC  : 5C:A6:E6:11:22:33
  Sender IP   : 192.168.0.1
  Target MAC  : 3C:22:FB:8A:41:D7
  Target IP   : 192.168.0.107

2. Notice the request is a broadcast and the reply is a unicast. The question must reach everyone. The answer only needs to reach the asker.
3. Notice also that the router learns the laptop’s mapping for free, from the sender fields of the request. Both sides end up with a cache entry after one exchange.
4. Now the laptop can build a real frame. To send an IP packet to GitHub at 20.207.73.82, it puts:
   1. Destination MAC: 5C:A6:E6:11:22:33, the router.
   2. Source MAC: 3C:22:FB:8A:41:D7, itself.
   3. Destination IP: 20.207.73.82, GitHub.
   4. Source IP: 192.168.0.107, itself.
5. Read that again, because it is the sentence that confuses everyone. The MAC addresses are local and change at every hop. The IP addresses are end to end and do not change.
6. Here is the reader’s ARP cache, read on macOS:

$ arp -a
? (192.168.0.1) at 5c:a6:e6:11:22:33 on en0 ifscope [ethernet]
? (192.168.0.107) at 3c:22:fb:8a:41:d7 on en0 [ethernet]
? (192.168.0.255) at ff:ff:ff:ff:ff:ff on en0 ifscope [ethernet]

7. The question marks are hostnames macOS could not resolve. en0 is the interface. ifscope means the entry is bound to that specific interface.

PLAIN23.10.4 what is really happening inside#

1. Before sending any IP packet, the machine asks one question: is this destination on my own local network, or somewhere else?
2. It answers that by comparing the destination IP against its own IP and subnet mask. The next chapter covers exactly how that comparison works.
3. If the destination is local, it ARPs for the destination itself.
4. If the destination is not local, it does not ARP for the destination at all. It ARPs for the default gateway, which here is 192.168.0.1.
5. This is why every machine on the internet has a cache entry for its router and for nothing else outside its own street.
6. Packets waiting for an ARP reply sit in a small queue. If no reply comes, the request is retried a few times and then the packets are dropped and an error is reported to the application.
7. A gratuitous ARP is an unsolicited announcement: a machine broadcasts an ARP for its own address, which nobody asked for.
8. It has two honest uses. It tells everyone to update their cache when an address moves to new hardware, and it detects duplicate addresses at startup.
9. It also has a dishonest use. A machine can send gratuitous replies claiming to own the router’s IP address, and every machine on the network will believe it.
10. That is ARP spoofing. The attacker now receives traffic meant for the router, can read it, change it, and pass it on. It is a textbook man-in-the-middle attack, and it requires only being on the same network.

TECHNICAL23.10.5 the engineer’s version#

1. ARP is defined in RFC 826, published November 1982, by David C. Plummer. It is Internet Standard STD 37 and it has never been revised.
2. It rides directly on Ethernet with EtherType 0x0806. It is not carried in IP, so it is neither TCP nor UDP and has no port number.
3. The packet is 28 bytes for the IPv4-over-Ethernet case:

4. 28 bytes is below the 46-byte minimum payload, so 18 bytes of padding are always added. Some old implementations leaked memory contents in that padding; this was named “Etherleak” and assigned CVE-2003-0001.
5. Cache timeouts are implementation details, not standards. On macOS and other BSD-derived systems the default is 1200 seconds, tunable with sysctl net.link.ether.inet.arp_keep on some versions. On Linux the relevant knobs are net.ipv4.neigh.default.base_reachable_time_ms, default 30000, and gc_stale_time, default 60 seconds. Cisco IOS defaults to 14400 seconds, four hours, which is far longer than most people expect.
6. Variants worth knowing by name: proxy ARP (RFC 1027, 1987), where a router answers on behalf of a host on another segment; ARP probe and ARP announcement (RFC 5227, 2008), used for duplicate address detection; and RARP (RFC 903, 1984), the obsolete reverse lookup replaced by BOOTP and DHCP.
7. IPv6 does not use ARP at all. It uses Neighbor Discovery Protocol, RFC 4861 (2007), which runs over ICMPv6 and uses multicast rather than broadcast. The reader’s session reported IPv6: (none), so no NDP was in play in this case.
8. ARP spoofing defences: dynamic ARP inspection on managed switches, which validates ARP packets against the DHCP snooping binding table; static ARP entries for critical addresses; 802.1X port authentication; and simply encrypting everything so that intercepting the traffic gains the attacker little. TLS, covered later in Part F, is the real answer.
9. Commands to observe and manipulate ARP on macOS:

# show the whole table
arp -a

# show one entry, numeric only
arp -n 192.168.0.1

# delete one entry, forcing a fresh ARP
sudo arp -d 192.168.0.1

# delete every entry
sudo arp -a -d

# add a permanent static entry
sudo arp -S 192.168.0.1 5c:a6:e6:11:22:33

# watch the exchange live
sudo tcpdump -i en0 -e -n arp

10. On Linux the modern equivalents are ip neigh show, ip neigh flush all and ip neigh replace. The old arp command still works but is deprecated in favour of ip.
11. Diagnostic value: if arp -a shows no entry for your gateway, and deleting and re-pinging does not create one, the problem is beneath IP entirely. Your machine is not reaching the router at layer 2. That is a different fault from the reader’s, whose ARP was clearly fine, since traffic reached hop 1 and twelve hops beyond it.
12. The honest version of “ARP is a local protocol”: ARP messages are never forwarded by routers, so an ARP table only ever contains addresses on your own link. Any tool that appears to show you a remote machine’s MAC address is showing you something else, usually a router’s.

WORDS23.10.6 remember these#

1. ARP — asking who owns an IP address on this network — the Address Resolution Protocol, RFC 826, EtherType 0x0806.
2. ARP request — the broadcast question — opcode 1, sent to FF:FF:FF:FF:FF:FF with the target hardware address zeroed.
3. ARP reply — the direct answer — opcode 2, unicast back to the requester.
4. ARP cache — the notes you keep so you need not ask again — the neighbour table mapping IP to MAC, with a per-entry ageing timer.
5. Default gateway — the router you send everything distant to — the next hop for destinations outside your own subnet.
6. Gratuitous ARP — announcing yourself without being asked — an unsolicited ARP with sender and target protocol addresses equal.
7. ARP spoofing — lying about who owns an address — forging ARP replies to redirect traffic, enabling man-in-the-middle interception.
8. Proxy ARP — a router answering on someone else’s behalf — RFC 1027 behaviour where a gateway replies for hosts it can reach.
9. Neighbor Discovery — the IPv6 replacement for ARP — RFC 4861, running over ICMPv6 with multicast solicitations.

23.11 DHCP: how the laptop got an address#

PLAIN23.11.1 in simple words#

1. Nobody configured the reader’s laptop with an IP address. It joined the WiFi and simply had one, in the 192.168.0.x range.
2. Something handed it that address. That something is a DHCP server, and in a home it lives inside the router at 192.168.0.1.
3. DHCP stands for dynamic host configuration protocol. It automates the whole job of joining a network.
4. There is a chicken-and-egg problem. To ask for an address, you must send a message. To send a message, you normally need an address.
5. DHCP solves it by letting the client send from the address 0.0.0.0, meaning “no address yet”, to the address 255.255.255.255, meaning “everyone here”.
6. The conversation takes four messages, and they have a memorable name: DORA.
7. Discover: the client shouts “is there a DHCP server out there?”
8. Offer: a server replies “yes, and you may have this address”.
9. Request: the client says “I accept that address” — and it says it out loud so any other servers know they were not chosen.
10. Acknowledge: the server confirms and records the assignment.
11. The address is not given away. It is lent, for a fixed period called a lease, and the client must renew before it runs out.

PLAIN23.11.2 a picture in your head#

1. Picture arriving at a large hotel that has no reception desk you can see.
2. You stand in the lobby and call out: “does anyone here give out rooms?” That is Discover.
3. A clerk calls back: “yes, room 107 is free, it comes with a floor plan, the number for the concierge, and it is yours for one day.” That is Offer.
4. You call back, loudly enough for the whole lobby: “I will take room 107 from that clerk.” That is Request. The loudness matters, because a second clerk who also had a room ready now knows to put it back.
5. The clerk writes your name in the register and hands you the key. That is Acknowledge.
6. Halfway through the day, you go and ask to extend. Usually you get it. If the clerk has gone, you keep trying, and when the time is nearly up you start asking any clerk at all.
7. If nobody at all answers your first call, you do not stand in the lobby forever. You pick a corner of the lobby to sit in and hope somebody notices. That corner is the 169.254.x.x address, and it means you never got a room.

Where this comparison breaks: in a hotel, a second clerk handing out the same room would be caught by the register. On a network, two DHCP servers on the same segment genuinely can hand out conflicting addresses, and the usual cause is someone plugging a home router into an office network by mistake. There is no central register. There is only a convention that clients take the first offer and servers ping an address before offering it.

PLAIN23.11.3 a worked example#

1. This is the reader’s laptop joining their home network. The address 192.168.0.107 is illustrative; the session record confirms only that it was in the 192.168.0.x range.

1. DISCOVER   client -> broadcast
   Ethernet dst : FF:FF:FF:FF:FF:FF
   IP  src      : 0.0.0.0
   IP  dst      : 255.255.255.255
   UDP src port : 68
   UDP dst port : 67
   Contains     : client MAC, requested options

2. OFFER      server -> client
   IP  src      : 192.168.0.1
   IP  dst      : 255.255.255.255 (or the offered IP)
   UDP src port : 67
   UDP dst port : 68
   Contains     : yiaddr 192.168.0.107, mask, gateway,
                  DNS servers, lease time

3. REQUEST    client -> broadcast
   IP  src      : 0.0.0.0
   IP  dst      : 255.255.255.255
   UDP 68 -> 67
   Contains     : "I want 192.168.0.107 from 192.168.0.1"

4. ACK        server -> client
   IP  src      : 192.168.0.1
   UDP 67 -> 68
   Contains     : final confirmed values and lease time

2. Note the ports. The server always listens on 67. The client always listens on
   68. They are fixed, and a client that used a random source port would never hear the reply.
3. Note that Request is broadcast even though the client now knows the server’s address. That is deliberate, so other servers can withdraw their offers.
4. Suppose the lease is 86400 seconds, one day. Two timers are set from it.
5. T1, the renewal timer, fires at 50 percent: 43200 seconds, twelve hours in. The client asks its own server directly, by unicast, to extend.
6. T2, the rebinding timer, fires at 87.5 percent: 75600 seconds, twenty-one hours in. If the original server never answered, the client now broadcasts to any server at all.
7. If nothing answers by 86400 seconds, the client must stop using the address entirely and start again from Discover.
8. The reader can see all of this on macOS:

# current lease and everything the server sent
ipconfig getpacket en0

# just the router and DNS the server supplied
ipconfig getoption en0 router
ipconfig getoption en0 domain_name_server

# force a fresh lease
sudo ipconfig set en0 DHCP

# watch the whole DORA exchange live
sudo tcpdump -i en0 -n port 67 or port 68

PLAIN23.11.4 what is really happening inside#

1. A DHCP server does not just give an address. It gives a whole configuration, and this is the part people forget.
2. The standard things it hands out are the IP address, the subnet mask, the default gateway, the DNS servers, the domain name and the lease time.
3. It can also hand out time servers, TFTP boot servers, and dozens of other things, each identified by a number.
4. Here is a fact from the reader’s session worth pausing on. Their DNS resolver was 1.1.1.1, the Cloudflare public resolver, not the router.
5. That means either the DHCP server was configured to hand out 1.1.1.1, or somebody overrode DHCP’s suggestion on the laptop itself. Both are common. The session data does not say which.
6. Now the failure case, which is the diagnostically useful one.
7. If no DHCP server answers, the operating system assigns itself an address in the range 169.254.x.x, picked at random and checked for clashes with ARP.
8. That is a link-local address. Two machines on the same cable with 169.254 addresses can talk to each other, and to nothing else at all.
9. So seeing 169.254.x.x on your machine is not a random fault. It is a specific message, and the message is: “I never heard from a DHCP server”.
10. That single observation narrows the fault enormously. The cable or radio link is probably up, since the machine is on a network. But nothing answered.
11. Common causes: the router’s DHCP service is off or crashed, the pool is exhausted, the machine is on the wrong VLAN, or the DHCP relay is missing on a routed segment.
12. The reader never saw 169.254. Their laptop held a valid 192.168.0.x address throughout, and reached twelve hops out. So DHCP was not the fault here. This is what ruling something out looks like.

TECHNICAL23.11.5 the engineer’s version#

1. Lineage: RARP, RFC 903 (June 1984). BOOTP, RFC 951 (September 1985). DHCP, RFC 1531 (October 1993), then RFC 1541, then the current RFC 2131 (March 1997). Options are in RFC 2132 (March 1997). DHCPv6 is RFC 3315 (2003), superseded by RFC 8415 (2018).
2. DHCP is backward compatible with BOOTP and reuses its packet format, which is why the fields still have BOOTP names such as yiaddr (your IP address), siaddr (next server), giaddr (relay agent address) and chaddr (client hardware address).
3. Message types are carried in option 53. The eight defined values are DISCOVER 1, OFFER 2, REQUEST 3, DECLINE 4, ACK 5, NAK 6, RELEASE 7 and INFORM
4. Options seen in nearly every real exchange:

5. Timers per RFC 2131: T1 defaults to 0.5 times the lease duration, T2 to 0.875 times it. Servers may override both with options 58 and 59. At T1 the client unicasts a REQUEST in RENEWING state. At T2 it broadcasts in REBINDING state.
6. A server may reply DHCPNAK if the requested address is no longer valid, for example after a laptop moves between networks and asks for its old address. The client must then restart from DISCOVER.
7. DHCP is UDP, not TCP. Ports 67 (server, “bootps”) and 68 (client, “bootpc”) are fixed by IANA. Both are below 1024, so a DHCP client needs privilege to bind port 68.
8. Broadcasts do not cross routers, so on any network larger than one segment a DHCP relay agent, also called an IP helper, forwards DISCOVER messages as unicast to a central server and records its own address in giaddr so the server knows which pool to use. RFC 3046 (2001) adds option 82, the relay agent information option, which carries the physical port the request came from.
9. Static assignment versus reservation, and the distinction matters operationally:
   1. A static address is configured on the client itself. The DHCP server does not know about it and may hand the same address to someone else.
   2. A reservation, or static lease, is configured on the server and keyed to the client’s MAC address. The client still runs normal DORA and still gets the gateway, DNS and domain automatically.
   3. Reservations are almost always the right choice, because configuration stays in one place. Their weakness is MAC randomization, which changes the key the reservation is matched on.
10. IPv4 link-local is 169.254.0.0/16, defined in RFC 3927 (May 2005). The first and last 256 addresses, 169.254.0.0/24 and 169.254.255.0/24, are reserved. Microsoft calls the mechanism APIPA. The IPv6 equivalent is fe80::/10 per RFC 4291 (February 2006), and it is always present, not a fallback.
11. Security notes: DHCP has no authentication. A rogue server can hand out its own address as the gateway and intercept everything. The defence on managed switches is DHCP snooping, which permits server replies only on ports explicitly marked trusted, and builds a binding table that dynamic ARP inspection then uses.
12. Observation on Linux: dhclient -v, networkctl status, journalctl -u systemd-networkd, or the lease files under /var/lib/dhcp/. On macOS the lease lives under /var/db/dhcpclient/leases/.

WORDS23.11.6 remember these#

1. DHCP — the thing that gives your device an address automatically — Dynamic Host Configuration Protocol, RFC 2131, over UDP ports 67 and 68.
2. DORA — the four-step exchange — Discover, Offer, Request, Acknowledge.
3. Lease — the address is lent, not given — a time-limited allocation with renewal at T1 and rebinding at T2.
4. T1 and T2 — when to ask for more time — 50 percent and 87.5 percent of the lease duration by default.
5. Default gateway — the router address DHCP hands you — option 3, the next hop for non-local destinations.
6. Reservation — always giving the same device the same address — a server-side static lease keyed on the client MAC address.
7. Relay agent — how DHCP crosses a router — an IP helper that unicasts client broadcasts to a central server and fills in giaddr.
8. Link-local address — the address you give yourself when nothing answers — 169.254.0.0/16 per RFC 3927, reachable only on the same link.
9. DHCP snooping — the switch refusing to believe rogue servers — a security feature permitting server messages only on trusted ports.

23.12 The network interface card#

PLAIN23.12.1 in simple words#

1. The network interface card, or NIC, is the piece of hardware that turns data in memory into signals on the medium, and back.
2. In a laptop it is not a card at all. It is a chip soldered to the board, and on WiFi it is a radio chip with an antenna.
3. It has four jobs. Frame the data, put the signal on the wire, filter incoming frames by address, and check the FCS.
4. It cannot do any of this alone. It needs a driver, which is the piece of operating system software that knows how to talk to that exact chip.
5. The card and the operating system share a set of memory slots called ring buffers. The name is because the slots are used in a circle, over and over.
6. There is one ring for outgoing frames and one for incoming.
7. To send, the operating system writes a frame into a free slot and tells the card. The card copies it out and marks the slot free again.
8. To receive, the card writes into a free slot and raises an interrupt, a hardware signal that tells the CPU to stop what it is doing and look.
9. At high speed, interrupting for every single frame would consume the whole CPU. So the card waits a little and interrupts once for a batch instead.
10. Normally the card throws away frames not addressed to it. If you turn that filtering off, it passes up everything it hears. That is how packet capture works.

PLAIN23.12.2 a picture in your head#

1. Picture a busy loading bay behind a shop, with a row of numbered bays around a circular driveway.
2. The staff inside load goods into a free bay and raise a small flag. The driver outside takes whatever bay has a flag, loads the lorry, and lowers the flag.
3. Deliveries arriving work the same way in reverse. The driver fills a free bay and rings a bell so the staff come out.
4. If lorries arrive every few seconds, ringing the bell each time means the staff never get anything else done.
5. So the driver waits until three lorries have arrived, or ten seconds have passed, then rings once. That is interrupt coalescing.
6. If all the bays are full because the staff are busy, arriving lorries are turned away. Those are dropped packets, and they are counted.

Where this comparison breaks: the flags in a real NIC are not checked by a person walking out to look. The card writes directly into main memory over the system bus, with no CPU involvement at all, using DMA. The CPU only finds out afterwards. Also, a lorry turned away can come back tomorrow. A dropped frame is gone, and it is entirely up to a higher-layer protocol to notice and resend it.

PLAIN23.12.3 a worked example#

1. Here is a real captured frame, decoded field by field. It is the first frame of an attempt to reach GitHub from the reader’s laptop. MAC addresses are illustrative.

Raw bytes as tcpdump -xx would show them:

5c a6 e6 11 22 33 3c 22 fb 8a 41 d7 08 00 45 00
00 3c 1c 46 40 00 40 06 xx xx c0 a8 00 6b 14 cf
49 52 e3 f4 01 bb ...

Decoded:

5c a6 e6 11 22 33   Destination MAC, the router
3c 22 fb 8a 41 d7   Source MAC, the laptop
08 00               EtherType 0x0800, so IPv4
45                  IP version 4, header length 5 words
00                  Differentiated services, none set
00 3c               Total length 60 bytes
1c 46               Identification
40 00               Flags: Don't Fragment set
40                  TTL 64, the usual Linux/macOS start
06                  Protocol 6, which is TCP
xx xx               Header checksum
c0 a8 00 6b         Source IP 192.168.0.107
14 cf 49 52         Destination IP 20.207.73.82
e3 f4               Source port 58356
01 bb               Destination port 443

2. 14 cf 49 52 in decimal is 20, 207, 73, 82. That is exactly the address the reader’s curl printed: Trying 20.207.73.82:443....
3. 01 bb in decimal is 443, the HTTPS port.
4. Notice the destination MAC is the router, not GitHub. GitHub is not on this network, so the frame is addressed locally and routed onwards.
5. Notice the TTL is 64. Every router that forwards this packet subtracts one. That mechanism is what makes traceroute possible, and it gets a chapter of its own later in Part F.
6. Everything from 45 onwards is the IP packet, which is the next chapter. Everything before it is this chapter.
7. In the reader’s outage, frames exactly like this one went out and nothing ever came back. The frame was correct. The path was not.

PLAIN23.12.4 what is really happening inside#

1. A modern NIC does far more than move bytes. It does work that the CPU used to do, and this is called offload.
2. Checksum offload means the card computes the IP and TCP checksums in hardware as the frame goes out, and verifies them on the way in.
3. This has a confusing side effect for anyone capturing traffic. A capture taken on the sending machine shows outgoing checksums as invalid, because the card has not filled them in yet. That is normal, not a fault.
4. Segmentation offload means the operating system hands the card one huge block, say 64 kilobytes, and the card chops it into 1500-byte frames itself, generating all the headers.
5. That turns forty-odd trips through the network stack into one, which is a large saving at gigabit speeds and above.
6. The same trick in reverse merges many arriving small frames into one big one before handing it up. A capture on such a machine will show impossible 64-kilobyte “frames”. They never existed on the wire.
7. Promiscuous mode turns off the address filter. The card passes up every frame it receives, regardless of destination.
8. On a switched network that gains you less than people expect, because the switch already only sends you frames for you, plus broadcasts and floods.
9. To capture everything on a switched network you need a mirror port, where the switch is configured to copy all traffic from certain ports to yours.
10. On WiFi, promiscuous mode is not enough either. You need monitor mode, which captures raw radio frames including management frames, and on many drivers it means disconnecting from the network to do it.

TECHNICAL23.12.5 the engineer’s version#

1. The descriptor rings are arrays of small structures in host memory. Each descriptor holds a physical buffer address, a length and status bits. The driver owns some entries, the card owns others, and ownership is passed by writing a status bit and advancing a tail pointer.
2. Data movement is DMA, direct memory access, so the frame body never passes through the CPU. On systems with an IOMMU the addresses are translated and bounded, which is what stops a compromised NIC from reading arbitrary memory.
3. Interrupt coalescing is configured by two thresholds: a packet count and a time. Tuned with ethtool -C eth0 rx-usecs 50 rx-frames 32 on Linux. Lower values reduce latency and raise CPU usage; higher values do the opposite.
4. Linux replaced pure interrupt-driven receive with NAPI in kernel 2.4.20 (2002). On the first interrupt the driver disables further receive interrupts and switches to polling, then re-enables them when the ring drains. This is what stops a machine from livelocking under a packet flood.
5. Offload features, with their Linux names:

6. Receive side scaling hashes each flow’s addresses and ports to choose one of several receive queues, each bound to a different CPU core. Without it a single core would have to handle a whole 10 or 100 Gbit/s interface.
7. Modern high-speed NICs support SR-IOV, presenting many virtual functions to virtual machines directly, and kernel-bypass frameworks such as DPDK and AF_XDP that let user-space code poll the ring itself. That is how software routers reach line rate.
8. Packet capture on Unix goes through BPF, the Berkeley Packet Filter, described by Steven McCanne and Van Jacobson in the 1993 USENIX paper “The BSD Packet Filter: A New Architecture for User-level Packet Capture”. The filter expression is compiled to bytecode and run in the kernel, so unwanted packets are discarded before they are copied to user space.
9. Linux later generalized this into eBPF, which now runs far more than packet filters, and into AF_PACKET with a memory-mapped ring for capture.
10. tcpdump was written at Lawrence Berkeley Laboratory starting in 1988 by Van Jacobson, Craig Leres and Steven McCanne. libpcap, the capture library underneath it, came from the same group and is what almost every capture tool uses today.
11. Wireshark began as Ethereal, written by Gerald Combs in 1998, and was renamed Wireshark in 2006 over a trademark issue. Its command-line form is tshark.
12. Useful capture commands:

# macOS: capture ARP and DHCP with Ethernet headers shown
sudo tcpdump -i en0 -e -n 'arp or port 67 or port 68'

# capture to a file for Wireshark, full frames
sudo tcpdump -i en0 -s 0 -w capture.pcap

# only traffic to the address curl was trying
sudo tcpdump -i en0 -n host 20.207.73.82

# show raw bytes including the Ethernet header
sudo tcpdump -i en0 -xx -c 1

13. On macOS, tcpdump on a WiFi interface will not show other stations’ traffic even with -p disabled, because the radio decrypts only frames for you. Apple’s airport utility historically enabled monitor mode, but it was removed in macOS 14.4 (2024); the supported route now is the Wireless Diagnostics application. That is an implementation detail and it has changed more than once.
14. Interface statistics worth reading before anything else: netstat -i on macOS, ip -s link on Linux, and ethtool -S eth0 for the full vendor-specific counter set. Non-zero Ierrs, Oerrs or rx_crc_errors point at a physical problem: a bad cable, a dirty optic, or a duplex mismatch.
15. The honest version: a capture taken on the machine under test is not the truth about the wire. Offloads reshape it, the driver may drop frames before the filter, and the timestamps come from the host clock unless the NIC does hardware timestamping. For a definitive answer you capture from a mirror port or a passive tap on another machine.

WORDS23.12.6 remember these#

1. NIC — the hardware that connects you to the network — network interface controller, implementing the physical and MAC layers.
2. Driver — the software that knows this exact chip — the kernel module managing descriptor rings, interrupts and offload configuration.
3. Ring buffer — a circle of memory slots shared with the card — a descriptor ring with head and tail pointers and DMA buffer addresses.
4. DMA — the card writing to memory by itself — direct memory access, moving frame data without CPU involvement.
5. Interrupt coalescing — batching the “look at me” signals — delaying interrupts by packet count or time to trade latency for CPU cost.
6. Checksum offload — the card doing the arithmetic — hardware computation and verification of IP, TCP and UDP checksums.
7. Segmentation offload — handing the card one big block — TSO or GSO, splitting a large buffer into MTU-sized frames in hardware or late in the stack.
8. Promiscuous mode — accepting frames not addressed to you — disabling the MAC destination filter for capture.
9. Mirror port — a switch port that copies other ports’ traffic — SPAN or port mirroring, required to capture on a switched network.
10. BPF — the filter that decides what to capture — Berkeley Packet Filter bytecode executed in the kernel, used by libpcap and tcpdump.

23.13 Putting the local network together#

PLAIN23.13.1 in simple words#

1. We now have every piece. Let us watch the reader’s laptop join the network from cold, in order, naming everything.
2. It starts with radio, because the reader is on WiFi. The card scans for networks, picks one, and authenticates and associates with the access point.
3. From that moment the laptop is on a network, but it has no address and knows nothing about it.
4. So it asks. It broadcasts a DHCP Discover from 0.0.0.0 to 255.255.255.255.
5. The router answers with an Offer containing an address in the 192.168.0.x range, the mask, the gateway 192.168.0.1, and the DNS servers.
6. The laptop broadcasts a Request accepting it, and the router sends an Acknowledge. The laptop now has an address and a lease.
7. Before it uses that address, it usually checks nobody else has it, by sending an ARP probe for its own address and listening for a reply.
8. Now it wants to reach something. It compares the destination against its own subnet and finds the destination is elsewhere.
9. So it needs the router’s hardware address. It broadcasts an ARP request for 192.168.0.1 and gets a reply.
10. Only now can the first real frame be built and sent. Everything up to this point was preparation, and all of it is beneath IP.

PLAIN23.13.2 a picture in your head#

1. Think of arriving in a new city where you know nobody.
2. First you have to get through the door of the building. That is joining the WiFi.
3. Then you need an address of your own, so anyone can reply to you. That is DHCP.
4. Then you check nobody else already lives at that address. That is the ARP probe.
5. Then you need to know which door leads out of the building. That is the default gateway.
6. Then you need to know the physical location of that door, not just its name. That is ARP for the gateway.
7. Only then can you post your first letter.

Where this comparison breaks: a person does all of this once and remembers it for years. A laptop redoes almost all of it every time it wakes from sleep, changes network, or the lease timer expires. It also does it again silently every few minutes as ARP entries age out. The steady state of a network is not stillness; it is this whole sequence running quietly, over and over.

PLAIN23.13.3 a worked example#

1. Here is the full sequence for the reader’s laptop, from cold to first packet towards GitHub.

TIME  WHO        WHAT                          PROTOCOL
----  ---------  ----------------------------  ---------
t+0   laptop     scan for networks             802.11
t+1   laptop     authenticate, associate       802.11
t+2   both       4-way key handshake           802.11i
t+3   laptop     DHCP Discover, broadcast      DHCP/UDP68
t+3   router     DHCP Offer 192.168.0.107      DHCP/UDP67
t+3   laptop     DHCP Request, broadcast       DHCP/UDP68
t+3   router     DHCP Ack, lease granted       DHCP/UDP67
t+3   laptop     ARP probe for own address     ARP
t+4   laptop     ARP who has 192.168.0.1       ARP
t+4   router     ARP reply, here is my MAC     ARP
t+5   laptop     DNS query for github.com      DNS/UDP53
t+5   resolver   answer 20.207.73.82           DNS/UDP53
t+5   laptop     TCP SYN to 20.207.73.82:443   TCP
t+20  ...        nothing comes back at all     silence

2. Steps t+0 to t+4 are this chapter. Everything from t+5 belongs to later chapters in Part F, and each gets its own.
3. The DNS lookup went to 1.1.1.1, the Cloudflare public resolver, and it worked. The name resolved to 20.207.73.82, which is in a Microsoft-owned range; GitHub has been owned by Microsoft since 2018 and fronts traffic through Microsoft’s network edge.
4. So name resolution succeeded and the local network worked perfectly.
5. The TCP connection attempt then received no response at all for fifteen seconds. No SYN-ACK, no reset, no ICMP unreachable.
6. What that proves: layers 1 and 2 were healthy end to end on the local segment, DHCP worked, ARP worked, and DNS worked.
7. What it merely suggests: the packets were being discarded silently somewhere beyond the local network. The evidence is consistent with a silent drop on that path. It is not consistent with the server being down, because the same site loaded instantly over mobile data on the same phone.
8. We will not name a culprit. The observation supports a path-specific silent drop and nothing more.

PLAIN23.13.4 what is really happening inside#

1. Here is the same journey drawn as a picture, with the layers each hop cares about.

   LAPTOP                ROUTER              THE INTERNET
   192.168.0.107         192.168.0.1
   +----------+          +----------+
   | app      |          |          |
   | TCP      |          |          |
   | IP       |--------->| IP       |------> hop 2 ...
   | Ethernet |  frame   | Ethernet |
   | radio    |<-------->| radio    |
   +----------+  802.11  +----------+

   The frame stops at the router. It is unwrapped,
   the IP packet inside is read, and a brand new
   frame is built for the next hop.

   MAC addresses: change at every single hop.
   IP addresses : stay the same all the way.

2. That last pair of lines is the single most important idea in local networking, and the reason MAC and IP both exist.
3. The reader’s traceroute confirms it. Hop 1 was 192.168.0.1, the router in the flat. Hop 2 was 172.31.0.17, inside the ISP.
4. Between those two hops the frame was completely rebuilt, with different source and destination MAC addresses, on a different physical medium.
5. The IP packet inside was unchanged apart from the TTL being decremented by one and the header checksum being recomputed.
6. Hops 2 to 6 in the reader’s trace were private addresses in the RFC 1918 ranges, belonging to the ISP’s own core. 172.16.0.0/12 covers 172.16.x through 172.31.x, so 172.31.0.17, 172.26.22.235 and 172.16.18.33 are all private.
7. Hop 3, 137.97.29.249, was a public address belonging to the ISP.
8. From hop 7 the path entered Microsoft’s backbone at ntwk.msn.net, went through Delhi, then Mumbai, then Pune, and then produced nothing.
9. All of that is layer 3 and above, and it is the subject of the next several chapters. This chapter ends where the frame ends: at the router in the flat.

TECHNICAL23.13.5 the engineer’s version#

1. Sequence with the exact standards involved, for a WPA2 or WPA3 personal WiFi network:
   1. Probe request and response, or passive beacon reception. IEEE 802.11.
   2. Open system authentication, then association request and response. IEEE 802.11.
   3. 4-way handshake deriving the pairwise transient key. IEEE 802.11i, ratified 2004, marketed as WPA2. WPA3 (2018) replaces the pre-shared key exchange with SAE.
   4. DHCP DORA. RFC 2131, UDP 67 and 68.
   5. ARP announcement or probe for the assigned address. RFC 5227.
   6. ARP request for the default gateway. RFC 826.
   7. DNS query. RFC 1035, and later chapters cover it.
   8. TCP three-way handshake. RFC 9293, which in 2022 replaced RFC 793 from
2. On a wired port, replace the first three steps with autonegotiation, and optionally 802.1X authentication using EAP over LAN before the port is opened.
3. Everything above layer 2 in that list is deferred. IP addressing and subnets, NAT, DNS, routing and traceroute, TCP, and TLS each get a full chapter in Part F.
4. Diagnostic ordering follows the same sequence, and this is the practical payoff of the whole chapter. Test bottom-up and stop at the first failure.

5. The reader’s fault sits on the last row. Every row above it had been passed.
6. A more precise statement of the observation: a TCP SYN was transmitted to 20.207.73.82 port 443 and no segment, no ICMP message, and no RST arrived within 15 seconds. Under RFC 9293 the client will retransmit the SYN with exponential backoff before giving up, which is why the wait was long and featureless rather than an immediate error.
7. An immediate “connection refused” would have been a TCP RST, proving something answered. An “unreachable” message would have been ICMP type 3, proving a router made a decision and told you. Silence proves only that nothing chose to speak.
8. Silence is the least informative failure mode in networking, and it is the most common one on the modern internet, because filtering devices are usually configured to drop rather than reject.

WORDS23.13.6 remember these#

1. Association — joining a WiFi network — the 802.11 exchange that binds a station to an access point before any data frames.
2. Default gateway — the way out of your network — the router address used for any destination outside your own subnet.
3. Encapsulation — wrapping data in a header at each layer — a TCP segment inside an IP packet inside an Ethernet frame.
4. Hop — one router-to-router step — a single layer 3 forwarding decision, with a complete new layer 2 frame each time.
5. TTL — the counter that stops packets looping forever — time to live, decremented at every hop, packet discarded at zero.
6. Silent drop — a packet discarded with no message — a firewall DROP rather than a REJECT, giving the sender no information.
7. Proven versus suggested — what an observation actually establishes — the discipline of separating direct evidence from inference, which is the whole method of Part F.

23.98 Common wrong ideas#

1. Wrong: bandwidth and speed are the same thing. Right: bandwidth is capacity per second, latency is delay before anything arrives. Doubling capacity does not reduce delay by a single microsecond.
2. Wrong: a MAC address travels with the packet across the internet. Right: it is replaced at every single hop. The MAC addresses in a frame are only ever the two ends of one physical link.
3. Wrong: a switch and a hub do the same job. Right: a hub copies every frame to every port and shares one collision domain. A switch learns and forwards to one port, giving each port its own collision domain.
4. Wrong: collisions are a normal part of modern Ethernet. Right: on a switched full-duplex link collisions are impossible. If your interface counters show collisions, you have a duplex mismatch or a real hub.
5. Wrong: a 1500-byte MTU is a physical limit of the cable. Right: it is a number chosen in the 1980 DIX specification to limit how long one station could hold a shared coax cable, kept ever since for compatibility.
6. Wrong: category 6 cable makes your internet faster. Right: cable category sets the maximum a link can carry over 100 m. If your line delivers 100 Mbit/s, Cat 5e was already twenty times more than enough.
7. Wrong: ARP is part of IP. Right: ARP rides directly on Ethernet with EtherType 0x0806. It is not carried inside IP, has no port number, and never crosses a router.
8. Wrong: seeing 169.254.x.x means the network card is broken. Right: it means no DHCP server answered. The link is probably up. That single observation rules out several faults and points at one.
9. Wrong: a traceroute ending in stars proves the destination is unreachable. Right: many routers rate-limit or block ICMP time-exceeded replies. Silence at the end of a trace is normal and proves nothing on its own.
10. Wrong: MAC address filtering secures a WiFi network. Right: addresses are broadcast in clear text in every frame header and can be changed in one command. Since MAC randomization became the phone default it does not even work as an inconvenience.
11. Wrong: fibre is faster because light is faster than electricity. Right: a signal moves through fibre at about two-thirds of the vacuum speed of light, similar to copper. Fibre wins on attenuation and bandwidth over distance, not on propagation speed.
12. Wrong: a packet capture on your own machine shows exactly what was on the wire. Right: segmentation and checksum offload reshape what you see. Frames larger than the MTU and invalid outbound checksums are normal artefacts.

23.99 Chapter summary in 20 lines#

1. A network is machines sharing a medium, and every network must solve addressing, framing, error detection and medium sharing.
2. Topologies are bus, star, ring and mesh; modern local networks are stars and the internet core is a mesh.
3. A wire carries voltages, not numbers, and plain high-low signalling fails on clock recovery and DC balance.
4. Line codes fix this: Manchester doubles the bandwidth, 4B/5B and 8b/10b cost 25 percent, 64b/66b costs 3.1 percent, PAM-4 trades margin for symbols.
5. Twisting cancels interference because both wires spend equal time nearest the noise, and the receiver reads only the difference.
6. Cable categories are frequency ratings: Cat 5e at 100 MHz, Cat 6A at 500 MHz, Cat 8 at 2000 MHz, with 100 m channels except Cat 8’s 30 m.
7. Fibre guides light by total internal reflection; single-mode goes tens of kilometres, multi-mode a few hundred metres, and SFP modules set the reach.
8. Bandwidth, throughput, latency and jitter are four different things, and the bandwidth-delay product explains why a fast link can still feel slow.
9. The reader’s line was not slow. It was silent, which is a different fault entirely and needs a different diagnosis.
10. Ethernet began at Xerox PARC on 22 May 1973 with Metcalfe and Boggs at 2.94 Mbit/s, was standardized as IEEE 802.3 in 1985, and now runs to 800 Gbit/s.
11. CSMA/CD and the 64-byte minimum frame exist because of collision timing on a shared cable, and neither matters on a modern full-duplex link.
12. The frame is preamble 7, SFD 1, destination 6, source 6, EtherType 2, payload 46 to 1500, FCS 4, for 64 to 1518 bytes.
13. The 1500-byte MTU came from the 1980 DIX specification and has never changed; fragmentation costs headers, memory and all-or-nothing loss.
14. A MAC address is 48 bits: a 24-bit vendor OUI plus a device part, with bits for multicast and for locally administered addresses.
15. Phones and laptops now randomize their MAC addresses by default, which breaks MAC filtering and MAC-keyed DHCP reservations.
16. A switch learns which address is on which port by reading source addresses, floods when it does not know, and ages entries after about 300 seconds.
17. VLANs split one switch into several broadcast domains using a 4-byte 802.1Q tag, and spanning tree stops loops that would otherwise storm in seconds.
18. ARP maps an IP address to a MAC address by broadcast question and unicast answer, has no authentication at all, and never crosses a router.
19. DHCP hands out address, mask, gateway, DNS and more in four messages over UDP ports 67 and 68, with renewal at 50 percent and rebinding at 87.5 percent.
20. Everything in this chapter happens before a single IP packet leaves the building, and the next six chapters take that packet the rest of the way.