There is a lot of code between the wire and your breakpoint
Set a breakpoint on the first line of a Go HTTP handler and look at what the debugger hands you.
r.Method "POST"
r.URL.Path "/api/save"
r.Header map[14 entries]
r.Host "example.com"
r.ContentLength 9
r.Body (*http.body)(0xc0001a4040)
It looks like a request arrived.
Nothing arrived. A *http.Request was manufactured, by four systems that don't know about each other, out of bytes that stopped resembling anything on the wire several steps earlier. Most of what was actually transmitted has been permanently destroyed by the time you're reading that struct — not hidden, not expensive to get at, destroyed.
And r.Body doesn't contain the body. It's a reader aimed back at a socket that may not have received it yet.
I've written the first half of this journey before: Journey of a Packet: From Wire to User Space covers the NIC, DMA, the RX ring, and the softirq that carries a frame up into the kernel's network stack. What I never wrote is what happens after that, and — more useful — how the halves fit together into one picture you can hold. This is that picture. The deep dives come after it.
It's HTTP in the examples because HTTP is what most people are debugging. The phases below are not HTTP's. Postgres, SSH, Redis and FTP all climb the same ladder and only diverge near the top.
Seven phases, and the boundaries between them
The phases are the easy part. The boundaries are where the article is.
Every boundary does two things at once: it changes the unit you can think in, and it throws information away. At the bottom of the ladder the unit is a voltage transition. At the top it's a struct with named fields. Nothing carries all the way up.
| boundary | you gain | destroyed, unrecoverably |
|---|---|---|
| wire → RX ring | addressable memory | timing, signal shape, inter-frame gaps |
| sk_buff → socket queue | a position in a stream | arrival order, retransmits, duplicates, RTT |
recv() | a byte stream | segment boundaries, packet identity, MSS, TTL |
| framing loop | a message | nothing — this one only adds |
| struct population | typed fields | raw header bytes and their order |
The framing row is the odd one out and it's worth noticing early. It's the only boundary in the stack that invents structure rather than shedding it. Everything below it is the kernel narrowing things down; everything at and above it is your process building something up.
The kernel never assembles a buffer
This surprised me when I went looking, and it's the fact that makes the rest click.
There is no contiguous byte stream anywhere in the kernel. The socket's receive queue is a linked list of sk_buffs — the same allocations the driver made, still holding their original packet-sized payloads. TCP's ordering work is queue placement, not memcpy: in-order segments get appended to sk_receive_queue, and out-of-order ones sit in a separate red-black tree until the gap ahead of them fills in, at which point they're spliced across.
So when recv() runs, it isn't handing you a buffer the kernel prepared. It walks that list, copies out as many bytes as you asked for, and if your request runs out mid-sk_buff it records an offset and leaves the remainder for next time.
Which means the byte stream is manufactured at the copy, and has never existed anywhere before that moment. The sender had message objects. The wire had frames. The kernel had a list of structs. The flat, boundary-free stream comes into being exactly once — in your process's memory, as a side effect of you asking for it.
That reframes the famous line. TCP is a byte stream isn't a description of what travels; it's a description of what recv() does.
The kernel does have one hedge. Under memory pressure it may collapse and coalesce queued sk_buffs to reclaim overhead. So "no contiguous buffer" is the normal case and the one to reason from, not an invariant to bet money on.
The same bytes, three ways
This is the thing the whole series exists to make obvious. Two HTTP requests, one connection, four TCP segments — and the boundaries refuse to line up.
Four reads off the socket. Segment 2 holds the tail of request A and the head of request B — the network had no idea a request boundary was in there.
Segment 2 is the one to sit with. It contains the tail of request A and the head of request B, and there is nothing anywhere in that segment marking the join. No flag, no length field, no delimiter the network understands. The \r\n\r\n that ends request A is meaningful only to a program that has already decided it's reading HTTP.
Toggle to the third view and the dividers vanish entirely. That's not a simplification — that's what phase 5 actually looks like from inside your process. Both of the other two views are annotations drawn on top of it: one reconstructed from information that no longer exists, one invented by a parser that hasn't run yet.
The phases are code, and you can go read it
Nothing above is a metaphor. Here is phase 5 through phase 7 in Go's standard library, with the context plumbing, TLS negotiation and HTTP/2 branches stripped out — the frames that actually touch a byte. Line numbers are go1.24.7 and will drift.
ListenAndServe(":8080", h)
├─ net.Listen("tcp", addr) → a file descriptor
└─ srv.Serve(ln)
└─ for { rw, err := l.Accept() server.go:3424 ← loop 1: connections
go c.serve(connCtx) } server.go:3454
└─ c.serve(ctx)
├─ c.tlsState = ... server.go:1991 ← once, before any request
├─ c.bufr = newBufioReader(c.r) server.go:2013 ← the buffer. phase 5 lives here.
└─ for { w, err := c.readRequest(ctx) server.go:2026 ← loop 2: requests
serverHandler{}.ServeHTTP(w, w.req) }
└─ readRequest(c.bufr) request.go:1079 ← phase 6
├─ tp.ReadLine() request.go:1087 → Method, RequestURI, Proto
├─ req.TLS = c.tlsState request.go:1086 → copied, never parsed
├─ tp.ReadMIMEHeader() request.go:1133 → Header
└─ readTransfer(req, b) request.go:1158 → ContentLength, Body
└─ t.Body = &body{src: io.LimitReader(r, realLength)}
transfer.go:573 ← phase 7
Two loops, nested. The outer one accepts connections; the inner one reads requests off a single connection. And the buffer is created between them, at server.go:2013 — which is the single most consequential line in the file. It's why a buffer outlives any individual request, why HTTP keep-alive works at all, and why leftover bytes from one request are sitting right there waiting to be misread as the next one.
That last sentence is not hypothetical, and watching it happen is where the next article starts.
Phase 7 reaches back into phase 5. Go builds the Request as soon as the headers are complete, so when your handler calls io.ReadAll(r.Body) you drop back down through the copy boundary and pull more bytes off the socket. The ladder isn't a pipeline you exit at the top — it's a stack you re-enter. Which also means r.Body can block, and can fail, in the middle of your handler.
Once, or twice
Almost nobody runs a Go process directly on port 443. There's an nginx, an ALB, an Envoy, something.
A reverse proxy is not a special networking object. It's a program that climbs the ladder, gets a message at phase 6, and then turns that message back into bytes and writes it to a second socket — where a second kernel re-segments it according to conn B's MSS.
Conn A's segment boundaries have no relationship whatsoever to conn B's. A request that arrived in nine dribbled pieces can leave as one write. The two halves of the picture are genuinely independent, which is exactly why X-Forwarded-For has to exist: it's a header whose entire job is carrying information that a boundary destroyed one hop earlier.
This isn't really about HTTP
Swap the protocol and phases 1 through 5 don't move at all. Only phase 6 changes, and it changes less than you'd expect — every protocol I can think of frames one of three ways:
- by delimiter — HTTP headers end at
\r\n\r\n, Redis lines end at\r\n, SMTP ends a body with a lone dot - by length prefix — Postgres sends
[type][length][payload], SSH sends a packet length up front, HTTP bodies useContent-Length - self-describing — chunked transfer encoding, HTTP/2 frames, TLS records, where the stream tells you how much more to read as you go
That's the whole family. Once you've seen the ladder, learning a new protocol is learning which of those three it picked and what its messages mean — not relearning networking.
Match each thing to the phase where it happens.
What the series is for
Four parts, of which two exist:
- This one — the phases and the boundaries
- Journey of a Packet: From Wire to User Space — phases 1 through 4, the NIC through the softirq
- The copy is where the packet dies — phases 5 through 7 in detail, the framing loop, and what it looks like when it goes wrong
- Now do it twice — proxies, re-serialization, and buffering you didn't ask for
The part I'm not sure about
The whole model above puts the interesting boundary at recv(). That's clean when the kernel owns the stack — but it isn't obviously true anymore once you leave that arrangement.
With AF_XDP or DPDK you take delivery at phase 2. You get frames, in a ring, and everything above that is now yours to build: your own IP handling, your own TCP state machine, your own decision about when — and whether — to flatten anything into a stream at all. The phases don't disappear. The line moves, and you end up on the other side of it.
So I'm not sure whether "the copy is where the packet dies" is a fact about networking or a fact about the sockets API. My instinct is the latter, which would mean the durable version of the claim is something weaker and more interesting: some boundary always destroys packet identity, because a byte stream is a lossy view and somebody has to choose when to take it. The kernel just usually chooses for you.
I don't have a good enough feel for the bypass path yet to know if that holds up. That's the one I'd like to be wrong about.
Comments
No comments yet. Be the first!