Skip to main content

www Scanner: Design Notes

Working notes for the scanner behind Does www Still Work?.

This is a learning exercise, not a spec. The point is to pick a goal for a session, run the thing, and see what it does. Nothing here is committed to. Ideas worth trying and things worth watching for — expect it to be wrong in places, and revise it from what actually happens rather than defending it.

Infrastructure isn't a constraint either; this can move off the laptop whenever that's the interesting variable.

The goals are separate tasks​

These share a core and nothing else. Conflating them is the main trap.

Census. Keep an up-to-date, complete record of the state of www across a domain corpus. Correctness and completeness matter; speed is secondary. Full per-domain records land on disk. This is a deliverable in its own right, not instrumentation for the benchmark.

Throughput. How many domains/sec, and how long for the whole corpus. Runs with a null sink so the storage path can't contaminate the measurement. Push workers 10 → 100 → 1,000 → 10,000 and find what gives out.

Because they never run at the same time, storage design serves the census and owes the benchmark nothing. There is no completeness-vs-speed tradeoff to make.

Likely one article per goal rather than a fixed 2- or 3-part plan — each goal has its own bottleneck and its own story.

Corpus​

Upstream of everything and the genuinely hard part: DNS is not enumerable.

ICANN CZDS zone files — the real answer for whole-internet scale. Per-TLD application and approval, then daily downloads. .com is ~160M. Note these list delegated domains (ones with NS records), not all registered ones — arguably a better denominator here, since a domain with no nameservers cannot serve anything. Not all ccTLDs participate.

Certificate Transparency logs — every publicly-trusted cert with its SAN list. A cert covering www.example.com is direct evidence someone provisioned it; an apex-only SAN is evidence they didn't. Answers part of the question with zero DNS queries. Biased toward names someone bothered to certify, but the bias is legible and may be interesting on its own.

Tranco — research-grade top-1M, built to resist manipulation. For the representative-sample run where the percentage needs to be defensible.

Two runs, not one​

The www percentage wants a curated unbiased sample with a defensible denominator. The throughput run wants raw volume and doesn't care what's in it. Same code path, different inputs — the input abstraction should not assume either.

Prefetch, decided​

Materialize the corpus first, as a versioned immutable artifact with its own refresh cadence. Acquisition is bulk, sequential, bandwidth-bound; scanning is per-domain, latency-bound, concurrent. Fused, neither can restart without redoing the other and the two bottlenecks alias in every measurement.

Storage​

Corpus: sorted, immutable, versioned by date. One file.

Results: fixed-width records written positionally — the Nth record belongs to the Nth corpus entry, so the domain string is never stored in the results. Enums and small ints only: resolve status, record type, TLS outcome, HTTP status, redirect terminal. ~16-24 bytes/domain, ~20GB at a billion, no text, no index.

Consequences worth having:

  • No write coordination. Each worker writes at i * width in an mmap'd file. Disjoint offsets, no lock, no writer goroutine. Feeding workers in corpus order keeps completions roughly sequential, so this doesn't thrash page cache.
  • Resumption is the zero value. Status 0 means not attempted; the file is its own checkpoint. Nothing separate to keep consistent.
  • Run diffs are a merge join. Two (corpus, results) pairs streamed in lockstep, emitting deltas. This is how the census becomes a trend series without querying a billion rows.

SQLite: aggregates, run metadata, and the interesting individual cases worth looking at by hand. Small enough to query interactively, which the blob never will be. Batched writes, WAL.

Hard constraint throughout: this runs on a laptop. Bytes-per-domain gets multiplied by a billion and shares a disk with everything else.

Methodology: don't poison the trend​

The report is a series, so run-to-run comparability is a correctness property.

A domain that NXDOMAINs on run 1 might be a lapsed registration (permanent) or a resolver hiccup (transient), and one run cannot distinguish them. Prune aggressively and run 2 measures a different population — the trend line becomes survivorship.

Keep dead entries with last_seen_alive and a consecutive-failure count. Prune only after N strikes, and record what was pruned so the denominator stays reconstructible. Cheap now, expensive to retrofit once run 1 exists.

Negative caching contaminates results​

A NXDOMAIN answer is not evidence the name is absent — it may be a cached negative from before someone fixed it. The SOA's last field sets how long resolvers may cache that absence, and it is routinely a full day. csv-helper.com runs the Route 53 default:

$ dig @ns-214.awsdns-26.com +short csv-helper.com SOA
ns-214.awsdns-26.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
                                                                      ^^^^^ 24h

So a domain that added www this morning can keep reading NXDOMAIN through a caching resolver until tomorrow. Directionally this biases the headline number down — the scan will under-report www support, never over-report it. Worth deciding whether to query authoritative nameservers for the final classification pass, which removes the bias at the cost of hitting the delegation chain for every domain.

Ground truth: my own domains as fixtures​

A classifier for the open web has no test set. But I own four domains and control their DNS, so each can be parked in a known state and used to assert the classifier lands them in the right bucket.

csv-helper.com is the first one, deliberately left in the both-serve-200, no-canonical state — apex and www both return an identical 200 with no <link rel="canonical"> anywhere. That is the third branch of the canonical-direction taxonomy and the least obvious one to get right, since nothing is failing. Add a 301 or a canonical tag later, re-run, and confirm the classification moves.

Branches still needing a fixture, spread across the remaining domains:

  • www → apex 301
  • apex → www 301
  • www NXDOMAIN (the state all four were in to begin with)
  • TLS handshake succeeds but the cert has no SAN for the name presented
  • wildcard *.domain making an unserved www resolve anyway

Caveat that follows from the section above: flipping a fixture and immediately re-running tests the resolver's cache, not the fixture. Verify against authoritative nameservers, or give the test zone a short SOA minimum so fixtures can be cycled quickly.

What to classify​

Per domain, for both apex and www:

  • resolves? NXDOMAIN vs NOERROR-no-data vs answer
  • record type (A / AAAA / CNAME)
  • TCP connect
  • TLS handshake — including cert name mismatch, which is its own failure class
  • HTTP status
  • redirect chain terminal host

Comparing the two terminal hosts gives canonical direction (www→apex, apex→www, or both 200 with no canonical form). Comparing failures gives a taxonomy.

Two things the naive "does www resolve" boolean gets wrong:

  • False positives from wildcards. *.example.com makes every label resolve, including www, without necessarily serving anything. Counting these as support inflates the number. NXDOMAIN must be distinguished from resolves-and-does-nothing.
  • False negatives from collapsing the ladder. Resolves-but-no-TCP, TLS-fails-on-SAN, and 404-because-the-vhost-only-knows-the-apex are three different mistakes by three different people. Flattening them to "no" throws away the finding.

No browser anywhere in the pipeline. Omnibox fixups add and strip www, the HSTS preload list rewrites requests before DNS, and one navigation emits a pile of speculative rows. Raw resolver, raw HTTP client, redirects reported rather than followed, timeouts described as timeouts.

Bottlenecks: what to aim at​

Saturating the NIC is probably the wrong target. DNS is tiny — ~100 byte queries, few-hundred byte responses. The bytes here are almost entirely TLS cert chains, a few KB per handshake. At 1,000 domains/sec with TLS on both names that's ~100-150 Mbps; you'd need ~10k/sec to threaten a gigabit link.

What actually gives out first, in rough order of suspicion: file descriptors, ephemeral port exhaustion, conntrack table (including the household router's, which will take the whole house's internet with it), resolver rate limits, connection pool behavior when most connections fail, memory held by in-flight response buffers.

Aim at connection concurrency and packets-per-second, not throughput.

Null sink behind an interface so the network ceiling and the storage ceiling can be measured independently. Go profiler in play.

When it plateaus and nothing looks busy​

The interesting case: CPU idle, memory flat, bandwidth nothing, and throughput still won't move. Frame it with Little's Law.

throughput = in-flight concurrency / mean completion time

If throughput is stuck and the machine is bored, one of those two terms is not what you think it is. Either something is serializing so real in-flight is far below the worker count, or mean completion time is much worse than the happy path suggests.

Tail latency is the likely first wall. Happy path is ~300ms, but if 5% of domains hang to a 10s timeout, the mean is 0.95×0.3 + 0.05×10 = 0.79s — nearly triple. Dead hosts are exactly what a www scan is full of. The lever there is the timeout value and failing fast on hopeless names, not more workers.

Things that serialize in-flight, roughly by likelihood:

  • Resolver. GODEBUG=netdns=2 prints which one you got. The cgo path blocks an OS thread per getaddrinfo. The pure-Go path inherits /etc/resolv.conf, typically 2 attempts × 5s — a dead upstream costs 10s before you learn anything.
  • conntrack. If Docker is installed, netfilter is loaded; nf_conntrack_max is often 262144. A full table drops packets silently, which reaches the code as timeouts and masquerades as the tail-latency problem. dmesg shows table full, dropping packet; /proc/sys/net/netfilter/nf_conntrack_count warns first.
  • http.Transport defaults. MaxIdleConnsPerHost 2, MaxIdleConns 100. Milder here than in a normal load test since each host is hit once, but the churn is real.

Ephemeral port exhaustion is probably not it, despite being the reflex answer. The 4-tuple includes the destination, and destinations vary constantly here, so the port space isn't shared across targets the way it is when hammering one host.

Instrumentation consequence: logging only domains/sec makes all of these look identical — "stuck." Emit in-flight count and a completion-time histogram from the start. Little's Law then says which term is lying, and the search becomes a measurement.

Open questions​

  • Which TLDs to apply for on CZDS, and whether run 1 is .com-only.
  • Whether CT logs answer enough of the question to be worth a separate pass.
  • Resolver strategy: system resolver, own recursive, or a pool of public ones — and what each does to rate limiting and to result bias.
  • Whether IPs are worth storing (variable-width, breaks the fixed-record scheme) or whether the classification alone is enough.
  • Reservoir sample of interesting failures for the SQLite side — sample size, and what qualifies as interesting.
  • Refresh cadence for the report. Quarterly is probably right for a trend.

Comments

No comments yet. Be the first!