Nothing in AWS Tells You www Is Missing

The first one took one DNS record. The certificate already covered www, CloudFront already accepted it, and the only thing missing was the record pointing the name at the distribution.

I assumed that was luck — that past me had gotten most of the way through one setup and stalled at the last step. So I went through the other four expecting to find real work: a certificate to re-issue here, an alternate domain name to add there.

There wasn't any. Four for four, the fix was DNS and nothing else.

The sweep

apresapply.com, unpivot-sql.com, thomasrones.com, fxf.social. Every one of them had the same shape as csv-helper.com:

• The CloudFront distribution already listed www.<domain> in its alternate domain names.
• The distribution already had IPv6 enabled.
• The certificate already covered www.

And on every one, www returned NXDOMAIN.

The certificates are the part that convinced me this wasn't carelessness at the last step. They were issued months earlier — April, May, July — and two of them aren't even www-specific:

$ for d in csv-helper.com apresapply.com unpivot-sql.com thomasrones.com fxf.social; do
    openssl s_client -connect $d:443 -servername $d </dev/null 2>/dev/null \
      | openssl x509 -noout -ext subjectAltName | tail -1
  done

csv-helper.com,  www.csv-helper.com
apresapply.com,  *.apresapply.com
unpivot-sql.com, www.unpivot-sql.com
thomasrones.com, www.thomasrones.com
fxf.social,      *.fxf.social

apresapply.com and fxf.social have wildcard certificates. There was never a decision to make about www on those two — a wildcard covers it automatically, along with every other label. The TLS layer was ready for a hostname that did not exist, and had been for months.

So the certificate was never the blocker on any of the five. Neither was CloudFront. The entire outage, across five domains and an unknown number of months, was a missing row in a DNS zone.

What varied was which record was missing

The one thing that wasn't uniform is the interesting one.

fxf.social had an AAAA alias on the apex but no www record of any kind. The others had an A alias on the apex and nothing else — no apex AAAA, no www in either family. So the gaps didn't line up.

If these zones had come out of a template, the same records would be present and the same ones absent every time. They don't, which means each was typed by hand, on a different day, stopping wherever it looked finished. That's the actual root cause: not a wrong decision repeated five times, but five separate sessions of manual data entry with no definition of done.

Why nothing reported it

This is the part worth keeping, because it generalises past www and past AWS.

Three services were involved and each did its job correctly:

• ACM issued a certificate covering www.<domain>. It has no way to know whether anything resolves there. A certificate is a statement about names you may serve, not names you do serve, so an unused SAN isn't an error — it's the normal state of a wildcard.
• CloudFront accepted www.<domain> as an alternate domain name. It validates that your certificate covers the name, because that's a constraint it can check locally. It does not check DNS. A distribution configured for a hostname nobody points at is perfectly valid — that's exactly what a migration looks like halfway through.
• Route 53 served the records in the zone. It has no opinion about what CloudFront expects, because it's a DNS server, not an integration test.

Nobody is wrong. There's just no service whose job is the join, and the failure only exists in the join.

Monitoring doesn't catch it either, and the reason is worth stating plainly: you cannot alert on a hostname you never configured. A health check watches an endpoint you told it about. There's no error rate, no elevated latency, no expiring certificate, no failed deploy. A name that doesn't exist produces no telemetry at all, because nothing is running there to produce any.

The only signal is a person typing the habitual www. and quietly deciding you're down.

And the check that finds it is embarrassingly cheap:

$ dig +short www.example.com

One command per domain, and I'd never run it — because there was nothing prompting me to. Every dashboard I own was green, and every one of them was right.

The one I skipped

talktop.us already had www, and it's the only one of the six not behind CloudFront — it resolves to a single origin IP rather than a CDN edge. Different hosting, different shape, different failure mode. It's missing IPv6, which I've left alone because the whole thing is getting redeployed shortly and fixing infrastructure you're about to replace is just a slower way of deleting it.

Worth noting as the counter-example, though. The five that shared the bug also shared a setup process. The one that didn't share the process didn't have the bug.

What I broke by fixing it

Being honest about the state I've actually left things in:

$ for h in ... ; do curl -sSo /dev/null -w '%{http_code} %{redirect_url}\n' https://$h/; done
200
200      # www

All five domains, both hostnames, 200 with no redirect and no <link rel="canonical">.

That's the third case in the taxonomy I set out to measure — apex and www both serving identical content with nothing declaring which is real. Before this week, four of those five had an unambiguous canonical form for the stupidest possible reason: only one of the two names existed. I fixed the outage and created a duplicate-content problem in the same motion.

It's a much smaller problem than the one it replaced, and it's the right order to fix them in — a name that doesn't resolve fails for everyone, while duplicate content is a ranking-signal inefficiency. But it's the kind of thing that stays "temporary" for two years if you don't write it down, so: it's written down.

One of them is staying broken on purpose. csv-helper.com is a fixture for the scanner this all leads to, parked in the both-serve-200 state so I can assert the classifier recognises the case where nothing is failing. The other four just need a redirect.

The general version

Everything above is one instance of a shape worth recognising: a system composed of services that are each individually correct, failing in the gaps between them, silently, because no component can see the whole.

The fix is never the hard part. The hard part is that nothing tells you.