One Cached CORS Header Broke the Fonts

Fonts stopped loading on a production web app. Text fell back to the browser's default font and the browser console filled with CORS errors. It lasted 1 hour and 44 minutes.

The strange part was that it did not break for everyone. Pages on one subdomain worked fine. Pages on the others did not. And which subdomain worked depended on which one had loaded the font first.

Why fonts care about CORS

Browsers treat web fonts differently from images. If a page loads a font from another origin, such as a CDN domain, the browser makes a CORS request, and it only uses the font if the response carries an Access-Control-Allow-Origin header that matches the page's origin.

So the response for the same font file has to change depending on who asks. A page on one subdomain needs that subdomain in the header. A page on another subdomain needs its own.

S3 handles this correctly on its own. Give it a CORS configuration that allows several origins, send it a request with an Origin header, and it answers with that origin in Access-Control-Allow-Origin.

The cache key was missing one header

The CDN in front of the bucket, CloudFront in this case, forwarded the Origin header to S3. So S3 saw the real origin and answered correctly.

But Origin was not part of the cache key. The CDN stored the first response for each font and served it to everyone who asked afterwards, whatever their origin.

So the first subdomain to request a font after it entered the cache "owned" that cached copy. Its origin was baked into the header. Every other subdomain got a response that named someone else, and their browsers refused to use the font.

That is why it looked random. It depended on which page happened to warm each object.

The first fix did not work

The obvious move was to fix the CORS response and clear the cache. I made the change and expected the next requests to come back right.

They did not. CloudFront did not overwrite a CORS header that was already in its cached copy. The objects still in cache kept serving the old, wrong header. From the outside it looked like the fix had done nothing, which cost time and confidence in the middle of an incident.

The lesson I took: with a CDN, a fix to the origin and a fix to what users receive are two different fixes. Until the cached copies are gone, users still get the old answer.

SEV1 or SEV2

There was also a judgement call during the incident. Broken fonts on a production product look bad, and it was customer-facing. But the product still worked, and there was a workaround.

I called it a SEV2: a real, customer-visible problem with a workaround, not a full outage. That kept the response proportionate: immediate attention and a postmortem, without treating it like a full outage.

Getting severity right matters more than it seems. If every visible bug is a SEV1, the label stops meaning anything, and the next real outage competes with a font.

How to avoid it

The underlying rule is simple: if the origin varies its response by a request header, that header has to be in the cache key, or the cache will serve one requester's answer to everyone.

For CORS behind a CDN, that means one of these:

• Include Origin in the cache key, so each origin gets its own cached copy.
• Or let the CDN set the CORS headers itself with a response header policy, so the cached object does not carry a per-origin header at all.
• And either way, after changing it, invalidate the affected objects so the old copies are gone.

The first option is the most direct. It multiplies cached copies by the number of origins, which for a handful of fonts is nothing.

It is also worth checking the opposite problem. Forwarding a header to the origin without caching on it is almost always a mistake. Either the origin needs it, in which case the response probably depends on it, or it does not, in which case there is no reason to forward it.

How to test for it

This bug is easy to miss in testing because one origin always works. To catch it:

• Request the same asset with two different Origin headers, one after the other, and compare the Access-Control-Allow-Origin you get back.
• Do it after a cache miss and after a cache hit.
• If the second response names the first origin, you have this bug.

A two-line script with curl -H "Origin: ..." against a test object does it. It takes less time than reading this section.

A short checklist

• For every header your CDN forwards to the origin, ask whether it should be in the cache key.
• Test CORS assets with at least two different origins, through the CDN, after a cache hit.
• After any header or cache behaviour change, invalidate the affected objects before calling it fixed.
• Set incident severity by customer impact and workarounds, not by how visible the bug looks.

If your CDN behaves differently depending on who asked first, I can help you untangle it. See AWS Architecture & Migration.