The Next.js caching bug that can serve Googlebot a blob of JavaScript instead of your page

We found this on our own site, which is the only reason we can describe it precisely. A routine crawl reported that six pages had no title tag, no H1 and no viewport meta. Those pages demonstrably had all three. The crawler was not wrong — it had genuinely been served pages without them.

What Next.js is actually doing

In the App Router, every <Link> in the viewport prefetches its destination in the background so that clicking it feels instant. That prefetch is not a request for HTML. It asks for a React Server Component payload — a serialized description of the page the client router can apply directly.

Those requests are distinguished from ordinary navigations by request headers, and the response differs accordingly.

This is correct behaviour and nothing is wrong yet. The problem appears one layer out.

Where it breaks

A CDN decides what counts as "the same request" using a cache key. If the RSC header is not part of that key, then both requests in the table above are, as far as the CDN is concerned, identical. One cache entry serves both.

1. 01

   A visitor lands on your homepage

   Next.js prefetches the links in view. Those prefetch requests reach the CDN carrying RSC: 1.

2. 02

   The CDN has no entry for that URL yet

   It forwards to the origin, which correctly returns text/x-component because the request asked for it.

3. 03

   The CDN caches that response

   Under a cache key that ignores the header that made the response different.

4. 04

   Googlebot requests the same URL

   No RSC header, so it should get HTML. It gets the cached component payload: no <title>, no <h1>, no readable content.

The page is perfectly built, perfectly rendered, and perfectly indexable. It just was not the thing that got sent.

Why it is so easy to miss

When we first tried to reproduce it with curl, everything came back text/html. Every URL, repeatedly. It looked like the crawler had malfunctioned.

It had not. The poisoned entries were populated by the crawl itself — a real browser, following real links, firing real prefetches — and had since expired or been evicted. A tool that fetches URLs without executing JavaScript never generates the prefetch that causes the problem, so it never sees it.

• It only appears after a real browser has prefetched that URL
• It affects whoever requests the page next, then heals on its own
• It is invisible in your browser, because your browser is the thing causing it
• Different CDN edge locations hold different entries, so it varies by geography

How to check your own site in two minutes

Send a prefetch-shaped request, then immediately send a normal one to the same URL. If the second response is not HTML, you have this bug.

U="https://example.com/some-page?cb=$RANDOM"

# 1. Imitate a Next.js prefetch
curl -sI -H 'RSC: 1' -H 'Next-Router-Prefetch: 1' "$U" | grep -i content-type

# 2. Now request it the way a visitor or Googlebot would
curl -sI "$U" | grep -i content-type

The first response should be text/x-component. The second must be text/html. If the second one also returns text/x-component, the cache is serving the prefetch to everybody.

The fix

Add the router headers to the cache key so the two response types are stored separately. On CloudFront, the managed CachingOptimized policy is the usual culprit — it is the sensible default for static sites and its HeaderBehavior is none, meaning no request header influences the cache key at all.

{
  "Name": "nextjs-rsc",
  "DefaultTTL": 86400,
  "MaxTTL": 31536000,
  "MinTTL": 0,
  "ParametersInCacheKeyAndForwardedToOrigin": {
    "EnableAcceptEncodingGzip": true,
    "EnableAcceptEncodingBrotli": true,
    "HeadersConfig": {
      "HeaderBehavior": "whitelist",
      "Headers": { "Quantity": 2, "Items": ["RSC", "Next-Router-Prefetch"] }
    },
    "CookiesConfig": { "CookieBehavior": "none" },
    "QueryStringsConfig": { "QueryStringBehavior": "none" }
  }
}

Both headers matter. RSC separates component payloads from HTML; Next-Router-Prefetch separates a prefetch from a full navigation payload, which are also not the same response. Two headers means at most four variants per URL — a negligible cost against serving Googlebot an empty page.

The principle is identical on any CDN. On Cloudflare it is a Cache Rule with a custom cache key including those headers; on Fastly it is a vary on them. What matters is that no CDN in front of an App Router application may treat the RSC header as irrelevant.

The related `_rsc` question

The other half of this problem shows up as URLs like /services?_rsc=a1b2c3 appearing in Search Console. Next.js appends that parameter to prefetch requests, and those URLs can end up discovered and crawled.

They are far less dangerous than the cache issue, and the instinct to block them in robots.txt is worth resisting. A blocked URL cannot be crawled, so Google never sees the canonical tag that would consolidate it — you trade a tidy report for an unresolved duplicate. Two things handle it properly:

• A self-referencing canonical on every page, pointing at the clean URL without the parameter. Google consolidates the variant into the canonical on its own.
• A cache key that ignores query strings, so ?_rsc= requests without the header are served the same HTML as the clean URL rather than a payload.

What we would take from this

The uncomfortable part of this bug is not the fix, which is four lines of configuration. It is that our own site was serving component payloads to crawlers while every dashboard said it was healthy — indexed, ranking, all audits passing. We only found it because a crawler with a real browser engine disagreed with a curl request, and we chose to investigate rather than assume the tool was broken.

• Crawl your own site with something that executes JavaScript, not only with a fetcher
• When a tool reports something you know is false, find out why before dismissing it
• Treat any CDN in front of a rendering framework as part of the SEO surface, not as infrastructure someone else owns

If you run Next.js behind a CDN, the two-minute test above is worth doing today. It costs nothing, and the failure it detects is one that no amount of good content or clean markup will compensate for.