You built that page for a browser. A browser renders the markup and throws the rest away. An agent has to read the markup, and every tag and script costs it tokens it could have spent understanding the content. The same page as clean Markdown can run up to 97% smaller. That gap is the difference between an agent quoting your client accurately and an agent guessing.

I have shipped this layer on production Sanity builds, on Next.js and on Astro, and the surprise every time is how little of it is hard once the boundary is drawn right. Two features cover it. An editor serves a clean /llms.txt index that an agent reads to discover what the site contains, drafted by AI from the site's own content in one click. Any public page returns a token-light Markdown version of itself on its own URL when an agent asks. Discovery, then retrieval, and it is about to be the part clients ask for by name.

Why agents read your site differently

A browser and an agent want opposite things from the same URL. The browser wants HTML it can paint: layout, fonts, interactivity, the whole rendered document. The agent wants the content and nothing else, because everything else is tokens spent on noise it has to parse before it reaches a single useful sentence.

Two problems sit between an agent and your content. Discovery: before an agent reads a page, it has to know the page exists, and a large site does not fit in a context window for it to crawl around. Retrieval: once it knows a page exists, reading it as HTML is expensive and noisy. Solve one without the other and the agent either cannot find your pages or chokes on them when it does.

The fix is a layered one, and it is not new thinking. Sanity's own field guide on serving content to agents frames the exact model: an llms.txt index for discovery, plus per-page Markdown through content negotiation for retrieval. Vercel's write-up on agent-friendly pages lands on the same approach. The honest caveat: the research does not settle whether any of this lifts AI citations. Profound ran a randomized test across 381 pages and found no significant change in bot traffic from Markdown. This is early and experimental, and nobody can promise you rankings from it. The case for doing it anyway is that it costs almost nothing. Agents are already fetching your pages, the work is a toggle and a serializer, and clean content is the right thing to serve whether or not the citation upside ever materializes. Low cost, real upside if the bet pays, no downside if it does not.

Discovery: an llms.txt an editor owns

An /llms.txt is a single Markdown file at a known URL that hands an agent a concise map of your site with links to its pages. It follows the llmstxt.org convention that Jeremy Howard proposed in September 2024, now adopted by sites like Next.js, Cloudflare, and Hugging Face, with Anthropic publishing its own as a lightweight sitemap. It is an index of links with short descriptions, not a dump of your content. The whole file is tiny, often under a thousand tokens. Its job is to orient an agent at the moment its context window cannot hold the whole site.

A served file reads like this:

# Index

> Freelance creative frontend engineer based in Vienna, working
> worldwide on animation-heavy, interaction-rich sites on headless
> CMS stacks (Next.js, TypeScript, Sanity, Tailwind, Motion,
> Vercel). Shipped work for Buck, Disney, Porsche, Red Bull, and
> Getty. Awwwards juror and Codrops contributor.

## Core pages

- [Index](https://www.edoardolunardi.dev/): Role, stack, selected
  work, and the primary way to get in touch.
- [Works](https://www.edoardolunardi.dev/works): Portfolio index of
  production client projects.
- [Lab](https://www.edoardolunardi.dev/lab): Interaction experiments
  in motion, scroll, and novel UI.
- [Blog](https://www.edoardolunardi.dev/blog): Writing, including
  deep dives on content architecture.

## Articles

- [The Content Architecture I: CMS Structure](https://www.edoardolunardi.dev/blog/the-content-architecture-cms-structure):
  How to structure a CMS for long-term maintainability.
- [The Content Architecture II: Content Models](https://www.edoardolunardi.dev/blog/the-content-architecture-content-models):
  Designing schemas that match real content.
- [The Content Architecture III: Page Composition](https://www.edoardolunardi.dev/blog/the-content-architecture-page-composition):
  Assembling pages from reusable sections.
- [The Content Architecture IV: Content Primitives](https://www.edoardolunardi.dev/blog/the-content-architecture-content-primitives):
  Low-level building blocks for flexible pages.

## Optional

- For a quick hiring decision: Index, Works, a recent case study,
  and the Content Architecture series. To start a project, use the
  contact path on the Index page.

Most guides to building this file land in one of two places, and both have a cost. You hardcode a static file in public/, and it drifts the moment anyone publishes, because nothing updates it. Or you write a dynamic route handler that queries your CMS, which stays accurate but lives in code, so every change to what the file emphasizes means a developer and a deploy. An editor who wants to reword the summary or reorder which pages matter has to file a ticket.

The version I ship moves the file into the CMS. The content lives in a field on the Site document, so an editor owns it and edits it like any other field. It carries a Generate button. One click drafts the entire file from the site's own content using Sanity Agent Actions, Sanity's AI capability for generating content from a prompt plus structured inputs. The editor reviews the draft, edits anything, and publishes. No deploy.

The generation is grounded, which is what makes it trustworthy. The route reads an inventory of every indexable page from the CMS, the same visibility rules the sitemap uses, so nothing private leaks in. It builds one entry per page with the real title and a URL constructed in code, then passes that to the model as structured data. The model writes the descriptions but never touches a URL, because the URLs never pass through it. An agent gets links that resolve, every time, with descriptions an editor approved.

There is a deliberate split underneath. Generation reads your content with drafts overlaid, so the draft reflects work in progress. Serving only ever returns the published field. A draft never appears at /llms.txt. And the whole file is gated by a single toggle: turn Serve /llms.txt off and the URL returns 404 without deleting a word of the content.

In the Studio it is one tab and two switches. An editor opens the Site document, clicks Generate, reviews the draft, and flips the toggle to serve it. Every page carries the same Agents tab: Generate its Markdown, store it, serve it with a switch. The next section is that side.

Retrieval: Markdown on the same URL

When an agent fetches a page and its request says it prefers Markdown, the server returns the page's stored Markdown instead of the full HTML document, on the same URL. An editor generated that Markdown once and published it, so serving is a read, not a build. A browser, which asks for HTML, still gets the HTML page. Same address, two representations, decided by who is asking.

The signal is the HTTP Accept header. A browser sends Accept: text/html,... and never lists Markdown, so it always gets HTML. An agent that sends Accept: text/markdown gets the Markdown. This is content negotiation, a part of HTTP that has existed for decades, used here to serve agents without giving them a separate set of URLs to discover. The /llms.txt index links your normal page URLs, and those same URLs answer in Markdown when an agent asks. The two features lock together with no extra wiring.

Every page now carries its own Agents tab, mirroring the Site document's: a Generate button that drafts the page's Markdown, a field that stores it, and a Serve Markdown to agents toggle, on by default. Same generate, review, publish loop as the llms.txt file. Turn it off and that page stays HTML-only, decided per page without touching code. This is the control a bare llms.txt cannot give you: one file is all-or-nothing, with no way to hold a page back. A per-page switch is per-page governance.

The property that makes this safe to ship is what it costs normal traffic. Nothing. A browser's Accept header never contains "markdown," so the check that detects an agent fails on the first character and the request continues down the normal HTML path with no added work. Every reader who is not an agent pays nothing. Only the rare agent request does the eligibility read, and that read hits a short-lived hot cache, so even agents barely touch the database. A webhook busts the serve route's own fetch when content changes.

The part that took three tries

Turning a CMS page into Markdown sounds like a formatting problem. It is an architecture problem, and the difference is whether the feature survives the next section type someone adds.

The naive version is one branch of code per section type. It works until someone builds a new section, at which point it silently renders nothing, and nobody notices until an agent reads an empty page. Every section type is a new place for the serializer to fall behind the schema.

The version that holds reads by convention. The schema is built from reusable field factories that always emit the same field names: a rich-text factory produces appRichText. A media factory always produces appMedia. A link factory always produces appLink. Those names are guaranteed on any section built from the factories, so the serializer reads them wherever they appear and renders each to Markdown, with no per-section branching. A new section still serializes with no new code. Because the output is stored, an editor regenerates the page to fold it in, then publishes.

That is the line most DIY versions never reach. They couple the converter to a fixed list of section types, and it rots as the schema grows. Anchoring to factory-guaranteed names handles the common case for free, leaving the rare project-specific field as a single labeled extension. The honest ceiling: the query layer cannot be generated from a config object, because the type generator needs static query strings. So the mapping is one small, well-marked edit point rather than a pretense that the coupling vanishes. That is the difference between a feature you ship and a demo that breaks when the schema changes.

It looks inconsistent that llms.txt is AI-drafted and this Markdown is not. It is deliberate. The index is an editorial job, grouping pages and writing descriptions, so a model drafts it. Per-page Markdown is the opposite: content negotiation promises the agent the same page, faithfully, so its Generate button runs the deterministic serializer, not Agent Actions. Ask an LLM to convert a page and it paraphrases a heading, drops a list item, or reorders a section, unpredictably. The serializer reads the structured content and emits it one to one, identical every run. Same button, same stored field, two engines: faithful retrieval wants a serializer, an editorial index wants a model.

How the two features combine

An agent finds your client's site through llms.txt. It reads the index, a map of real page URLs with descriptions an editor approved. It picks the page it needs and fetches that same URL with Accept: text/markdown. The page answers in lean Markdown, the body and nothing else, internal links resolved to absolute URLs so the agent can follow them deeper. The same URLs serve both halves, no special infrastructure between them. For agents that read the index but do not negotiate Markdown on their own, one line in the generation notes that every page is also available through the Accept header.

This is what AI-ready means underneath the marketing word. Not a chatbot bolted onto the site, but a clean map and clean content on the URLs agents already have. When a server answers with Content-Type: text/markdown, Claude Code skips its summarization step and feeds your content to the model verbatim, a fast path you earn by serving structured Markdown instead of letting the agent reverse-engineer it from HTML.

Why this is a competitive line now, not later

Every site your clients run is already being read by agents, and most are serving them the same token-heavy HTML they serve a browser. The agent spends most of its budget on noise, and the answer it gives about your client is worse for it. That cost is invisible right now because nobody is measuring which sites agents cite well. That will not stay invisible.

Serving content to agents is becoming part of the brief the way responsive design and Core Web Vitals did. The studios that treat it as a first-class layer, owned by editors, drafted by AI, correct after launch, will quote it as a differentiator while everyone else is still pasting a static file into public/ and watching it drift. The work is not large. The boundary is what is hard, and the boundary is what most implementations get wrong.

This is the kind of layer that gets rebuilt from scratch on every project because it never makes it into an estimate. In The Content Architecture, it is already decided and already shipped: the llms.txt generator wired to Agent Actions, the per-page Markdown toggle, the factory-guaranteed serializer, the proxy that serves agents without taxing browsers. A client build starts with the agentic layer done, past the part that usually costs the first few days and gets cut for time anyway.

The Next.js edition and the Astro edition both ship it. The kit is available now, one payment, lifetime updates.

Common questions

What is llms.txt and do I need it?

An llms.txt is a Markdown file at the root of your site that gives AI agents a concise, linked index of your pages, so they can find your content without crawling the whole site. It follows a convention Jeremy Howard proposed in September 2024, now used by sites like Next.js, Cloudflare, and Hugging Face. Whether it lifts AI citations is still unproven and experimental, but it costs almost nothing to add, so the practical answer is that it is worth doing as a low-cost discovery layer while the evidence catches up.

How do I generate llms.txt from a CMS instead of hardcoding it?

Query your CMS for every indexable page, build the entries in code with real URLs, and serve the result from a route handler so the file reflects published content automatically. A static file in public/ drifts the moment anyone publishes. The stronger version stores the file in the CMS as an editor-owned field and drafts it with an AI action, so editors reword and reorder it without a deploy while the URLs stay grounded in code.

How do I serve Markdown to AI agents from a Next.js or Astro site?

Detect the Accept: text/markdown header on incoming requests and return a Markdown version of the page on the same URL, while browsers asking for text/html get the normal page. This is HTTP content negotiation, and the shape is the same in both frameworks. Put the detection in the proxy so it can short-circuit browser traffic for free and consult per-page state, like whether the page is private or toggled off, before serving.

Two caches, not one

Between a published document in Sanity and the HTML a browser receives, your content passes through two independent caches. The Sanity CDN caches API responses at the edge. The Next.js Data Cache stores the result of every server fetch. They answer to different controls, expire on different triggers, and neither one knows the other exists. A page serves stale content when either cache holds an old copy, so you can fix one and still ship the wrong price because the other is lying.

• The Sanity CDN, controlled by the useCdn flag on your client. With it on, Sanity serves a cached response from the edge: fast, and a few seconds to a few minutes behind the latest publish.
• The Next.js Data Cache, controlled by the revalidate and tags options you pass to a fetch. It persists across requests and, on Vercel, survives redeploys.

Once you know there are two, debugging stops being guesswork. You ask which cache is stale, not whether the code is broken. The pairing is not specific to Next.js either. Swap the framework and the Sanity CDN stays exactly where it was while the second cache changes shape, which is what the Astro section below works through.

Find which cache is serving the old copy

Before changing any code, make Next.js tell you what it is doing. Set logging.fetches.fullUrl to true in next.config.ts and every server fetch prints to the terminal with a cache HIT or MISS and the tags applied. A HIT where you expected fresh data means the Next.js Data Cache is holding the old copy and your revalidation never ran. A MISS that still returns the old value means the request read Sanity through the CDN and got a stale edge copy, the case bypassing the CDN in production removes. One line of config turns a guessing game into a reading.

When the terminal shows fresh data but the browser shows stale, the webhook is the suspect. A publish should fire a POST at your revalidation route within a second or two. If nothing arrives, the GROQ-powered webhook is misconfigured or filtered too tightly, and no cache tuning helps until it fires.

On Astro the reading comes off the response instead of the terminal. In production Vercel reports x-vercel-cache: HIT | MISS | STALE, and the dev provider sends X-Astro-Cache. A HIT on a route you just published means the tag was never busted, which points at the webhook rather than the client.

The Sanity CDN serves the first stale copy

Set useCdn: true and Sanity returns content from its edge cache. For a marketing page that updates a few times a week, that delay is invisible and the bandwidth savings are real. The trouble starts when you wire Next.js revalidation on top of it. Next.js revalidates, calls Sanity for fresh data, and Sanity hands back the same cached response it had a moment ago. Your revalidation worked. It refilled the Next.js cache with stale data from the CDN.

The fix in production is to bypass the CDN. Set useCdn: false so a production read hits Sanity's live API and returns the current content, with no edge copy to fall behind. The cost worry, that the live API allowance is smaller and pricier than the CDN's, does not bite: in production the Next.js Data Cache fronts every fetch, with force-cache and tags, so a page serves from cache until a webhook busts its tag. The live API call fires only on the first request after an invalidation, so a busy site makes a handful of Sanity calls a day, not one per visitor. You touch the API only when content actually changed.

Development flips the trade: local iteration would otherwise spend live API calls on every reload, so useCdn goes true and a few seconds of edge lag costs nothing while you build. Bypassing it in production also closes a gap that catches teams who leave it on, where a webhook fires before the edge finishes propagating and the revalidation re-caches the old value.

Time-based revalidation is a guess

revalidate: 60 tells Next.js to serve a cached page for up to sixty seconds before regenerating it. For content that changes on a schedule, that is fine. For a typo fix an editor wants live now, it means up to sixty seconds of the wrong text on a page someone is reading. Shorten the window and you trade freshness for load on Sanity. Set revalidate: 0 and you remove caching entirely, which solves staleness by giving up the performance you came to Next.js for. The value you actually want is revalidate: false, cache until something invalidates it by hand, which is where tags come in.

Tag-based revalidation is the production answer

Time-based revalidation expires a page on a clock. Tag-based revalidation expires it on an event, and the event is an editor pressing publish. You attach tags to a fetch, then bust those exact tags when matching content changes. Nothing else regenerates. A homepage tagged homepage and a post tagged post:${slug} revalidate independently, each one only when its own document moves.

The wiring has three parts. Your sanityFetch helper passes tags and, when tags are present, sets revalidate: false so the two strategies never fight. A route handler at /api/revalidate in the App Router validates the request and calls revalidateTag for whatever changed. A GROQ-powered webhook in Sanity calls that route on create, update, and delete, with a filter narrow enough that you are not revalidating the world on every keystroke.

// Invalidation flow
//
// Publish -> GROQ webhook -> POST /api/revalidate -> revalidateTag(tag)
//   -> Next.js Data Cache drops the tag -> next request MISS
//   -> Sanity live API (useCdn: false) -> fresh page

This is more setup than a single number, and it is the difference between a site that updates the instant an editor publishes and one that updates eventually. The publish half is framework-neutral, one webhook busting tags for the document that changed, and on-demand revalidation through tags is the pattern every production Sanity site converges on, whatever renders it.

Astro caches responses, not fetches

The two-cache model is not a Next.js quirk. It is what happens whenever a framework caches between the CMS and the browser, so swapping Next.js for Astro leaves the Sanity CDN exactly where it was and changes only the second cache. Astro has no data cache wrapping every fetch. It caches the rendered response for a route through Astro.cache, backed by a provider configured in astro.config.mjs: Vercel's CDN in production, an in-memory LRU in development.

The difference shows up when you go looking for the stale copy. In Next.js you tag a fetch, so one page can hold a fresh call next to a stale one. In Astro you tag the response, so a route is a hit or a miss as a whole. Astro.cache.set takes the tags for the page it just rendered: the document's _type, a doc:<id> tag, and the path it is routed at. Because the header and footer come from site-wide singletons that render on every page, every response carries those types too, so publishing one of them invalidates the site through a single tag. The publish half is unchanged from above: the same webhook, busting the same tags, with useCdn false in production.

What Astro adds is the bypass. Because it stores whole responses, any request that must not receive cached HTML has to miss on purpose: an editor in draft mode carrying the preview cookie, an agent negotiating for Markdown, a request behind Basic Auth. In memory that is a lookup you skip. On the CDN the lookup happens at the edge before your function runs, so it becomes a header contract instead: Vary: Cookie, Authorization, Accept gives those requests their own cache key, and anything rendered for one of them is marked private, no-store so it is never stored. Miss that and you have a fresh staleness bug with nothing to do with Sanity, where an editor sees a cached published page instead of their own draft.

Draft mode is the same problem inverted

Editors need the opposite of a cache. They want to see content that is not published yet. Draft mode handles it by switching the perspective of your fetch from published to drafts and forcing useCdn: false, so the Studio preview shows work in progress instead of the live page. It is the two-cache model again, read from the other side. Get the boundary right once and preview, live content, and revalidation all flow through the same sanityFetch, each picking the right cache for its job.

Why not let defineLive do all of this

On Next.js, and only there, Sanity ships defineLive, a fetch helper and a <SanityLive> component that handle caching, revalidation, and draft mode for you, with real-time updates as content changes. For most applications it is the recommended path, and it removes most of the wiring above. I still reach for the manual setup on production builds, for one current reason.

On Next.js 16, <SanityLive> interacts with the default link prefetch in a way that multiplies requests: a published change invalidates the client cache, prefetches fire again, tagged routes re-fetch and re-write, and your Sanity API and Vercel ISR bills climb with traffic. Sanity has documented this and for now recommends Next.js 15 with the older toolkit, or, on 16, driving revalidation from a Sanity Function instead of rendering <SanityLive> everywhere. Until it settles, a hand-built tag-based layer with the CDN bypassed in production is the cost I can predict.

The part nobody quotes for

The redeploy that fixed the price never fixed anything. It cleared two caches by accident and left the boundary between them undefined, which is why the bug came back. Owning that boundary, with the CDN off where freshness matters and the framework cache busted by tags on publish, is what turns stale content from a recurring incident into a solved problem.

It is also the work that never makes it into an estimate and gets rebuilt on every new project. The fetch layer in The Content Architecture is this, already decided in both the Next.js and Astro editions: useCdn handled per environment, tag-based revalidation wired to a webhook, draft mode and live preview connected, so a client build starts past the part that usually costs the first three days. For the reasoning behind the rest of the system, the companion piece on content architecture covers how the schema underneath it is modeled.

Common questions

Why is my Sanity content not updating in Next.js?

Two caches sit between your content and the page: the Sanity CDN and the Next.js Data Cache. An edit can be live in Sanity while one still serves the old copy. Bust the Next.js cache on publish with tag-based revalidation, and bypass the CDN in production so the refetch returns fresh content.

Why do my Sanity changes only show after a redeploy?

A redeploy clears the framework cache as a side effect, so the new content appears and then goes stale again on the next edit. Wire a webhook to revalidate the affected tags the moment an editor publishes, and the redeploy stops being part of the loop.

Should useCdn be true or false in production?

False in production, true in development. In production you bypass the CDN so reads return the current content at once, and the framework cache, the Next.js Data Cache or Astro's route cache, keeps API usage low by serving almost every request without calling Sanity. In development the CDN is the cheaper option, since local iteration would otherwise spend live API calls on every reload.

How do I revalidate a page when I publish in Sanity?

Tag your fetches, or your responses on Astro, then bust those tags on publish. A GROQ-powered webhook calls an endpoint that runs revalidateTag on Next.js, or cache.invalidate({ tags }) on Astro, for the document that changed.

Is defineLive safe to use on Next.js 16?

It works, but on Next.js 16 the <SanityLive> component can multiply requests through link prefetch and raise your Sanity API and Vercel ISR bills. Sanity suggests Next.js 15 with the older toolkit, or driving revalidation from a Sanity Function on 16.

Does Astro have the same stale content problem?

Yes, with a different second cache. Astro caches the rendered response for a route through Astro.cache, on Vercel's CDN in production, instead of caching each fetch. The fix is the same: bust the route's tags on publish and keep useCdn: false in production.