Foundational
Markdown content negotiation: serving clean text to AI bots
tl;dr
Content negotiation lets your server return markdown to clients that ask for it (Accept: text/markdown) and HTML to everyone else. It's a 20-line config change and it dramatically improves how AI assistants understand your pages.
// your move
See what AI assistants say about your site.
Free, 30 seconds, no signup. We'll check all 15 signals and tell you which ones to fix first.
Run the auditWhat's happening here?
Every HTTP request carries an Accept header that says what content types the client wants. Browsers ask for text/html. A growing number of AI fetchers ask for text/markdown.
If your server only knows how to serve HTML, the bot gets a noisy page full of nav, sidebars, popups, and inline scripts. If you respond with clean markdown, it gets exactly the content you wrote.
How it works in practice
- 01Detect the Accept header on incoming requests at the edge or in middleware.
- 02If it includes text/markdown, route to a handler that returns the page as markdown.
- 03Otherwise, serve your normal HTML page.
- 04Make sure your markdown version has the same canonical content as the HTML version — same headings, same body. Just strip the chrome.
- 05Cache aggressively. Markdown payloads are tiny and rarely change between requests.
Cloudflare Worker (any stack)
This sits in front of your origin and converts on the fly. Drop it in a Worker and route your domain through it.
- ·Read request.headers.get('Accept')
- ·If it includes 'text/markdown', fetch the page, run it through a Readability + Turndown pipeline, return with Content-Type: text/markdown
- ·Otherwise, pass through to origin
- ·Add Vary: Accept to the response so caches don't mix the two
Next.js / TanStack Start
Use middleware to detect the header and rewrite to a parallel /md route. Write your page once in MDX, render to HTML at the normal path and to markdown at /md/[slug]. Both stay in sync because they share a source file.
“This sits in front of your origin and converts on the fly.”
WordPress
There's no first-party support. The cleanest path is the Cloudflare Worker approach above. If you can't put Cloudflare in front, install a plugin like WP REST API and route /md/* through wp-json/wp/v2/pages plus a small filter that renders the_content as markdown.
Nginx (custom stack)
Add a map block that flips a variable based on the Accept header, then route accordingly. Pseudocode: map $http_accept $serve_md { default 0; ~text/markdown 1; } and then in your location block, if ($serve_md) { rewrite ^ /md$uri; }.
Common questions
Do I have to host a separate URL for markdown?
No. Content negotiation means the same URL returns different content based on what the client asks for. The bot and the browser hit /pricing — one gets markdown, the other gets HTML.
Will this confuse Google?
No. Googlebot asks for HTML and gets HTML. The negotiation only fires for clients that explicitly request markdown.
What about the Vary header?
Always include 'Vary: Accept' on responses. Otherwise CDNs and browser caches will serve the wrong version to the wrong client.
Keep reading
// your move
See what AI assistants say about your site.
Free, 30 seconds, no signup. We'll check all 15 signals and tell you which ones to fix first.
Run the audit ⚡