I Made My Blog Discoverable by AI Agents — llms.txt, Content-Type, and the Agent Discovery Flow

There’s a new standard gaining traction: llmstxt.org. It’s what robots.txt was for search engine crawlers in 1994 — a convention that every website can follow to make itself machine-discoverable, but this time for AI agents. Instead of a disallow list, it’s a structured index with links and context.

I implemented it on nurazhar.com. Here’s how it works, how I built it, and what it looks like to an AI agent.

The problem: agents can’t scrape a static blog

Static blogs are hard for AI agents. They’re a tree of HTML pages with navigation, footers, and boilerplate. An agent has to scrape every page, parse the DOM, extract meaningful text from <article> tags, and figure out what’s content vs chrome. It wastes tokens on <nav> bars and <footer> copyright notices.

The llmstxt.org standard solves this with three conventions:

/llms.txt — a plain-text index file at the site root. Agents check it first, by convention, the same way they’d check robots.txt.
/llms-full.txt — the complete content in one response, for agents that want everything at once.
Content-Type: text/plain — agents use the Content-Type header to decide how to parse the response. Text is cheap; HTML is expensive.

The implementation

The blog is a Clojure/Babashka static site generator. Posts are markdown files with frontmatter, compiled to HTML via Pandoc, deployed to Cloudflare Pages. I added two functions to the build pipeline:

The generation lives in sitemap.clj alongside the sitemap and robots.txt generators — it’s the same category of machine-readable output:

(defn generate-llms-txt [posts]
  (let [featured (take 10 posts)
        all-tags (->> posts
                      (mapcat #(str/split (:tags %) #","))
                      frequencies
                      (sort-by val >)
                      (take 12))]
    (str
     "# Nur Azhar — Technical Blog\n"
     "> Systems automation, AI agents...\n"
     "A personal technical blog...\n"
     "## Getting Started\n"
     "- [Homepage](https://nurazhar.com/)...\n"
     "- [Full Index](https://nurazhar.com/llms-full.txt)...\n"
     "## Recent Posts\n"
     (str/join "\n" (map post->link featured))
     "## Topics\n"
     (str/join "\n" (map tag->link all-tags))
     "## For AI Agents\n"
     "- This file is at `/llms.txt`...\n")))

The generate-llms-full-txt function does the same but iterates over all 191 posts, including title, date, URL, tags, and description for each one. Both files are regenerated on every bb build — they’re never stale.

What an agent sees

When an AI agent visits nurazhar.com for the first time, the discovery flow is:

# Step 1: Convention-based discovery
curl -sI https://nurazhar.com/llms.txt
# → HTTP/2 200
# → content-type: text/plain; charset=utf-8
# → cache-control: public, max-age=3600

# Step 2: Read the index (5.5 KB)
curl -s https://nurazhar.com/llms.txt

And the agent gets a structured Markdown document:

# Nur Azhar — Technical Blog

> Systems automation, AI agents, decentralized infrastructure,
  and Linux engineering. 191 articles since 2025.

A personal technical blog by Nur Azhar covering Clojure,
Babashka, AI/LLM agents, Bitcoin/BSV, CachyOS/Arch Linux,
CI/CD pipelines, and PDPA Singapore compliance.

## Getting Started
- [Homepage](https://nurazhar.com/)...
- [Full Index](https://nurazhar.com/llms-full.txt)...
- [RSS Feed](https://nurazhar.com/feed.xml)...

## Recent Posts
- [How I Made My Blog PDPA SG Compliant](https://nurazhar.com/...)
- [The 2026 nurazhar.com Architecture](https://nurazhar.com/...)
... (10 posts with descriptions)

## Topics
- [architecture (37 posts)](https://nurazhar.com/tag-architecture.html)
- [clojure (34 posts)](https://nurazhar.com/tag-clojure.html)
... (12 topics with post counts)

## For AI Agents
- This file is at `/llms.txt` per the llmstxt.org standard.
- All links are absolute URLs.
- Content-Type: `text/plain; charset=utf-8`
- `/llms-full.txt` contains every post with descriptions.
- `/sitemap.xml` lists all URLs for crawling.

The agent now has a complete map of the site — no HTML parsing, no guessing, no wasted tokens on navigation bars.

Why dynamic generation matters

The old approach was a hand-maintained static llms.txt. It had three problems:

The static version was 766 bytes — a bare link list, manually written, that didn’t update when I published new posts. It said “171 articles” when there were actually 185. The dynamic version is 5,478 bytes and always accurate because it’s generated from the same post data that builds the HTML.

The Content-Type header

This detail matters more than you’d think:

Cloudflare Pages serves .txt files as text/plain by default, which is correct. But I added explicit headers via the _headers file to guarantee it and set a 1-hour cache:

/llms.txt
  Content-Type: text/plain; charset=utf-8
  Cache-Control: public, max-age=3600

The charset=utf-8 is important — without it, agents might interpret the response as ASCII and mangle Unicode characters in post titles. The cache header means agents that check multiple times in an hour don’t re-download the same 5.5 KB file.

The full index: /llms-full.txt

Some agents prefer to load everything in one request. llms-full.txt serves this need — it’s ~80 KB containing every post with structured metadata:

## How I Made My Blog PDPA SG Compliant
- **Date:** 2026-08-08
- **URL:** https://nurazhar.com/how-i-made-blog-pdpa-compliant.html
- **Tags:** pdpa, singapore, compliance, clojure, babashka, devops
- **Summary:** A step-by-step builder's journal of integrating
  pdpa-sg-clj into a static blog publishing pipeline.

An agent that fetches this gets: - Title — for deciding relevance - Date — for temporal sorting - URL — ready to follow, no path reconstruction - Tags — for topic filtering - Description — for relevance scoring, extracted from frontmatter

This is essentially a machine-readable sitemap with semantic metadata. It’s Markdown-heavy because Markdown is the closest thing we have to a universal structured text format that both humans and machines can parse.

The agent-friendly-apis skill

I used the agent-friendly-apis skill from Vercel Academy as a reference for the implementation. It covers seven documentation patterns for making APIs consumable by AI agents, but the llms.txt section was the most directly applicable to a static blog:

Pattern	Applied?	How
Endpoint signatures in code blocks	N/A	No API endpoints
Parameters as markdown tables	N/A	No API parameters
Curl examples with real values	N/A	No API calls
Complete response bodies	✅	`/llms-full.txt` contains every post
Schema + errors	✅ / N/A	Frontmatter schema in AGENTS.md; HTTP only
Workflow examples	✅	Agent guidance section in `/llms.txt`

The skill also emphasizes progressive disclosure — splitting content so agents can load what they need without wasting context window. This maps directly to the /llms.txt (index) → follow links (specific content) or /llms-full.txt (everything) pattern.

The result

Before this change, an AI agent encountering nurazhar.com had to scrape HTML and guess. After:

Metric	Before	After
Discovery mechanism	Agent must crawl HTML	Convention-based `/llms.txt`
Content format + token cost	HTML DOM (~thousands of tokens)	`text/plain` Markdown (~1,400 tokens)
Full content access	642 HTML pages to crawl	Single 80 KB `/llms-full.txt`
Freshness guarantee	None (static file)	Every build regenerates
Post counts	Stale (said 171, was 185)	Always accurate (191)
Content-Type	Implicit	Explicit `text/plain; charset=utf-8`

The implementation took two Clojure functions, one _headers file, and zero new dependencies. It’s generated on every build, costs nothing to serve, and makes the blog discoverable by any AI agent that follows the llmstxt.org convention.

Built with Babashka 1.4, Clojure, Cloudflare Pages. Follows llmstxt.org standard. Agent-friendly-apis skill from Vercel Academy. Check it yourself: curl https://nurazhar.com/llms.txt.