llms.txt: we publish one, and no AI crawler has ever read it

There is a proposal called llms.txt: you put a markdown file at the root of your site, and it hands a language model a curated index of your best pages so the model does not have to crawl the whole site and guess what matters. A companion llms-full.txt holds the full text in one document.

We publish both, and have done since May 2026: prosopo.io/llms.txt is a hand-maintained 19 KB index, and llms-full.txt is 370 KB of full page text.

Nothing had ever read them when we wrote this. Verified AI clients have fetched llms.txt three times since, against 511 fetches of robots.txt over the same twelve days, and the numbers are in the update at the end.

What four days of logs say

prosopo.io sits behind a CDN that writes one access log line per request, with the client IP and user-agent. We took four days of those logs and counted requests to the three files a crawler might use to find its way around: robots.txt, the XML sitemaps, and llms.txt.

User-agent strings are no use here, since anyone can send any string they like, so we checked every request against the crawler IP ranges the vendors publish themselves — OpenAI's per-crawler JSON files, Perplexity's, DuckDuckGo's, and Anthropic's published AWS range — and counted only those from a verified address.

Zero. Not one verified AI crawler asked for the file built for it.

The files did get requests — 33 hits on llms.txt and 7 on llms-full.txt over the same period. Every one came from an address we could not match to any published crawler range. Some of that traffic asked for /blog/procaptcha-vs-hcaptcha-comparison/llms.txt, a path that has never existed. That is the mark of a scanner working through a wordlist, not an agent following a convention.

Why this is the expected result

This is not a scandal, and llms.txt is not a scam: it is an unratified community proposal from September 2024 which site owners have taken up, but the companies that run the crawlers have not.

OpenAI publishes the names and IP ranges of GPTBot, OAI-SearchBot and ChatGPT-User along with the robots.txt directives each one obeys, Anthropic does the same for ClaudeBot and Claude-User, and Perplexity, DuckDuckGo, Apple and Google all publish crawler names and robots directives. None of them documents llms.txt support, and our logs match that.

There is also a structural reason the file is a harder sell than it looks. A hand-curated index is the site owner's own claim about which of its pages are most valuable. That is the claim a search system exists to judge for itself, and it is easy to game. robots.txt works because it states a restriction — something only the site owner gets to decide. llms.txt states a recommendation, and nobody has to take it.

What is actually working

We have measured which of our pages AI assistants really fetch. The answer has nothing to do with llms.txt.

Over the same four days, verified AI crawlers fetched our pages 446 times, about 112 a day. The fetches cluster on one shape of page: analyst-report explainers, standards explainers, structured pricing comparisons. Our Forrester Wave breakdown, our Gartner and WAAP piece and our enterprise pricing comparison are fetched several times more often per page than anything else on the site.

Those pages all answer a comparative or factual question with structured, numeric, attributable data — named vendors, named analysts, named standards, real prices in a table. An assistant writing an answer needs facts it can quote and credit, and no index file replaces that.

What to do instead

If you have an afternoon to spend on being useful to AI systems, spend it in this order.

Get robots.txt right. It is the file they do fetch: 120 times in four days in our case. Decide which AI crawlers you want reading your content and name them. The distinction worth knowing is between training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and live retrieval agents (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, DuckAssistBot). Block the first group and you stay out of training corpora. Block the second group and you drop out of answers today, which is almost certainly not what you want.

Keep your sitemap current. Fetched 43 times in four days, mostly by Anthropic. It is a solved problem and it still works.

Write the pages that get cited. Structured, factual, comparative, with the answer in the first paragraph and the numbers in a table rather than in prose.

Verify crawlers by IP, not user-agent. This matters more than the rest put together: get it wrong and your logs will tell you the wrong thing. In our four days, five Google Cloud addresses rotated fifteen separate AI crawler user-agents between them while requesting /.env, /.ssh/config, /actuator/threaddump and /api/env. Of their requests, 45 to 59% went to paths that have never existed on our site. If you had trusted the user-agent you would have concluded that Grok, Mistral and Cohere were crawling you hard, and every single xAI-branded request we saw came from that fleet rather than from a verifiable address.

And publish an llms.txt if you want. The convention may yet be adopted, and being early is cheap. Just do not put it in a report as a channel, and do not let it crowd out the robots.txt work. On today's evidence it is a file you maintain for an audience that has not arrived.

Our numbers, for reference

To reproduce this on your own domain you need CDN or server access logs with the client IP intact, and the published range files. Our one caveat was retention: our CDN keeps only four days of logs, and until 5 October 2026 it archived nothing, which is why the original measurement covers four days rather than four months. If you want to track AI retrieval over time, turn on log archiving before you need it, not after — we did that on the day this article went up, and the update below is the result.

Update 18 October 2026

llms.txt has since been read. Over the twelve full days from 6 to 17 October 2026, verified AI crawlers made 7,585 fetches from prosopo.io, and these are the ones that went to the files a crawler uses to find its way around:

The three llms.txt requests were GPTBot on 6 October, Claude Code fetching the file on behalf of a person on 11 October, and ClaudeBot on 13 October, and all three returned 200. The one llms-full.txt request came from a Meta address on 14 October.

So the file is not completely ignored, but at one fetch every four days against 43 a day for robots.txt, the conclusion does not change: robots.txt and a current sitemap are the files that carry AI crawler traffic, and llms.txt is still a bet rather than a channel.