llms.txt: we publish one, and no AI crawler has ever read it

There is a proposal called llms.txt: you put a markdown file at the root of your site, and it hands a language model a curated index of your best pages so the model does not have to crawl the whole site and guess what matters. A companion llms-full.txt holds the full text in one document.

We publish both, and have done since May 2026: prosopo.io/llms.txt is a hand-maintained 19 KB index, and llms-full.txt is 370 KB of full page text.

Nothing has ever read them.

What four days of logs say

prosopo.io sits behind a CDN that writes one access log line per request, with the client IP and user-agent. We took four days of those logs and counted requests to the three files a crawler might use to find its way around: robots.txt, the XML sitemaps, and llms.txt.

User-agent strings are no use here, since anyone can send any string they like, so we checked every request against the crawler IP ranges the vendors publish themselves — OpenAI's per-crawler JSON files, Perplexity's, DuckDuckGo's, and Anthropic's published AWS range — and counted only those from a verified address.

Zero. Not one verified AI crawler asked for the file built for it.

The files did get requests — 33 hits on llms.txt and 7 on llms-full.txt over the same period. Every one came from an address we could not match to any published crawler range. Some of that traffic asked for /blog/procaptcha-vs-hcaptcha-comparison/llms.txt, a path that has never existed. That is the mark of a scanner working through a wordlist, not an agent following a convention.

Why this is the expected result

This is not a scandal, and llms.txt is not a scam: it is an unratified community proposal from September 2024 which site owners have taken up, but the companies that run the crawlers have not.

OpenAI publishes the names and IP ranges of GPTBot, OAI-SearchBot and ChatGPT-User along with the robots.txt directives each one obeys, Anthropic does the same for ClaudeBot and Claude-User, and Perplexity, DuckDuckGo, Apple and Google all publish crawler names and robots directives. None of them documents llms.txt support, and our logs match that.

There is also a structural reason the file is a harder sell than it looks. A hand-curated index is the site owner's own claim about which of its pages are most valuable. That is the claim a search system exists to judge for itself, and it is easy to game. robots.txt works because it states a restriction — something only the site owner gets to decide. llms.txt states a recommendation, and nobody has to take it.

What is actually working

We have measured which of our pages AI assistants really fetch. The answer has nothing to do with llms.txt.

Over the same four days, verified AI crawlers fetched our pages 446 times, about 112 a day. The fetches cluster on one shape of page: analyst-report explainers, standards explainers, structured pricing comparisons. Our Forrester Wave breakdown, our Gartner and WAAP piece and our enterprise pricing comparison are fetched several times more often per page than anything else on the site.

Those pages all answer a comparative or factual question with structured, numeric, attributable data — named vendors, named analysts, named standards, real prices in a table. An assistant writing an answer needs facts it can quote and credit, and no index file replaces that.

What to do instead

If you have an afternoon to spend on being useful to AI systems, spend it in this order.

Get robots.txt right. It is the file they do fetch: 120 times in four days in our case. Decide which AI crawlers you want reading your content and name them. The distinction worth knowing is between training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and live retrieval agents (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, DuckAssistBot). Block the first group and you stay out of training corpora. Block the second group and you drop out of answers today, which is almost certainly not what you want.

Keep your sitemap current. Fetched 43 times in four days, mostly by Anthropic. It is a solved problem and it still works.

Write the pages that get cited. Structured, factual, comparative, with the answer in the first paragraph and the numbers in a table rather than in prose.

Verify crawlers by IP, not user-agent. This matters more than the rest put together: get it wrong and your logs will tell you the wrong thing. In our four days, five Google Cloud addresses rotated fifteen separate AI crawler user-agents between them while requesting /.env, /.ssh/config, /actuator/threaddump and /api/env. Of their requests, 45 to 59% went to paths that have never existed on our site. If you had trusted the user-agent you would have concluded that Grok, Mistral and Cohere were crawling you hard, and every single xAI-branded request we saw came from that fleet rather than from a verifiable address.

And publish an llms.txt if you want. The convention may yet be adopted, and being early is cheap. Just do not put it in a report as a channel, and do not let it crowd out the robots.txt work. On today's evidence it is a file you maintain for an audience that has not arrived.

Our numbers, for reference

To reproduce this on your own domain you need CDN or server access logs with the client IP intact, and the published range files. Our one caveat is retention: our CDN keeps four days of logs and archives nothing, which is why this measurement covers four days rather than four months. If you want to track AI retrieval over time, turn on log archiving before you need it, not after.