llms.txt one year on: what AI systems actually read on your website.

In November 2025 SE Ranking published a study of llms.txt, the Markdown file that was supposed to tell AI systems what a website is about, across close to 300,000 domains. It found the file on about one in ten of them and no relationship between having it and being cited in AI answers. A year on, the evidence is much thicker: server logs from 137,000 domains, the contents of more than half a million files, and Google's own documentation. All of it points the same way.

This article sets out what the file was designed to do, what the studies found, what Google has said on the record, what AI systems actually read when they visit, and why we publish llms.txt and llms-full.txt on our own website anyway.

What llms.txt was designed to do.

The llms.txt proposal was published by Jeremy Howard in September 2024 as, in its own words, a proposal to standardise on using an /llms.txt file to provide information to help agents use a website. The file is plain Markdown: the name of the website, a short summary, then sections of links with a line of notes on each, plus an Optional section an agent can skip when it is short on room. The proposal also suggests publishing a Markdown copy of each page, so that an assistant can read the content without the navigation and scripts that surround it in HTML.

Read that way, the audience is an assistant already on the website that needs a map. The proposal, revised as recently as August 2026, still describes agents using a website; it does not promise that a search engine or an answer engine will find the website because of it. The second reading, llms.txt as a visibility tactic, grew up within months, complete with plugins to generate it and checklists that demanded it. That second reading is the one the evidence has now tested.

What the studies found.

SE Ranking found an llms.txt file on 10.13 per cent of the domains it checked, with adoption much the same across low, medium and high traffic bands. It then tested whether having the file related to how often a domain was cited by the prominent assistants, using correlation tests and a machine learning model. There was no meaningful relationship, and removing the llms.txt variable from the model slightly improved its accuracy. The authors concluded that the file does not affect how AI systems see or cite content today.

In June 2026 Ahrefs went to the server logs. Its study of 137,210 domains with traffic in May 2026 found that 28 per cent published a valid llms.txt, a figure the authors call an upper bound because their customers skew technical. Of those files, 97 per cent received no requests at all during the month. The roughly 1,100 domains whose file was fetched received about 22,000 requests between them, 96 per cent from bots. AI retrieval bots, the crawlers that OpenAI, Perplexity and Anthropic use to ground answers, made 1.1 per cent of those requests; AI agents and agent infrastructure, Claude Code among them, made 10.5 per cent, and SEO audit tools 21.7 per cent. Googlebot fetched the file about 900 times, which the authors read as ordinary crawling of a public URL, and Slackbot, unfurling links pasted into chat, fetched it more often than PerplexityBot did.

Two months later Common Crawl opened the files themselves. Working from its July 2026 crawl, it received a text body at /llms.txt from 11.72 per cent of the hosts it sampled and analysed 584,107 non-empty files. More than two thirds, 68.27 per cent, were generated by a template or plugin, and Wix alone accounted for 41.34 per cent of the corpus. Just over a fifth, 22.56 per cent, contained no links at all, so they could not do the one job the format exists for, and 6.59 per cent carried policy language, such as rate limits or demands to be cited, that nothing in the format can enforce. The authors' summary deserves repeating: despite the name, llms.txt is not like robots.txt. It grants nothing and blocks nothing, and no crawler is obliged to read it.

What Google has said on the record.

Google's position has been steady for seventeen months, moving from a social media reply to published documentation. In June 2025 John Mueller wrote on Bluesky that no AI system currently uses llms.txt, and when challenged he said the answer was obvious in anyone's server logs: the consumer chatbots fetch pages, for training and for grounding, and none of them fetch the file. A month later he offered a test that still stands: if an AI system that sends you significant traffic starts checking for the file as often as it checks robots.txt, that is the moment to think about it. His own logs did not show it.

In June 2026 the documentation caught up. On 15 June Google added a note on llms.txt to its guide to optimising for generative AI features in Search. It says that you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI capabilities, because Google Search itself does not use them. It is completely fine to create and maintain llms.txt files, and doing so will neither harm nor help a website's visibility or rankings, because Google Search ignores them. Google's separate page on AI features in Search says the same thing from the other direction: there are no additional requirements to appear in AI Overviews or AI Mode.

One nuance in Google's own behaviour is the honest case for the file. In May 2026 the platform that hosts Google's developer documentation began publishing Markdown versions of Search Central pages, and people asked why. Mueller answered that it was not done for search. He drew a line between discovery, being found through a search engine, and functionality, helping someone who has already found a page to do what they came to do. A coding assistant can work faster from Markdown, although he noted that these systems read HTML perfectly well and called the Markdown copy a temporary crutch that saves a few tokens. For websites that are not developer documentation he did not think it made much sense, and his closing advice was to prioritise needs before dreams.

What AI systems actually read.

Three kinds of automated visitor matter here, and only the third ever touches the file. Training crawlers such as GPTBot and ClaudeBot fetch pages at scale to build training data. Retrieval bots such as OAI-SearchBot and PerplexityBot fetch pages to ground an answer while a person waits, usually starting from a search index. Agents acting for a person, from ChatGPT opening a link to a coding assistant reading documentation, fetch the page they were pointed at, and some read llms.txt once they are there. In every case the unit of work is the page: its HTML and the text in it. Ahrefs' logs bear this out: Claude Code, a coding agent, fetched llms.txt more often than any AI retrieval bot or assistant, the audience the proposal described.

So the questions that decide what an AI system sees on your website are the ordinary ones. Can the page be fetched at all, or does robots.txt, a security rule or a firewall turn the crawler away? Is the answer to a plausible question in the HTML in plain words, under a heading that says what the section is about, or is it inside an image, a PDF or a script? Does the structured data agree with the visible copy? Do the pages that matter link to each other? None of this is new, which is Google's point.

Our own website gave us a small lesson in late October. For a few days the hosting security layer served a browser verification challenge to automated requests, including those carrying AI crawler user agents, while real browsers passed and our llms-full.txt was served without challenge. The text file helped nobody, because the systems that mattered wanted the pages. The host traced it to a server security tool and removed it, and checking that the pages answer to those user agents is now part of our monthly review.

Why we still publish one.

Emote publishes both files. Our /llms.txt is an index: who we are, a short block of key facts, and the main pages grouped by section, each with a one line note. Our /llms-full.txt carries the full text of every page, more than 370 of them, in a single file. Both are generated from the page text and refreshed as part of each release, so keeping them costs a few minutes in a deploy checklist. We keep them for the audience the proposal describes: a person who pastes our address into an assistant and asks a question, or an agent that has landed on one page and wants the rest. We make no claim about who fetches them and we do not expect them to bring a single visitor.

That leaves a short set of rules for anyone deciding whether to keep, add or remove the file.

• Generate it, never hand write it. A file that drifts from the pages is worse than no file.
• Put nothing in it that is not on the pages. No claims, no instructions to the model, no demands to be cited. The format cannot enforce them, and a file that reads like a sales pitch tells an agent what to discount.
• Do not use it for access control. robots.txt and your firewall decide who may crawl; llms.txt grants nothing and blocks nothing.
• If someone proposes it as the thing that will get you cited, the money is better spent on the pages and on reading your server logs to see which bots actually visit.

Mueller's test is the one to keep in mind. The day a system that sends you real traffic asks for llms.txt as routinely as it asks for robots.txt, the file has earned a place in the plan. Until then it is a courtesy to agents already on your website, and it should cost a few minutes of a developer's time.

How we can help.

AI search work at Emote starts with what the systems can fetch and what the pages say, not with a text file. We check crawler access and security rules, read the server logs for which AI bots are actually visiting, tidy the HTML so the answer to a question sits on the page in plain words, and keep the structured data in step with the visible copy. If you want to know what ChatGPT, Gemini, Perplexity or Google's AI Mode can see on your website today, that check is where a conversation with us begins.