What an llms.txt file does, what the crawl data says about whether AI reads it, and whether you should bother to put one on your site.
An llms.txt file will not get you found in AI search. Surprised? Ahrefs, an SEO software company, checked about 38,000 sites that had an llms.txt file. In May 2026, 97 percent of those sites got no requests for the file at all.
An llms.txt file is good for one thing: helping AI agents that are already working inside your site. That is totally different than being cited by AI.
If you thought llms.txt helped you get found by AI, you have been sold a lie.
That 97 percent comes out of Ahrefs' own web traffic data, covering about 137,000 websites, published in June 2026. Twenty-eight percent of those websites had an llms.txt file.
Ahrefs says two things that most people quoting them don't pass along.
The 97 percent counts every kind of request, not just AI crawlers trying to make sense of your site.
And these are Ahrefs customers. People who think about search for a living. If only 28 percent of them bothered, the share of ordinary business websites carrying an llms.txt file is lower still.
None of the AI companies have agreed to use llms.txt. OpenAI's crawler follows robots.txt and has never adopted llms.txt. Anthropic publishes its own llms.txt file but has never said its crawlers read the llms.txt files on other websites. And Google runs its AI crawling through robots.txt.
John Mueller, who speaks for Google Search, said it more bluntly. Look at the server logs for any website and you will see the AI crawlers never ask for llms.txt.
Now I'm going to argue the other side, because llms.txt files are not a bad idea.
The file is cheap. It is honest. It tells an AI what is on your site and where to find it, and it doesn't pretend to be a ranking trick the way most of what gets sold as GEO does. If agents do start reading these files, anyone who already has one loses nothing by being early.
The problem isn't the file. It is that nobody needs it yet. The companies who would have to use llms.txt already crawl every public website they want. They don't ask for permission to read your site. The most you get is the chance to block them in robots.txt afterwards.
So they don't need you to hand them a tidy summary of your site.
Jeremy Howard, an AI researcher, came up with llms.txt in September 2024. He wrote that the file exists so an AI agent can use a website. His exact words: used on demand, when an agent needs information about a topic while assisting a user.
Howard even separates llms.txt from the two files people mix it up with. robots.txt tells crawlers what they are allowed to read. sitemap.xml lists your pages for search engines.
Nowhere does he say an llms.txt file gets you cited.
llms.txt was never built to help anyone find you.
Mueller made a second point on Google's own podcast, and it is the one that sticks. You write a file listing your own best pages. So does everybody else.
Every one of those files says the same thing: these are my best pages. A model cannot use that to tell you apart from anyone else.
Google's Chrome team lists llms.txt in Lighthouse, a free tool it makes for checking web pages. People read that as Google backing the file.
Read the page. The Lighthouse check only looks at whether your website returns an error when something asks for the file. If you have no llms.txt file at all, Lighthouse marks it not applicable. Google's own words: providing one is optional at the moment.
That is Google checking the file loads for an agent. It is not Google saying the file helps you show up in search.
robots.txt is a different story. That file decides whether an AI can read your website at all, and OpenAI, Anthropic and Google all follow it. One wrong line in robots.txt will cost you more than a missing llms.txt ever will.
Go check robots.txt.
My own website has an llms.txt file. It holds my bio and the same description of my work that sits on the rest of the site.
I have no evidence it has done anything. And I sell AI visibility work, so weigh that.
The risk is that you might end up creating an llms.txt file and then check AI visibility off your list. Then AI continues to answer your buyer's question with the content on someone else's site.
My recommendation. Do both a robots.txt and an llms.txt file, and in that order. robots.txt decides whether AI can read you at all, so it goes first. The llms.txt file costs you an afternoon. Nobody can show you it helps. Not yet anyways.
Then ask AI the question your buyer would ask, without putting your name in it. If it names your competitor's website instead of yours, the page you designed to answer that question needs some work. I wrote last week about someone who paid an agency for sixty AI-optimised articles and got no citations, while a plain comparison page he had written himself kept getting quoted. That post is here.
Fix the page. Say your position in a sentence a model can lift whole. Put it near the top. If you would rather someone ran that test properly and handed you the list, that is what my AI Visibility Check is.
Save the file work for your last job. Don't do it first.