What Should You Actually Check in an llms.txt Audit?

What should you actually check in an llms.txt audit?

Six things: that the file exists where it claims to cover, that it follows the required structure, that every link resolves, that the linked targets are actually clean markdown, that the file is small enough to be useful, and that you have the link relations pointing at it. Most files I look at fail at least two of those.

The proposal has also moved, which is the part catching people out. The specification at llmstxt.org is now on v2, authored by Jeremy Howard, originally published September 3, 2024 and last modified August 10, 2026. A file built to the original guidance is not wrong, but it is missing things v2 asks for.

This is an audit checklist rather than a setup guide. If you have not created one yet, that is a different job.

Does the file need to be at the root?

No, and this is the first thing to check because it changes what your file is claiming. The spec says the file can be placed at the site root or at any path within it, covering the pages under that path, and that where more than one file applies, agents should use the most specific one.

So a file at /docs/llms.txt covers everything under /docs/, which the spec states directly. That is a deliberate design choice, and the specification explains the reasoning: unlike the well-known URIs standard, which reserves /.well-known/ at the origin root, a path-scoped file can be published by anyone who controls only a path on a shared host.

The audit question is therefore about scope rather than location. If your file sits at the root but only describes your documentation, it is claiming to cover pages it says nothing about. Either move it or broaden it.

Is your file structurally valid?

Check it against the spec's ordered sections, because the format is stricter than it looks. The specification lists an optional byte-order mark, then an H1 with the name of the project or site, then a blockquote with a short summary, then zero or more markdown sections of any type except headings, then zero or more sections delimited by H2 headers containing file lists.

The H1 is the only required section according to the spec, which surprises people who assume the whole structure is mandatory. Everything else is optional in the formal sense and extremely useful in practice.

File lists have their own rule worth checking line by line. The spec says each entry is a markdown list item containing a required markdown hyperlink in the name and url form, then optionally a colon and notes about the file. Entries without a link, or with the link in plain text, are not conformant and are trivially easy to introduce by hand.

Do your links point at clean markdown?

They should, and this is where most files quietly fail. The spec says links in an llms.txt file should point to LLM-friendly content, and proposes that pages agents might need provide a clean markdown version at the same URL as the original, either with .md appended, as in page.html.md, or with the extension replaced, as in page.md.

It also handles the directory case explicitly: URLs without file names should append index.html.md or index.md instead. That detail matters for any site where the useful pages are directory-style URLs, which is most content sites.

The audit step is mechanical and worth doing properly. Take every link in the file, request it, and check what comes back. A file pointing at HTML pages is doing half the job the spec describes, because the agent still has to strip navigation, and the specification opens by naming that exact problem as the reason the proposal exists.

Have you added the link relations?

Probably not, and this is the v2 addition people have missed. The specification recommends standard link relations so clients can find these files: rel alternate with type text slash markdown pointing to the markdown version of a page, and rel describedby pointing to the llms.txt file that covers it.

It gives two ways to provide them, and the second is the useful one for large sites. They can be HTML link elements, or an HTTP Link response header. The spec notes the header form also works for non-HTML resources such as the markdown files themselves, and can be added in web server or CDN configuration without modifying any pages.

That is a genuinely practical detail for anyone on a hosted platform where editing every page head is painful. If you can set headers at the CDN, you can implement this once for an entire site. I covered the platform side of setting these files up in setting up llms.txt on Webflow.

Is anyone actually reading it?

Some systems are, and some explicitly are not, and the honest audit says both. The specification reports that thousands of sites publish an llms.txt file, that documentation platforms generate one automatically, that Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, and that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer docs.

I can confirm part of the picture first hand, because I ran into it while researching other articles this week. Webflow's developer documentation tells AI agents a documentation index is available at /llms.txt, Zapier's documentation opens by pointing at the same path, and the Model Context Protocol documentation does too. Three unrelated vendors, same convention, visible on the page.

Google is the notable exception and says so plainly. Its optimization guidance states that you do not need such files to appear in Google Search including its generative capabilities, that Google Search itself does not use them, and that Google Search ignores them. That split is worth understanding properly, and I went into it in the Google split on llms.txt.

Is your file small enough to be worth reading?

Quite possibly not, and size is the failure nobody audits for. The spec's whole design is that the file stays small enough to fit in context, with the detail living behind the links and fetched only when needed. A file that has grown into a full index has defeated its own purpose.

Check the character count and then check the reason for it. Auto-generated files from documentation platforms are the usual culprit, because generation optimises for completeness while the format is asking for curation.

The format gives you a prioritisation mechanism most files ignore. The specification says the Optional section is used by convention for secondary information, meaning links an agent can skip when a shorter context is needed. If everything in your file is equally important, you have not made the decision the format is asking you to make, and an agent working with limited context has no way to triage.

So when I audit a file, I look at whether anything is in Optional at all. An llms.txt with forty links and no Optional section usually means somebody exported a sitemap into markdown and stopped thinking. The fix is moving two thirds of the entries into Optional or into a more specific file at a deeper path, both of which the spec supports.

Does it conflict with robots.txt or your sitemap?

No, and the spec is explicit about why. It states that llms.txt is designed to coexist with current web standards, that while sitemaps list all pages for search engines llms.txt offers a curated overview for LLMs, and that it can complement robots.txt by providing context for allowed content.

The distinction the specification draws between the two files is the one to keep straight in an audit. robots.txt lets automated tools know what access to a site is considered acceptable. llms.txt information is used on demand, when an agent needs information about a topic while assisting a user. Permission versus guidance.

So do not treat llms.txt as an access control mechanism, and do not let anyone tell you it replaces robots.txt. If your file lists pages that robots.txt disallows, you have a contradiction to resolve, and the resolution is a policy decision rather than a technical one.

What is the honest state of this standard?

It is a proposal with real adoption and no universal support, and anyone telling you otherwise is selling something. The spec describes itself as a proposal, and the author's own framing in v2 is that it was updated based on what he learned from two years of adoption.

My position is that the cost is low and the upside is real but unevenly distributed. If agents read your documentation to use your product, the file does work today. If your value is in marketing pages that no coding agent will ever fetch, the honest expected value is closer to zero, and I would not put it high on a client's list.

What I would not do is claim it drives rankings, because the largest search engine has published the opposite. The standards conversation is still moving too, which I wrote about in the llms.txt standards proposal.

What should you do next?

Open your own file and check three things in five minutes: is there an H1, does every list entry contain a real markdown link, and does each link return clean markdown rather than an HTML page. Those three catch most conformance failures I see.

Then decide whether you want the link relations, because that is the piece almost nobody has and the one the current version of the spec asks for. If you can set an HTTP Link header at your CDN, it is a single configuration change for the whole site. If you want someone to run this checklist over your file and tell you what is actually broken, reach out with the URL.