llms.txt vs robots.txt: the real difference (and whether you need both)

If you've read one guide telling you to fix your robots.txt so AI crawlers can reach your site, and another telling you to add an llms.txt so AI models understand what they find, you'd be forgiven for treating them as the same advice wearing two names. They are not. robots.txt is a permission file — it decides whether a crawler gets in at all. llms.txt is a description file — it only does anything once a crawler is already inside. Confuse the two and you can spend an afternoon writing a beautiful llms.txt while a stray wildcard rule in robots.txt keeps every AI crawler from ever reaching it. Here's the actual difference, what each file does, and — using real crawl-log data instead of a guess — how much either one is currently worth.

robots.txt: decades old, and it only grants or denies access

robots.txt is the Robots Exclusion Protocol, first defined by Martijn Koster in 1994 for site owners to tell automated crawlers where they may and may not go. It became a de facto web standard within months — every major search engine of the era complied — and stayed an informal convention, followed but never formally specified, for almost three decades, until the IETF finally standardized it as RFC 9309 in September 2022.

Its entire job, then and now, is access control, not description. A crawler reads it and learns exactly one kind of thing: am I allowed down this path, yes or no. It says nothing about what your site is, what you sell, or who it's for — and it was never meant to. In the AI-search era, this file governs a specific, current list of crawlers, not a vague "AI bots" category. Our own free crawlability checker reads your live robots.txt against the nine real user-agents behind the four engines this whole blog is about:

Block any of these at /, deliberately or by accident with a careless wildcard rule, and that engine cannot read your site — full stop, before content quality is even a question. The full field guide to all nine goes deep on the specific gotchas (training bots vs. answer bots, the Cloudflare edge-block trap); the point for this post is narrower: robots.txt is the gate, and nothing past it matters if the gate is shut.

llms.txt: two years old, and it only describes what's already inside

llms.txt is a much younger, unrelated convention — proposed by Jeremy Howard on September 3, 2024, and published at llmstxt.org. It doesn't grant or deny anything. It's a single plain-text file, hosted at your domain root, that hands a model a clean, curated summary of what your site is, in a format it reads well — instead of making it guess from scraped navigation menus and marketing copy. We've written the full spec and a real working example separately if you're ready to build one.

A gate and a welcome sign do two different jobs. robots.txt is the gate. llms.txt is the welcome sign posted just past it. A wide-open gate with no welcome sign is still readable — just messier, and the model has to work harder to summarize you correctly. A locked gate with the friendliest welcome sign ever written is never read at all, because nothing gets past the gate to see it.

Where the confusion actually comes from

Both files are plain text. Both live at the root of your domain. Both are conventions read by AI-related crawlers, and both are recent enough additions to the "machine-readable annotations" family that most people encountered them around the same time, in the same wave of "how to get AI to notice you" advice. That's the whole source of the mix-up — surface similarity, unrelated function.

Do you need both? Yes — but the order, and your expectations, matter

You need robots.txt, at minimum, to confirm you aren't accidentally blocking the crawlers you want in — most sites already have one; the risk is an overbroad rule, not a missing file. llms.txt is optional and additive: cheap to add, genuinely useful once the gate is open, worthless before it.

Here's the honest part, because a page telling you to "just add both" without checking whether either one moves the needle isn't much better than the guides that confuse them. Otterly.AI published a real 90-day server-log experiment: they put a correctly built llms.txt at a site's root and watched the raw traffic for three months. Of more than 62,000 AI-bot visits in that window, only 84 — about 0.1% — ever touched llms.txt. An ordinary content page on the same site pulled roughly 265 AI-bot visits over the same stretch: llms.txt got roughly a third the attention of a normal page, not more. (Worth disclosing plainly: these figures come from WebSearch summaries of Otterly's own published experiment, not a direct fetch of it — that fetch 403'd from this sandbox tonight, the same standing block every external domain has hit this pass. Two independently phrased searches converged on the identical numbers, which is the corroboration bar this blog holds itself to when a primary fetch isn't available. It's also one experiment, on one site, over one 90-day window — real data, not a universal law.)

That doesn't make llms.txt worthless. It makes it exactly what our own how-to post already called it: cheap, honest, worth doing, and not a lever you should expect to move much on its own. The sequencing that actually matters: confirm the gate is open first (a closed robots.txt makes everything downstream irrelevant), add llms.txt second because it's low-risk and can't hurt, and then stop treating either file as the thing that decides whether AI recommends you — because the data above says it mostly isn't.

The number neither file can show you

Both files are about whether a crawler can find and understand your site. Neither one tells you whether an AI engine actually names you when a real buyer asks for a recommendation — that's a different, bigger question, decided by content quality, third-party mentions, and structure, not by two small text files at your domain root.

Check your gate first with the free crawlability checker, draft your welcome sign with the free llms.txt generator if you don't have one yet, then run the free scan: 25 real buyer questions across ChatGPT, Claude, Gemini, and Perplexity, with the exact sentence each engine returned — no signup, about a minute. That's the number that actually decides deals.