robots.txt, sitemap.xml, and llms.txt: the simple checks most sites still get wrong
A brand can write the sharpest, most quotable page on the internet and still lose to a competitor with a worse page and a cleaner robots.txt. Here are the basic files worth auditing first.
The short answer
Before a single page gets rewritten for AI answers, check whether AI crawlers can even reach it. Three boring, fixable things quietly block visibility more often than content quality ever does: a robots.txt that accidentally disallows GPTBot, ClaudeBot, PerplexityBot, or Google-Extended; no sitemap.xml (or a broken one) telling crawlers what actually exists on the site; and no llms.txt pointing an assistant at what matters most before it has to guess from a full crawl. None of these guarantee a citation. All three can quietly guarantee you never get the chance.
A florist workspace we looked at had a real, hard-won advantage: it was named in every single tested AI answer for its category, a result most brands never see. Its robots.txt was clean. Its sitemap.xml was current. Neither had ever been checked, because neither had ever caused a visible problem. The lesson wasn’t that these files matter for that workspace specifically. It was that nobody had looked, and it was pure luck that looking wasn’t overdue.
Most technical-access problems are invisible until you go looking for them. A blocked crawler doesn’t throw an error a human ever sees. A missing sitemap doesn’t break the site. The page keeps loading, keeps looking fine in a browser, and keeps quietly failing to reach the one audience it was supposed to reach.
robots.txt: the file most likely to be wrong by accident
robots.txt is a plain text file at the root of a domain — yourdomain.com/robots.txt — that tells crawlers which parts of a site they may fetch. It’s advisory, not enforced by anything, but every major AI crawler respects it: OpenAI’s GPTBot, Anthropic’s ClaudeBot, Perplexity’s PerplexityBot, and Google-Extended (which governs Gemini’s use of a site separately from classic Googlebot).
The common failure isn’t a business deliberately blocking AI crawlers. It’s a blanket rule written for a different reason — blocking aggressive scrapers, or a security tool’s default configuration — that happens to catch these four names too, or catches everything under a wildcard User-agent: * without a more specific group carving out an exception. Nobody wrote it to keep ChatGPT from citing the business. It just does.
sitemap.xml: discovery without a treasure hunt
A sitemap is an XML file listing a site’s real URLs, usually at yourdomain.com/sitemap.xml, and it’s the difference between a crawler discovering pages by following links from wherever it happens to land, versus being handed the actual list. A new page, a page with no internal links pointing to it, or a page buried five clicks deep can all go unindexed for a long time without one.
The two failures here are almost mirror images of each other: no sitemap at all, or a sitemap that technically loads but isn’t valid XML — a redirect to the homepage that happens to return a 200 status, for instance, which looks fine to a human clicking the link and is useless to anything trying to parse it as a list of URLs.
llms.txt: a small bet, not a requirement
llms.txt (the convention lives at llmstxt.org) is a plain-text file that gives an AI assistant a curated summary of a site and links to the pages that matter most, instead of making it infer that from a full crawl. It’s genuinely emerging — support and actual use in generating an answer vary by system, and nothing about it is guaranteed the way a real citation is. Publishing one costs a few minutes and can’t hurt. It shouldn’t come before robots.txt or sitemap.xml on anyone’s list, but it’s a reasonable line item once those two are actually confirmed clean.
- 1
Check robots.txt
Load yourdomain.com/robots.txt directly. Look for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended by name, and check the wildcard group too.
- 2
Check sitemap.xml
Load it directly and confirm it's real XML — a <urlset> or <sitemapindex> element, not a redirect or an error page dressed up as a 200.
- 3
Add llms.txt
A short, honest summary of the business and links to the pages worth reading. Low effort, still optional, worth doing once the first two are settled.
- 4
Check for a stray noindex
A meta robots noindex tag left over from a staging environment blocks a page just as completely as a bad robots.txt rule, and has nothing to do with it.
A page can be the best answer to a buyer’s question on the entire internet and still lose, quietly, to a worse page with a cleaner robots.txt.
The one that isn’t a file at all
The fourth item above — a leftover noindex meta tag — is worth calling out on its own, because it’s the one that has nothing to do with any of these files and still produces the exact same symptom: a page that quietly can’t be found. It’s a tag in the page’s own HTML, usually left behind from a staging or pre-launch version of the site that got pushed to production without anyone checking. Rendering that depends entirely on client-side JavaScript causes a related, equally invisible problem: a crawler that only fetches static HTML sees an empty shell, even though the page looks completely normal to a person with a browser.
None of this replaces the work of actually writing a page AI assistants want to quote. It just makes sure that work has a chance to be read in the first place.
Common questions about robots.txt, sitemap.xml, and llms.txt
How do I check if my robots.txt blocks AI crawlers?+
Visit yourdomain.com/robots.txt directly and look for a User-agent line matching GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, followed by Disallow: /. That's a full block for that surface. A Disallow line under User-agent: * blocks everyone by default unless a bot has its own group overriding it, so check both.
Does having no robots.txt file at all block AI crawlers?+
No — the opposite, technically. No robots.txt means full access by default under the standard. But it also means you never declared a sitemap location or stated a policy, which is worth doing even though its absence isn't itself a block.
Is llms.txt actually used by ChatGPT, Claude, or Perplexity yet?+
Support is inconsistent and still emerging — llms.txt (the convention lives at llmstxt.org) is not yet a confirmed input for every assistant's answer generation. Publishing one costs very little and can't hurt, but treat it as a forward-looking bet, not a guaranteed lever, and don't skip robots.txt or sitemap.xml to spend time on it instead.
My sitemap.xml exists — how do I know if it's actually valid?+
Open the URL directly and confirm the response is real XML starting with a <urlset> or <sitemapindex> element, not an error page or a redirect to your homepage that happens to return a 200 status. A sitemap that technically loads but isn't valid XML is functionally the same as no sitemap at all for anything trying to parse it.
What else besides these three files commonly blocks AI crawlers?+
A leftover noindex meta tag from a staging environment is the most common one we see — it has nothing to do with robots.txt and blocks a page just as completely. Content that only renders after JavaScript runs is another: a crawler that only fetches static HTML sees an empty page, even though a human visitor sees everything fine.
See whether your own site passes these checks
We check a workspace's robots.txt, sitemap.xml, and llms.txt against the real AI crawlers on every scan, and flag exactly what's blocking access before it costs a citation.
