Why your best content is invisible to ChatGPT (and how to fix it)
A brand can write the sharpest, most quotable page on the internet and still lose to a worse competitor with a cleaner robots.txt. This is the full checklist — the three files, how they differ from classic SEO, and every related question worth asking before blaming the content.
The short answer
Before a single page gets rewritten for AI answers, check whether AI crawlers can even reach the site — content quality never gets a vote if the door is locked. Three boring, fixable things quietly block visibility more often than weak writing ever does: a robots.txt that accidentally disallows GPTBot, ClaudeBot, PerplexityBot, or Google-Extended; no sitemap.xml (or a broken one) telling crawlers what actually exists; and no llms.txt pointing an assistant at what matters most before it has to guess from a full crawl. None of these guarantee a citation. All three can quietly guarantee a brand never gets the chance at one.
If you’ve ever rewritten a page for the third time, convinced the problem was the headline, or the hook, or the proof — and AI assistants still act like the page doesn’t exist — you aren’t alone, and you probably aren’t wrong about the writing. A florist workspace we looked at had the opposite problem in reverse: it was named in every single tested AI answer for its category, a result most brands never see. Its robots.txt was clean. Its sitemap.xml was current. Neither had ever been checked on purpose — nobody had looked, and it was pure luck that looking wasn’t overdue. Most brands aren’t that lucky, and the only way to find out which kind you are is to actually look.
Here’s the part that stings: none of this shows up as an error. A blocked crawler doesn’t throw a message a human ever sees. A missing sitemap doesn’t break the site. The page keeps loading, keeps looking fine in a browser, keeps passing every check a person would think to run — and keeps quietly failing to reach the one audience it was written for.
robots.txt: the bouncer that doesn’t know who it’s turning away
Think of robots.txt as a bouncer standing at the door of a site, holding a clipboard of names. It’s a plain text file at the root of a domain — yourdomain.com/robots.txt — that tells crawlers which parts of a site they’re allowed to fetch. Nothing forces a crawler to obey it; it’s advisory, the honor system. But every major AI crawler honors it anyway: OpenAI’s GPTBot, Anthropic’s ClaudeBot, Perplexity’s PerplexityBot, and Google-Extended (which governs Gemini’s use of a site, separately from classic Googlebot).
The bouncer almost never turns someone away on purpose. The common failure is a blanket rule written for a completely different reason — blocking aggressive scrapers, or a security tool’s default configuration — that happens to catch these four names too, or catches everything under a wildcard User-agent: * without a more specific group carving out an exception. Nobody wrote the rule to keep ChatGPT from citing the business. It just does that anyway, silently, every single time it’s asked.
sitemap.xml: the floor plan versus the treasure hunt
A sitemap is an XML file listing a site’s real URLs, usually at yourdomain.com/sitemap.xml. Without one, a crawler discovers pages the hard way — following links from wherever it happens to land, hoping it eventually stumbles onto everything. Hand it a sitemap instead, and it’s the difference between a treasure hunt and a floor plan. A new page, a page with no internal links pointing to it, or a page buried five clicks deep can all sit undiscovered for a long time without one.
The two failures here are almost mirror images of each other: no sitemap at all, or a sitemap that technically loads but isn’t valid XML — a redirect to the homepage that happens to return a 200 status, for instance, which looks completely fine to a human clicking the link and is useless to anything trying to parse it as a list of URLs.
Do AI crawlers need different rules than Google?
Same file, same syntax, different names on the guest list. A classic SEO audit checks Googlebot and calls it done; an AI-visibility audit has to check four more names in the same file, because a rule aimed at one almost never automatically covers the others.
| Question | Classic SEO (Googlebot) | AI visibility (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) |
|---|---|---|
| What it feeds | Google Search rankings and snippets | ChatGPT, Claude, Perplexity, and Gemini answers — a separate pipeline, even for Google's own Gemini |
| Where the rule lives | The same robots.txt file | The same robots.txt file, under a different User-agent name |
| Most common mistake | Blocking staging or admin paths, usually on purpose | Getting caught by a blanket rule aimed at scrapers or bots in general, almost never on purpose |
| How to verify | Google Search Console's URL inspection tool | Load robots.txt directly and read it — there's no equivalent console for most of these yet |
llms.txt: a welcome packet, not a requirement
llms.txt (the convention lives at llmstxt.org) is a plain-text file that works like a short welcome packet handed to a new hire on day one — a curated summary of the business and links to the pages that actually matter, instead of making the assistant infer all of that from a full crawl. It’s genuinely emerging: support and actual use in generating an answer vary by system, and nothing about it is guaranteed the way a real citation is. Publishing one costs a few minutes and can’t hurt. It shouldn’t come before robots.txt or sitemap.xml on anyone’s list, but it’s a reasonable line item once those two are actually confirmed clean.
What if a site has none of these three files?
This is the scenario most people assume is the worst case, and it’s actually not. No robots.txt means full access by default under the standard — nothing is blocked. No sitemap means discovery slows down, since crawlers are stuck following links instead of reading a list, but nothing is actively refused. No llms.txt costs essentially nothing today, since support is still emerging.
The genuinely dangerous state isn’t having zero of these files. It’s having a robots.txt that blocks by accident. A site with no robots.txt at all is more visible to AI crawlers than a site whose robots.txt was written for an unrelated reason and happens to shut the door on GPTBot, ClaudeBot, PerplexityBot, or Google-Extended without anyone knowing it.
- 1
Check robots.txt
Load yourdomain.com/robots.txt directly. Look for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended by name, and check the wildcard group too.
- 2
Check sitemap.xml
Load it directly and confirm it's real XML — a <urlset> or <sitemapindex> element, not a redirect or an error page dressed up as a 200.
- 3
Add llms.txt
A short, honest summary of the business and links to the pages worth reading. Low effort, still optional, worth doing once the first two are settled.
- 4
Check for a stray noindex
A meta robots noindex tag left over from a staging environment blocks a page just as completely as a bad robots.txt rule, and has nothing to do with it.
A page can be the best answer to a buyer’s question on the entire internet and still lose, quietly, to a worse page with a cleaner robots.txt.
The failure that isn’t a file at all
The fourth item above — a leftover noindex meta tag — is worth calling out on its own, because it has nothing to do with any of these three files and still produces the exact same symptom: a page that quietly can’t be found. It’s a tag sitting in the page’s own HTML, usually left behind from a staging or pre-launch version of the site that got pushed to production without anyone checking. Rendering that depends entirely on client-side JavaScript causes a related, equally invisible problem: a crawler that only fetches static HTML sees an empty shell, even though the page looks completely normal to a person sitting in front of a browser.
None of this replaces the work of actually writing a page AI assistants want to quote. It just makes sure that work has a chance to be read in the first place — and that’s the whole point. Great content behind a locked door still doesn’t get cited.
Common questions about robots.txt, sitemap.xml, and llms.txt
How do I check if my robots.txt blocks AI crawlers?+
Visit yourdomain.com/robots.txt directly and look for a User-agent line matching GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, followed by Disallow: /. That's a full block for that surface. A Disallow line under User-agent: * blocks everyone by default unless a bot has its own group overriding it, so check both.
Does having no robots.txt file at all block AI crawlers?+
No — the opposite, technically. No robots.txt means full access by default under the standard. But it also means you never declared a sitemap location or stated a policy, which is worth doing even though its absence isn't itself a block.
Do I need separate rules for Google versus ChatGPT, Claude, or Perplexity?+
Yes, in the sense that they're different named user-agents and a blanket rule can catch some without catching others. No, in the sense that the file format and the logic are identical — one robots.txt, with a group for each bot that needs its own exception. Googlebot and GPTBot are just two more names on the same list.
Is llms.txt actually used by ChatGPT, Claude, or Perplexity yet?+
Support is inconsistent and still emerging — llms.txt (the convention lives at llmstxt.org) is not yet a confirmed input for every assistant's answer generation. Publishing one costs very little and can't hurt, but treat it as a forward-looking bet, not a guaranteed lever, and don't skip robots.txt or sitemap.xml to spend time on it instead.
My sitemap.xml exists — how do I know if it's actually valid?+
Open the URL directly and confirm the response is real XML starting with a <urlset> or <sitemapindex> element, not an error page or a redirect to your homepage that happens to return a 200 status. A sitemap that technically loads but isn't valid XML is functionally the same as no sitemap at all for anything trying to parse it.
What happens if a site has none of these three files at all?+
No robots.txt means default full access, which is actually fine on its own. No sitemap means crawlers rely entirely on following links to find pages, which quietly slows discovery of anything new or buried. No llms.txt means nothing lost yet, since support is still emerging. The real risk isn't having zero files — it's having a robots.txt that blocks by accident, which is worse than having none.
How often should these three files be rechecked?+
Whenever a security tool, CDN, or platform migration changes automatically, and otherwise once a quarter as a baseline. A robots.txt block is usually introduced by something unrelated changing configuration, not by anyone editing the file on purpose, so it can go unnoticed for months between checks.
What else besides these three files commonly blocks AI crawlers?+
A leftover noindex meta tag from a staging environment is the most common one we see — it has nothing to do with robots.txt and blocks a page just as completely. Content that only renders after JavaScript runs is another: a crawler that only fetches static HTML sees an empty page, even though a human visitor sees everything fine.
See whether your own site passes these checks
We check a workspace's robots.txt, sitemap.xml, and llms.txt against the real AI crawlers on every scan, and flag exactly what's blocking access before it costs a citation.
