llms.txt Explained: How to Configure Your Site for ChatGPT, Claude, and Perplexity in 2026

Somewhere between last year and this one, "just block the AI bots" stopped being a safe default. Now every AI provider runs at least two crawlers with opposite jobs, and getting the wrong one blocked can quietly erase a brand from ChatGPT, Claude, or Perplexity without a single error message.

Most sites still manage AI crawlers the way they managed search engines in 2015: one User-agent: * block, a couple of Disallow lines copied from a blog post, no review since. That approach made sense when a handful of crawlers all did the same job. It breaks down now that OpenAI, Anthropic, Perplexity, and Google each run separate bots for training versus live citations, and blocking the wrong one has a direct, measurable cost.

At Lureon, this is one of the first things we check in a technical GEO audit, because it is also one of the most common failure points. A client's content can be perfectly structured and still be invisible to ChatGPT Search, simply because a training-bot rule from 2023 is also blocking the search-time crawler that would have cited it. This guide covers the two pieces of infrastructure that decide whether AI systems can read your site at all: llms.txt and a properly configured robots.txt for the AI crawler landscape as it actually stands in 2026.

Key Takeaways

• llms.txt curates your best pages for AI models, not a ranking signal in 2026.
• Robots.txt, not llms.txt, actually decides whether AI crawlers can reach your content.
• Most providers run two bots, training and search; blocking one doesn't block the other.
• Roughly half of AI crawler traffic trains models; a smaller share drives citations.
• Blocking AI crawlers outright can cut traffic without reducing how often you're cited.

Why the Old Robots.txt Playbook Breaks for AI Crawlers

Traditional robots.txt management assumed a small, stable list of crawlers: Googlebot, Bingbot, maybe a handful of SEO tools. Each one did roughly one job, indexing your pages for a search result page a human would eventually click.

AI crawlers don't fit that model. According to Cloudflare Radar's AI Insights data, roughly half of all AI crawler requests across its network are for model training, a process with no return visit and no referral traffic. Only a smaller, faster-growing share is dedicated search or retrieval crawling, the kind that can put your page directly into a cited AI answer. Those two categories serve completely different purposes for your business, and treating them as one undifferentiated "AI bot" problem is exactly what causes sites to lose visibility without realizing why.

The stakes are asymmetric. Blocking a training crawler by mistake costs you nothing you'd notice this quarter. Blocking a search or retrieval crawler by mistake removes your brand from live AI answers immediately, often while your team is still congratulating itself on a "secure" robots.txt.

What llms.txt Actually Is

llms.txt is a plain Markdown file published at the root of your domain, at a fixed path: https://yoursite.com/llms.txt. It gives an AI model a short, curated list of your most important pages, each with a one-line description, rather than forcing it to parse a full HTML page full of navigation, cookie banners, and scripts to find the two paragraphs that actually matter.

The proposal came from Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024. The full specification is maintained at llmstxt.org and is deliberately minimal. A conformant file follows a fixed order:

• An H1 heading with your project or site name. This is the only required section.
• A blockquote summary, one or two sentences describing what the site does, containing the key context needed to understand everything below it.
• Optional, unheaded context paragraphs with more detail about the organization.
• One or more H2 sections, each containing a Markdown list of links with short, factual descriptions.

Some sites, particularly documentation-heavy platforms, also publish llms-full.txt, a companion file that concatenates full Markdown versions of key pages into one document. This is useful when an AI agent needs complete technical detail in a single fetch rather than following individual links.

Does it actually influence AI search rankings?

This is the question most guides either avoid or oversell, so it's worth being direct about it. As of mid-2026, there is no confirmed evidence that ChatGPT, Claude, or Perplexity weight llms.txt as a citation or ranking factor during live search. Adoption sits at roughly one in ten domains, concentrated heavily among developer tools and SaaS platforms, and the companies publishing the most polished examples, Anthropic, Stripe, Vercel, and Cloudflare among them, have not stated that it moves their own visibility.

Where llms.txt clearly does work is a step removed from search. Coding agents including Cursor, Windsurf, Claude Code, GitHub Copilot, and Cline actively fetch /llms.txt and /llms-full.txt when a developer points them at a library, an API, or a piece of documentation, using it to decide which pages are worth pulling into context before writing code. For a documentation-first product, that alone justifies building one. For a general marketing or content site, llms.txt is best framed as a low-cost hedge rather than a growth lever: it costs a few hours, carries essentially no downside, and is the kind of infrastructure that looks obvious in hindsight if a major provider ever does start weighting it directly.

llms.txt, robots.txt, and sitemap.xml: Three Different Jobs

These three files are frequently confused, but each answers a different question for a different audience:

• robots.txt answers "where am I allowed to go?" for any automated crawler. This is access control, and it is the only one of the three with a direct, provable effect on AI visibility.
• sitemap.xml answers "what pages exist?" for search engine indexers. It lists everything, with no curation.
• llms.txt answers "what matters most, and how should I interpret it?" for a language model working inside a limited context window. It is a hand-picked subset, not a complete inventory.

A well-curated llms.txt sitting behind a robots.txt that blocks every major AI crawler is invisible, full stop. Access always comes first. This is why the rest of this guide spends more time on robots.txt than on llms.txt itself: it's the part of the stack that has a proven, direct effect on whether your content ever gets read.

The AI Crawler Landscape: Every Bot That Actually Matters

Most AI providers now run at least two bots that do opposite jobs, and each needs its own explicit rule. Here is the current landscape as of 2026:

A few details are worth knowing before touching your own configuration:

• ChatGPT-User is not a bulk crawler. Per OpenAI's official documentation, GPTBot and OAI-SearchBot are controlled independently, so a site can allow one for search inclusion while disallowing the other for training; robots.txt changes can take roughly a day to be reflected in search behavior. ChatGPT-User, separately, fetches a single page only when a live user's prompt requires it.
• Anthropic runs three named bots. Per Anthropic's official crawler documentation, ClaudeBot collects training data, Claude-User fetches pages for live questions, and Claude-SearchBot indexes for Claude's search feature. All three honor robots.txt, including the non-standard Crawl-delay directive, and Anthropic states that blocking Claude-SearchBot may reduce a site's visibility in search results.
• PerplexityBot is a pure retrieval crawler with no separate training bot, and the least predictable of the major providers: Perplexity has stated its user-initiated agent is "not a bot" in the traditional sense and therefore not bound by robots.txt the same way, a position that has produced real disputes with publishers.
• Google-Extended is the cleanest opt-out lever in the stack. It controls inclusion in Gemini and AI Overviews training without touching Googlebot or Search rankings, making it one of the few decisions here with no real downside either way.

A Working robots.txt Configuration for 2026

This configuration allows the search and retrieval bots that drive AI citations to make an explicit choice on training crawlers and leaves standard search engines untouched. Adjust the training-bot section to match your own content policy.

# Allow AI search and retrieval crawlers (citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /

# Training crawlers: allow if you want your content
# used in future model training
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /

# Optional: block low-value bulk training crawlers
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /

# Standard search engines, unchanged
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Whichever posture you choose, review it at least quarterly. New user-agents get introduced regularly, Anthropic split Claude-SearchBot out of ClaudeBot in 2025, and a robots.txt copied from an older template is consistently the top finding in technical GEO audits at sites that quietly lost AI citations without anyone noticing.

A Worked Example: Auditing a Mid-Market SaaS Robots.txt

Here's how this plays out in practice. A mid-market B2B SaaS site comes in for a technical GEO audit with a robots.txt file last touched in 2023, during the early wave of "block everything AI" panic: a blanket Disallow: / under User-agent: GPTBot, and no rules at all for OAI-SearchBot, ChatGPT-User, or any Anthropic or Perplexity bot.

The diagnosis: the training block is doing almost nothing useful, since GPTBot had already crawled most of the site's public content before the rule was added. What it is doing is sitting next to a total absence of policy for the crawlers that actually drive citations, and depending on the CDN's default AI-bot handling, some of those may already be silently blocked at the edge too.

The fix follows the template above: explicit Allow rules for every search and retrieval bot, an explicit decision on each training bot, and a Sitemap directive that was also missing. Industry benchmarking from CapstonAI's Q1 2026 cohort study puts the average cost of each blocked major AI bot at 18 to 34 percent of that engine's potential citations, and separately, research from Rutgers Business School and The Wharton School found that publishers who block AI crawlers outright see a real decline in total traffic, without reliably reducing how often their content is still referenced elsewhere. Neither number is a guarantee for any specific site, but together they explain why an unaudited robots.txt is one of the highest-leverage two-hour fixes in a GEO program.

Building Your llms.txt, Step by Step

1. Audit your best pages. List the 10 to 20 pages that most precisely represent what your company does. This is not your sitemap; resist the urge to include everything you've ever published.
2. Write a one-sentence brand summary. This becomes the blockquote at the top of the file and often becomes a model's working understanding of who you are, so make it accurate rather than promotional.
3. Group links under clear H2 sections. Documentation, product pages, pricing, and key blog content each deserve their own section rather than one undifferentiated list.
4. Write descriptions for context, not for keywords. "Explains our pricing tiers and what each includes" is more useful to a model deciding what to fetch next than a marketing tagline stuffed with target keywords.
5. Publish at the domain root as llms.txt, using that exact filename and path. Subdirectory paths are not recognized by the specification.
6. Update it quarterly, or whenever you publish major new content or restructure the site. A stale file pointing at deleted pages signals an unmaintained site to anything that reads it.

How This Differs by Site Type

The formula above doesn't change, but how much weight to put on each piece does, depending on what kind of site you're running.

Developer tools and documentation platforms

This is where llms.txt earns its keep today. Coding agents fetch it routinely, and a well-structured llms-full.txt can meaningfully improve how accurately an AI assistant answers questions about your API. Robots.txt strategy here should favor allowing search and retrieval bots broadly, since developer audiences are exactly the users most likely to ask an AI assistant about the product directly.

Marketing and content sites (Lureon's core client base)

Here, robots.txt configuration does all of the work essentially, and llms.txt is a low-cost hedge rather than a priority. Effort is better spent auditing crawler access, fixing internal linking so citation-worthy pages are easy to find, and tracking share of voice against competitors, over polishing a curated file with speculative upside. See our guide on tracking AI search visibility across ChatGPT, Perplexity, Gemini, and Claude for how to measure whether the access work is translating into actual citations.

E-commerce and high-volume publishers

Crawl budget becomes a real constraint at scale, since training crawlers can generate meaningful server load across millions of pages. A selective posture, allowing search and retrieval bots broadly while being more deliberate about training-bot access to low-value pages like filtered category views, tends to balance visibility against infrastructure cost better than an all-or-nothing rule.

Common Mistakes That Quietly Cost You Citations

Blocking the training bot but forgetting its search counterpart

The single most common finding in technical GEO audits. A site blocks GPTBot out of a general data-privacy instinct, never adds a rule for OAI-SearchBot, and then wonders why it never appears in ChatGPT Search. Each provider's bots need independent, explicit rules; there is no shortcut that covers both with one line.

Dumping the entire sitemap into llms.txt

The most common llms.txt implementation failure, and it defeats the purpose of curation. If every URL on the site is listed, the file offers no more guidance than the sitemap it's meant to improve on.

Relying on robots.txt alone for enforcement

Robots.txt is a voluntary request, not a lock. OpenAI, Anthropic, and Google are documented to honor it, but Perplexity's stance on its own user-initiated agent is contested, and some less transparent crawlers ignore it outright. Anything that must stay fully off-limits needs server-level or CDN-level enforcement too.

Forgetting the CDN's own AI-bot toggle, and auto-generating indexable duplicates

A correctly written robots.txt can still be silently overridden by a "block AI scrapers" setting left on at the CDN layer, since several providers now ship that as an opt-out default rather than an opt-in. Separately, llms.txt tooling that auto-generates a parallel .md copy of every page can create duplicate content at scale if left indexable. Keep those copies non-indexable, or serve them at a clearly canonical URL pointing back to the HTML original.

Verifying It Actually Works

Configuration without verification is a guess. Three checks confirm whether the access layer is actually functioning:

• Check server logs directly for requests carrying GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot in the user-agent string, confirming crawlers are visiting at all before worrying about whether they're citing you.
• Verify by IP range, not user-agent string alone. Headers can be spoofed; OpenAI and other major providers publish machine-readable IP ranges specifically so requests can be authenticated rather than trusted at face value.
• Track AI-referral traffic separately from organic search in analytics, filtering for referrers from chat.openai.com, claude.ai, and perplexity.ai. Combined with the log data above, this closes the loop from "crawler can access the page" to "crawler is actually sending people here."

Conclusion

llms.txt and robots.txt solve different problems, and it's easy to spend disproportionate effort on the one with the smaller, less certain payoff. llms.txt is a genuinely useful, low-cost addition, especially for developer-facing products, but it has no confirmed effect on AI search citations as of 2026. Robots.txt is the opposite: unglamorous, easy to get wrong, and directly responsible for whether ChatGPT, Claude, and Perplexity can ever read your content in the first place.

Start with an audit of what's currently blocked, fix the training-versus-search split explicitly for every provider, and only then invest time in a curated llms.txt. Getting the access layer right doesn't guarantee a citation, but getting it wrong guarantees you won't get one, no matter how good the content behind it is. Once access is confirmed, the next question is whether that visibility is actually producing business results, which is where calculating ROI from GEO investment comes in.

If you'd rather have this audited and configured for you.

FAQs

1. Does llms.txt improve my ranking in ChatGPT or Perplexity?

Not directly, and not as of 2026. Its clearest, best-documented value today is guiding coding agents and developer tools, not influencing AI search citations, where no major provider has confirmed it as a ranking factor.

2. Can blocking one bot silently remove me from ChatGPT or Claude answers?

Yes. Blocking OAI-SearchBot while allowing GPTBot, or the reverse, is enough on its own to remove a site from that specific product's live answers, since the two bots serve entirely separate purposes and neither rule affects the other.

3. Do I need llms-full.txt in addition to llms.txt?

Only if you run a documentation-heavy platform where handing an agent complete page text in one fetch genuinely helps. Most marketing and content sites need only the standard curated file.

4. Is robots.txt enough to guarantee a bot won't access my content?

No. It's a voluntary, industry-standard request. Major providers like OpenAI and Anthropic are documented to honor it, but some crawlers, including parts of Perplexity's infrastructure, have disputed whether it applies to them, so anything that must stay fully off-limits needs server-level enforcement too.

5. How often should I review my AI crawler configuration?

At least quarterly. New user-agents and split bots get introduced regularly, and an outdated robots.txt is consistently one of the most common, highest-impact findings in technical GEO audits.