llms.txt for SEO: Useful Signal or AI Search Myth?

llms.txt doesn’t boost your Google rankings. Google has confirmed, multiple times, that their search systems don’t read or act on this file. But here’s the part most articles skip: that’s not the whole story.

I’ve spent the last few weeks digging through research, crawling server logs, and talking to teams who track AI bot behavior at scale. What I found is more nuanced than the “llms.txt is useless” headline suggests. The file isn’t a ranking factor. But it might be something else entirely—something the SEO world hasn’t labeled correctly yet.

Let’s dig in.

What Is llms.txt, Really?

llms.txt is a Markdown file you place at your domain root (https://yoursite.com/llms.txt). Proposed by Jeremy Howard in September 2024, it gives AI systems a curated map of your site: who you are, what matters, and which pages carry the most signal.

The structure is simple:

• An H1 with your brand name
• A blockquote summary (2-3 sentences)
• H2 sections grouping related links
• Annotated links with one-line descriptions

Unlike robots.txt, llms.txt has no directives. It can’t block crawlers or restrict access. It’s pure navigation—a way to tell an AI “start here, then go here.”

Key stat: Only 10.13% of websites have implemented llms.txt, according to SE Ranking’s study of 300,000 domains. That’s roughly 1 in 10 sites.

The Research Reality: Does llms.txt Move the Needle?

Here’s what the data actually shows—and it’s not encouraging if you’re hoping for a quick GEO win.

The SE Ranking Study

You want evidence? SE Ranking analyzed nearly 300,000 domains. They found:

• 10.13% adoption rate
• No statistically significant correlation between llms.txt and AI citation frequency
• When researchers removed llms.txt from their XGBoost prediction model, accuracy improved

Translation: the file was adding noise, not signal.

The Limy Data

Limy monitored over 500 million AI bot traffic events across a 90-day window. Only 408 requests targeted /llms.txt directly. That’s statistically negligible.

For context, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended—the bots that drive AI citations—almost never fetch the file. They crawl HTML directly, same as they always have.

Google Says No (Repeatedly)

Google’s position is unambiguous:

• Gary Illyes confirmed in 2025 that Google doesn’t support llms.txt and has no plans to
• John Mueller compared it to the discredited keywords meta tag—technically visible, but nobody uses it for ranking
• Google’s Search team explicitly removed llms.txt from their developer documentation after it was initially added by another team

So why are people still talking about it?

The Actually Valid Use Cases

Strip away the GEO hype and llms.txt has one legitimate claim: it’s a B2A (Business-to-Agent) protocol, not a search ranking factor. Different layer, different value.

Here’s where it actually shows up:

Use Case	Current Status	Who Uses It
IDE agents (Cursor, Copilot, Claude Code)	Actively fetched	Developers using AI coding tools
MCP servers (Mintlify, LangChain)	Built around it	Documentation platforms
AI search / answer engines	Almost never	Perplexity, ChatGPT, Gemini crawlers
Agentic commerce	Emerging	Early e-commerce adopters

Cursor, Windsurf, and Claude Code all check for llms.txt when you ask product-specific questions. They use it to orient themselves before fetching your docs—reducing token waste and improving answer accuracy.

This is real value. It’s just not SEO value.

llms.txt vs Robots.txt: What’s the Difference?

A lot of confusion comes from treating llms.txt like it’s the AI version of robots.txt. It’s not.

	robots.txt	llms.txt
Purpose	Access control	Content guidance
Can block crawlers	Yes	No
Industry standard	W3C recognized	Community proposal
Enforced	By Google, Bing, all major crawlers	By nobody (officially)
SEO impact	Direct	None confirmed
Google’s position	Core infrastructure	Not used

You need both. They operate in different parts of the stack—robots.txt manages what crawlers can access, llms.txt manages what agents should prioritize.

llms.txt Format: Structure It Right

If you do implement llms.txt, structure it correctly. Here’s an annotated template:

# Your Brand Name

> One-sentence description of what you do and who you serve.

Optional paragraph of context if your brand name is ambiguous.

## Products
- [Product Name](https://yoursite.com/product): What this page covers and when to fetch it.

## Documentation
- [Getting Started](https://yoursite.com/docs/start): How to begin.
- [API Reference](https://yoursite.com/api): Full API documentation.

## Optional
- [Blog](https://yoursite.com/blog): Lower-priority pages agents can skip.

Four structural rules:

1. H1 with brand name (no taglines)
2. Blockquote summary immediately after
3. H2 sections group related links
4. Links follow exact format: - [Title](URL): Description.

The ## Optional section has semantic meaning—agents know to deprioritize it under token pressure.

Who Should (and Shouldn’t) Build It?

Site Type	Recommendation	Why
B2B SaaS with documentation	Build it now	High value for AI coding tools
Developer tools / API products	Build both files	llms.txt + llms-full.txt for docs
Content/blog sites	Build it	Low effort, improves AI brand comprehension
E-commerce stores	Low priority	Schema, product feeds deliver more value
Local service businesses	Skip for now	Google Business Profile is higher ROI
News publishers	Low priority	Content velocity makes maintenance painful

Common Mistakes to Avoid

1. Treating it like a sitemap. Don’t list every URL. Curation beats completeness—20-50 high-value links beats 500 noisy ones.

2. Writing vague descriptions. “Click here” and “Read more” give AI zero context. Be specific: what page, what it covers, when to fetch it.

3. Letting it go stale. Dead links signal an unmaintained site. Review quarterly, or whenever you restructure.

4. Auto-generating .md copies of every page. This creates duplicate content that can dilute your crawl budget and hurt rankings.

5. Including duplicate links. Pick one section per URL. Agents use the first match.

The Honest Answer: Should You Implement It?

Here’s my honest take after reviewing the data:

Today: If you’re implementing llms.txt to boost AI search citations or Google rankings, you’ll be disappointed. The data doesn’t support it.

Forthcoming: If you’re a developer tools company, SaaS with docs, or any brand where AI agents might research on behalf of users, llms.txt is cheap insurance. The agentic web is building toward this infrastructure. Being early costs you half a day of work.

The risk: Close to zero. A text file at your root, served publicly. No indexing implications, no ranking downside.

The upside: If (and that’s a conditional “if”) AI providers ever adopt the standard at scale, you’ll have infrastructure in place. You’ll also make your docs significantly more useful for the IDE agents already fetching it today.

I’ve seen teams spend weeks chasing llms.txt as a GEO silver bullet. I’ve seen zero of them get measurable citation lifts from it.

I’ve also seen teams ship it in an afternoon and have their developer docs cited correctly by Cursor for the first time.

The difference is context. llms.txt isn’t strategy. It’s infrastructure—and it’s infrastructure with a narrow, specific use case.

Frequently Asked Questions

Does llms.txt help with Google SEO?

No. Google has explicitly confirmed their search systems don’t read or use llms.txt. It’s not a ranking factor and won’t affect your visibility in traditional search or AI Overviews.

Can llms.txt improve my ChatGPT or Perplexity citation rate?

No—not based on current research. Limy’s study of 500M+ AI bot events found crawler interest in llms.txt was statistically negligible. No independent study has shown a correlation between llms.txt and improved AI citations.

What’s the difference between llms.txt and llms-full.txt?

llms.txt is a curated index (links and one-line descriptions). llms-full.txt embeds the full content of linked pages inline—useful for documentation-heavy sites where agents want everything in one fetch. Most sites only need llms.txt.

Does llms.txt replace robots.txt?

No. robots.txt controls crawler access; llms.txt provides navigation guidance. They complement each other. For AI-specific access control, use robots.txt with specific user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended).

Who actually fetches llms.txt today?

IDE agents like Cursor, Windsurf, Claude Code, and GitHub Copilot fetch it routinely when working with developer documentation. MCP servers built around Mintlify and LangChain use it as a routing layer. Traditional AI search crawlers? Almost never.

---

Sources

• llms-txt.org — Official Specification
• SE Ranking: LLMs.txt Research Study
• Limy.ai: LLMs.txt in 2026 — The Full Guide
• Ahrefs: What Is llms.txt, and Should You Care?
• Semrush: What Is LLMs.txt & Should You Use It?
• DerivateX: LLMs.txt Guide — What It Does and Doesn’t Do
• Search Engine Roundtable: Google Search Team Does Not Endorse LLMs.txt Files
• W3Era: Google Says LLMs.txt on Its Websites Is Not for Search Discovery