LLMS.txt: making your website easier for AI search

Search is changing: people increasingly ask assistants to “find and summarize”, not just “search and click”. If your public website is a SPA, has heavy JS, or hides the core content behind navigation and UI chrome, LLM-based tools can miss your best pages.

llms.txt is a small, human-readable file that gives AI systems a curated map of what matters on your site.

This article is an original summary and practical checklist inspired by the discussion around llms.txt (see Sources).

What is llms.txt?

llms.txt is a convention proposal: a plain-text file served from your site root at /llms.txt.

It’s not an official web standard (like robots.txt or sitemap.xml), but it’s a low-cost way to:

• Provide high-signal links (best pages to read first).
• Describe what your product is in a compact way.
• Share preferred citations (canonical URLs).
• Point to machine-friendly pages, docs, and policies.

Why it helps (even if not “standard”)

AI crawlers and assistants have limited context windows. If they start from your homepage and it’s mostly UI, they waste budget and miss details. A curated file:

• Reduces “wandering” across irrelevant pages.
• Avoids indexing private areas (/app, /admin) as “product content”.
• Helps assistants answer accurately with fewer hallucinations.

What to include in llms.txt

A good starting template:

1. Short product description

• One paragraph describing what you do and for whom.

2. Key pages

• Your best landing pages, feature overviews, pricing, contact, demo booking.

3. Articles / resources

• A blog index and a few cornerstone posts (high-level + deep dives).

4. Crawling helpers

• Link to /sitemap.xml and /robots.txt.

5. Contact

• A support email (or a sales email) for clarification.

Don’t forget the basics

llms.txt is not a substitute for classic SEO hygiene:

• Server-render or prerender public landing pages so the content exists in HTML.
• Ensure each page has a unique title, description, and canonical URL.
• Add JSON-LD (Organization, WebSite, Product/Service, FAQ, Breadcrumbs) where relevant.
• Keep private areas noindex and blocked in robots.txt if appropriate.

Practical checklist (for SPAs)

• Prerender SEO pages (home, features, solutions, pricing, contact).
• Generate a sitemap and keep it up-to-date.
• Add llms.txt with a curated set of URLs.
• Add an “Articles” section and publish a few clear, factual posts.

Sources

• Niall McNulty, “The llms.txt: Making your website AI-friendly” (Medium): https://medium.com/@niall.mcnulty/the-llms-txt-making-your-website-ai-friendly-85edd0008bf3