llms.txt and structured content

AI tools like ChatGPT, Perplexity and Google’s AI Overviews are increasingly the first place people go looking for answers — including answers about your products, services and industry. Whether your site gets used as a source depends on whether an AI system can actually parse and trust your content. That’s a different problem to traditional SEO, and it needs a different checklist. We’ve written before about why your website needs to be ready for AI search; this post is the practical follow-up.

One file has become the poster child for this: llms.txt. It’s worth understanding what it does, what it doesn’t do, and where your effort is genuinely better spent.

What llms.txt actually is

llms.txt is a proposed convention — not an official web standard — introduced by Jeremy Howard of Answer.AI in 2024, and revised since. It’s a simple Markdown file placed at yoursite.com/llms.txt that gives a condensed, AI-friendly summary of a site: what it is, and links to its most important pages, described in plain language rather than the navigation-and-marketing-copy structure most websites use for humans.

The idea is sound in principle: large language models have limited context windows, and a curated summary is easier for a model to digest than crawling and parsing a whole site. Several major AI companies — including OpenAI, Anthropic and Google — publish llms.txt files on their own developer documentation sites.

The catch: adoption isn’t confirmed

Here’s the part that gets left out of a lot of the enthusiastic guides: no major AI crawler has confirmed it reads llms.txt files from third-party websites. The fact that OpenAI or Anthropic publish one for their own docs is a courtesy to visitors and developers, not evidence that their models fetch and use other people’s llms.txt files when browsing or answering questions.

Google has been the most direct about this. Its guidance states plainly that you don’t need special AI markup or an llms.txt file to gain visibility in Google’s generative Search features, and that llms.txt files won’t harm — but also won’t help — your visibility, because Google Search ignores them.

That doesn’t mean llms.txt is worthless. It costs almost nothing to add, it may be adopted more meaningfully as the ecosystem matures, and building one forces you to write a clear, jargon-free summary of your own site — a useful exercise regardless of whether any crawler reads the file. But it shouldn’t be the centrepiece of an “AI visibility” strategy, and anyone promising it will get you cited in ChatGPT overnight is overselling it.

What actually makes a site legible to AI systems

The more durable work is making your existing content genuinely easy for any crawler — human-built or AI — to parse, extract and trust. That’s less about a single file and more about how your pages are built.

Write in clean, semantic HTML. Proper heading hierarchy (one H1, logically nested H2s and H3s), real <ul>/<ol> lists instead of visually-styled divs, and tables marked up as tables. AI crawlers, like search engines, rely on structure to understand what’s a heading, what’s a list, and what’s incidental page furniture like navigation and footers.

Answer the question early and directly. AI systems that summarise or quote your content favour pages that state the key fact or answer plainly, near the top, rather than pages that bury the answer under three paragraphs of scene-setting. This is good writing practice anyway.

Keep content in text, not just images or JavaScript-rendered widgets. If a crawler has to execute JavaScript or run OCR on an image to find your pricing, opening hours or key facts, some crawlers will simply skip it. Server-rendered or statically-generated HTML — which is how we build on Craft CMS and Next.js — gives every kind of crawler a fair shot.

Use structured data where it’s genuinely warranted — Organization, Product, Article and similar schema.org markup — but treat it as a description of content that’s already there, not a substitute for it. More on which schema types are actually worth your time in our companion post on schema markup in 2026.

Make one page the canonical answer. If you’ve got the same information spread thinly across five pages, an AI system is more likely to synthesise it (sometimes inaccurately) than find and cite one clear source. Consolidating fragmented content into a single authoritative page helps both readers and machines.

Keep a fast, crawlable sitemap and clean URL structure. Nothing exotic here — this is the same technical hygiene that’s always mattered for SEO, and it still matters for AI crawlers, most of which reuse the same crawling infrastructure as traditional search.

A practical checklist

1. Add a /llms.txt file with a concise, plain-language summary and links to your most important pages — cheap to do, low risk, genuinely useful as a forcing function even if no crawler reads it yet.
2. Audit your top 20 pages for heading structure — one H1, sensible H2/H3 nesting.
3. Confirm key facts (pricing, hours, specs, answers to common questions) are in real HTML text, not locked inside images or client-side-only JavaScript.
4. Identify duplicate or fragmented content covering the same topic and consolidate into one canonical page.
5. Add Organization and Product/Service schema where relevant — see our schema markup guide for what’s actually worth doing.
6. Check your sitemap.xml is current and your robots.txt isn’t accidentally blocking AI crawlers you want visiting.
7. Re-read your key pages and ask: if someone pasted this paragraph into an AI chatbot with no other context, would they understand it?

The takeaway

Legibility to AI crawlers isn’t a separate discipline from good web fundamentals — it’s an extension of them. llms.txt is worth the ten minutes it takes to add, but it’s not a shortcut past the harder, more valuable work: clean markup, direct writing, real HTML text, and a site structure that doesn’t make either humans or machines guess where the answer is.

If you’d like us to audit your site’s AI and search legibility as part of a wider technical review, get in touch — it’s a natural companion to the accessibility and performance audits we already run for clients.