AI and answer-engine readiness
ChatGPT, Claude, Perplexity and Google's AI Overviews increasingly answer questions directly rather than sending a click. Being readable to them is becoming as important as ranking.
How it differs from classic SEO
Search crawlers render JavaScript. Most AI assistants fetching a page do not. If your content only appears after scripts run, an assistant frequently sees an empty shell.
Assistants also favour explicit, machine-readable facts. Structured data and clear headings do more work here than keyword placement ever did.
What to check
Whether the content is present in the raw HTML. View source, and if the text is not there, an assistant probably cannot read it.
Whether the page uses semantic landmarks such as main, header and nav, which separate the content from the navigation.
Which AI crawlers you allow. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others can be permitted or blocked in robots.txt. This is a commercial decision rather than a technical fault: blocking them protects your content from training use, but also removes you from answers those tools generate.
How to fix it
- Serve your main content in the initial HTML response rather than rendering it client side.
- Wrap the primary content in a main element, and use header, nav and footer around it.
- Add structured data describing the business, and FAQPage markup for genuine questions.
- Decide deliberately which AI crawlers to allow, and record that decision in robots.txt rather than leaving it to chance.
- Consider publishing an llms.txt file summarising the site. The convention is still emerging, and it costs very little to provide.
What good looks like
# Allow AI assistants to read the site
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Or decline training use while staying in search
User-agent: Google-Extended
Disallow: /Common mistakes
- Building the entire site as a client-rendered application with no server-rendered content.
- Blocking every AI crawler by reflex, without considering the referral traffic being given up.
- Assuming an existing search ranking guarantees an assistant can read the page.
Check your own site. ppclift checks this automatically on every page it audits. Run a free audit and see every check on one page, with no account required.