Skip to content
Start trialSee plans

Technical

AI crawler

An AI crawler is a bot that visits web pages on behalf of an AI company, either to gather training data or to fetch live pages while answering. Examples include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. If your robots.txt, firewall or CDN blocks them, assistants may be unable to read or cite your site.

Example

A security setting that silently blocks PerplexityBot can keep your pages out of Perplexity answers.

Next step

Let's switch your lights on.

Start any plan with a 7-day full refund, or see how AI sees you first with a free audit.