AI crawler
An AI crawler is a bot that visits web pages on behalf of an AI company, either to gather training data or to fetch live pages while answering. Examples include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. If your robots.txt, firewall or CDN blocks them, assistants may be unable to read or cite your site.
Example
A security setting that silently blocks PerplexityBot can keep your pages out of Perplexity answers.

