Skip to content
Start trialSee plans

Technical

robots.txt

robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may or may not visit, with separate rules per bot if needed. Well-behaved search and AI crawlers follow it. A single wrong line can hide an entire site, so it is one of the first things we check.

Example

Allowing GPTBot and PerplexityBot on public pages while disallowing checkout and account pages.

Next step

Let's switch your lights on.

Start any plan with a 7-day full refund, or see how AI sees you first with a free audit.