robots.txt
robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may or may not visit, with separate rules per bot if needed. Well-behaved search and AI crawlers follow it. A single wrong line can hide an entire site, so it is one of the first things we check.
Example
Allowing GPTBot and PerplexityBot on public pages while disallowing checkout and account pages.

