AI crawlers are the bots AI companies use to fetch web pages, either to build search indexes for AI answers or to collect training data.
Search-facing crawlers include OAI-SearchBot (ChatGPT search) and PerplexityBot. Training controls include GPTBot and Google-Extended. Google's AI features rely on Googlebot, and Copilot relies on Bingbot. Blocking a search crawler by accident removes a site from that engine's answers, so robots.txt and CDN bot rules are among the first checks in any audit.