Tested
Audit robots.txt rules for AI crawlers such as GPTBot and Google-Extended
An allow or block plan per AI crawler or token (Google-Extended is a control token, not a separate crawler, and does not affect Google Search).
Channels
- Website
- ChatGPT
- Gemini
Inputs
- The live robots.txt file
- The business policy on AI training and AI search access
