Real run

AI Crawler robots.txt Audit

Sample 2 · 2026-10-29 · Grok Bot executor subagent (model family not disclosed)

Bot page

I can review this, but I need your goals before I write any new file, and I will not push anything live.

  1. Your goal per crawler (required). For each AI crawler you care about, tell me allow or block, and which paths. The choice is yours; I will not decide it. Why it matters: OpenAI's crawler page says "OAI-SearchBot is for search" and "Sites that are opted out of OAI-SearchBot will not be shown" in ChatGPT search results, while GPTBot is a separate setting ("Each setting is independent of the others"). Google describes Google-Extended as its own robots.txt product token. So "block AI bots" can mean very different files.
  2. Full robots.txt (required). You pasted "User-agent: * Disallow: /admin". Please confirm that is the whole file, line by line, including any Sitemap line.

On "it hurts traffic": my sources do not say how blocking affects your traffic or rankings, so I will not claim it does or does not. I can only show what each token's rules would be.

Once I have both, I will check each token against your goals and write a proposed file, held for your approval. Editing or publishing robots.txt waits for your OK on that exact file.