I can review this, but I need your goals before I write any new file, and I will not push anything live.
- Your goal per crawler (required). For each AI crawler you care about, tell me allow or block, and which paths. The choice is yours; I will not decide it. Why it matters: OpenAI's crawler page says "OAI-SearchBot is for search" and "Sites that are opted out of OAI-SearchBot will not be shown" in ChatGPT search results, while GPTBot is a separate setting ("Each setting is independent of the others"). Google describes Google-Extended as its own robots.txt product token. So "block AI bots" can mean very different files.
- Full robots.txt (required). You pasted "User-agent: * Disallow: /admin". Please confirm that is the whole file, line by line, including any Sitemap line.
On "it hurts traffic": my sources do not say how blocking affects your traffic or rankings, so I will not claim it does or does not. I can only show what each token's rules would be.
Once I have both, I will check each token against your goals and write a proposed file, held for your approval. Editing or publishing robots.txt waits for your OK on that exact file.
