Groups and rules
Site: https://www.oakandloom.example. Crawler tested: Googlebot. File size: 1.2 KB, under Google's 500 KiB read limit.
| Rule | Group | Line | Supported by Google |
|---|---|---|---|
| R1 | * | Disallow: /admin/ | yes |
| R2 | * | Disallow: /cart | yes |
| R3 | * | Disallow: /checkout | yes |
| R4 | * | Disallow: /search | yes |
| R5 | * | Disallow: /assets/ | yes |
| R6 | * | Allow: /assets/img/ | yes |
| R7 | * | Disallow: /*?sort= | yes |
| R8 | * | Crawl-delay: 10 | no, Google ignores crawl-delay |
| R9 | * | Noindex: /tag/ | no, Google does not support noindex in robots.txt |
| R10 | Googlebot-Image | Disallow: / | yes |
| R11 | (any) | Sitemap: https://www.oakandloom.example/sitemap.xml | yes |
Group choice: the file has no Googlebot group, so Googlebot follows the * group (R1 to R9). Googlebot-Image (Google's image crawler token, per https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) matches its own group (R10), and only one group is valid for a crawler, so it follows R10 only.
URL test table
Rule: the matching rule with the longest path wins. On a tie between allow and disallow, Google uses the least restrictive rule (allow). Path lengths are in characters, computed by script. Your private list: /admin/, /cart, /checkout, /search.
| URL (what it is) | Matching rules (path length) | Deciding rule | Result |
|---|---|---|---|
| /sofas/harbor-3-seat (product page) | none | none | Crawl allowed |
| /assets/css/main.css (CSS) | R5 /assets/ (8) | R5 | Blocked |
| /assets/js/app.js (JS) | R5 /assets/ (8) | R5 | Blocked |
| /assets/img/harbor-3-seat.jpg (image) | R5 /assets/ (8), R6 /assets/img/ (12) | R6, since 12 > 8 | Crawl allowed for Googlebot |
| /sofas?sort=price (sorted listing) | R7 /*?sort= (8) | R7 | Blocked, not on your private list |
| /searching-for-the-perfect-sofa (blog post) | R4 /search (7) | R4 | Blocked |
| /cart (cart) | R2 /cart (5) | R2 | Blocked, on your private list |
URLs tested: 7. Crawl allowed: 2. Blocked: 5, of which 1 is on your private list (/cart) and 4 are flagged (counted by script).
Problems
- CSS and JS blocked (R5). Googlebot cannot fetch /assets/css/main.css or /assets/js/app.js, which your pages load.
- Blog post blocked by the search rule (R4). "Disallow: /search" matches any path that starts with "/search", so it also blocks /searching-for-the-perfect-sofa. Paths are case sensitive and matched from the start.
- Google Images blocked (R10). "User-agent: Googlebot-Image / Disallow: /" stops Google's image crawler from crawling any URL, including /assets/img/harbor-3-seat.jpg. R6 does not help, because Googlebot-Image follows only its own group. Do you want this block? I have kept R10 in the proposed file until you answer.
- Ignored lines (R8, R9). Google does not support crawl-delay or noindex in robots.txt. To keep /tag/ pages out of the index, use a noindex robots meta tag on those pages and let them be crawled. Robots.txt controls crawling, not indexing, and a disallowed URL can still be indexed if it is linked from elsewhere.
- Sorted listing blocked (R7). "Disallow: /*?sort=" blocks /sofas?sort=price, and sort pages are not on your private list. Do you want sorted listings kept out of crawling? I have kept R7 in the proposed file until you answer.
Counted from the list: 5 problems, covering 6 rules (R4, R5, R7, R8, R9, R10).
Proposed edit
Smallest change: remove R5, R6, R8 and R9; replace R4 with three narrower rules. Two items wait on your answers and stay in the file unchanged, each marked with a # comment (Google ignores everything after # on a line): R7 (problem 5) and the Googlebot-Image group R10 (problem 3). Delete a held block only after you confirm.
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /search$
Disallow: /search?
Disallow: /search/
# HELD: R7 kept until you answer problem 5 (sorted listings)
Disallow: /*?sort=
# HELD: R10 kept until you answer problem 3 (Google Images)
User-agent: Googlebot-Image
Disallow: /
Sitemap: https://www.oakandloom.example/sitemap.xml
Please confirm your internal search URL format (for example /search?q=sofa). The three search rules assume /search, /search? or /search/.
Re-test
Tested against the file exactly as proposed (matching computed by script).
| URL | Crawler | Matching rules in the proposed file (length) | Result |
|---|---|---|---|
| /sofas/harbor-3-seat | Googlebot | none | Crawl allowed |
| /assets/css/main.css | Googlebot | none | Crawl allowed |
| /assets/js/app.js | Googlebot | none | Crawl allowed |
| /assets/img/harbor-3-seat.jpg | Googlebot | none | Crawl allowed |
| /assets/img/harbor-3-seat.jpg | Googlebot-Image | / (1), held R10 | Blocked, held for problem 3 |
| /sofas?sort=price | Googlebot | /*?sort= (8), held R7 | Blocked, held for problem 5 |
| /searching-for-the-perfect-sofa | Googlebot | none | Crawl allowed |
| /cart | Googlebot | /cart (5) | Blocked, on your private list |
| /search?q=sofa (extra check, example format) | Googlebot | /search? (8) | Blocked, on your private list (/search) |
Rows: 9. Crawl allowed: 5. Blocked: 4 (2 held, 2 on your private list), counted by script.
Method notes
- Google generally caches robots.txt for up to 24 hours, so changes may take about a day to be picked up.
- If robots.txt returns a 4xx (except 429), Google treats it as if no robots.txt existed. For a 5xx, RFC 9309 says crawlers must assume complete disallow (https://www.rfc-editor.org/rfc/rfc9309.html), so make sure the new file returns 200.
- Matching used Google's rules (longest path, ties go to the least restrictive rule, * and $ as wildcards), checked by script. Nothing here says which URLs are indexed.
Needs your decision
- Approve the new file before it replaces the current one.
- Confirm whether Google Images should be blocked (R10, held in the file).
- Confirm whether sorted listings should be blocked (R7, held in the file).
- Confirm the internal search URL format.