The classic robots.txt mistake this prompt is built to catch: disallowing a URL pattern in robots.txt actually prevents Google from crawling it at all, which means it can never see a noindex tag placed on the same page. For faceted navigation at scale, that combination quietly leaves junk pages indexed while starving crawl budget from real product pages.
SEO
Audit a robots.txt file for crawl budget issues
A prompt that reviews a robots.txt line by line for rules that are silently wasting or restricting crawl budget on a large site.
Works with Claude / GPT830 uses★ 4.2
The prompt
seo-robots-txt-crawl-budget-audit
[yourdomain.com] is an e-commerce site with about 45,000 URLs (products, filtered category views, and blog posts combined) but Search Console shows Googlebot only crawls roughly 3,000 URLs/day, and many legitimate product pages sit in "Crawled, currently not indexed" for weeks. The team suspects the robots.txt, last edited two years ago by a developer who's since left, might be part of the problem. ROBOTS.TXT (paste full contents): [paste here] SITE SIZE: [~45,000 URLs, WooCommerce] KNOWN FACETED NAVIGATION PARAMETERS: [?color=, ?size=, ?sort=, ?filter=] Audit this line by line like someone who has to explain every rule's purpose to a client, not just skim it: 1. Go through each Disallow/Allow rule and state what it's actually blocking today, since rules written two years ago for a since-removed feature can silently block a current URL pattern nobody noticed. 2. Check whether faceted navigation parameters (?color=, ?size=, ?sort=) are blocked here, blocked via canonical tags instead, or not addressed at all, since unblocked faceted URLs at this site's scale can multiply the crawlable surface by 10-20x and starve budget from real product pages, that's the most likely explanation for the crawl numbers given. 3. Confirm the sitemap directive points to a current, valid sitemap URL, and that nothing in the file accidentally blocks the sitemap's own path or CSS/JS resources Google needs to render the page. 4. Distinguish rules that help crawl budget (blocking infinite filter combinations, internal search results pages, cart/checkout paths) from rules that are actively hurting it (blocking a legitimate product template by accident, overly broad wildcard patterns). 5. Propose a corrected robots.txt, and separately list which of the changes should instead be handled by canonical tags or noindex rather than a robots.txt disallow, since blocking a URL in robots.txt prevents Google from even seeing a noindex tag on it, a common mistake that keeps low-value pages indexed anyway.