Meta Robots Auditor

What it does

Cross-checks a set of pages against their meta robots tag, x-robots-tag header, and robots.txt rules, then compares that against what the page's URL pattern and content suggest it should be (indexable, or deliberately excluded). Surfaces the mismatches, not the full expected list.

Install

This skill isn't packaged in an installable registry. It's a plain SKILL.md file you can copy directly. Copy it from the SKILL.md tab and save it at the path for your assistant:

Use it in your AI assistant

Claude Code

.claude/skills/meta-robots-auditor/ (or ~/.claude/skills/meta-robots-auditor/ for all projects)

GitHub Copilot (VS Code)

.github/skills/meta-robots-auditor/

OpenAI Codex

.codex/skills/meta-robots-auditor/

Cursor

.cursor/skills/meta-robots-auditor/ (or ~/.cursor/skills/meta-robots-auditor/ globally)

Compatibility

Any WordPress site. Robots directives are read from the active SEO plugin's per-page settings and the site's robots.txt file.

The complete SKILL.md for this skill:

---
name: meta-robots-auditor
description: Use when a user suspects important pages are accidentally noindexed, nofollowed, or blocked from crawling on their WordPress site. Triggers on "why isn't this page indexed", "noindex check", "robots.txt audit", or "accidentally blocked pages".
---

# Meta Robots Auditor

## When to use
The user has pages that should be indexable but might have been accidentally excluded via a noindex tag, a nofollow directive, or a robots.txt rule, often after a theme change, plugin update, or staging-to-live migration.

## Procedure
1. Get the list of pages in question (or the full site if none specified) and pull each one's meta robots tag, any x-robots-tag header, and applicable robots.txt rules.
2. Determine what each page's status should be, based on its URL pattern, content type, and whether it's the kind of page (search results, admin, cart) that's deliberately excluded elsewhere on the site.
3. Flag every mismatch: pages that should be indexable but are blocked, and, just as important, pages that should be excluded but aren't.
4. Group findings by likely cause: a global setting (whole post type noindexed), a leftover staging-site setting, or a one-off page-level mistake.
5. Recommend the fix at the level the cause lives at, a plugin setting change for a global issue, not a page-by-page patch.

## Guardrails
- Never bulk-change robots settings across a post type without confirming that type is actually meant to be public, some custom post types are deliberately excluded.
- Check robots.txt disallow rules in addition to meta tags, a page can look fine at the meta level while still blocked from being crawled at all.