MCP server

Robots Txt Ai Crawler Audit

io.github.jmshinhwa/robots-txt-ai-crawler-audit

Checked 28 Oct 2026, 09:04 UTC · Deprecated by its author (its registry); checked against OSV.dev; never run by us

Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI

Runs on your computerDeprecated by its author: Not affiliated with ReadyStack (readystack.dev). No new versions are published here.

Source

At a glance

Type: MCP server
Publisher: Not verified
Status: Runs on your computer
Sets up in: Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code
Last checked: 28 Oct 2026, 09:04 UTC

Connect it to your agent

Add to Cursor Add to VS Code VS Code Insiders

Opens the app, which shows the server and asks you to confirm. It then runs the package on your computer with your permissions. For other apps, copy the setup below.

Claude Code

claude mcp add robots-txt-ai-crawler-audit -- npx -y @readystack/robots-txt-ai-crawler-audit@0.1.3

Claude Desktop

{"mcpServers":{"robots-txt-ai-crawler-audit":{"command":"npx","args":["-y","@readystack/robots-txt-ai-crawler-audit@0.1.3"]}}}

Codex (config.toml)

[mcp_servers.robots-txt-ai-crawler-audit]
command = "npx"
args = ["-y","@readystack/robots-txt-ai-crawler-audit@0.1.3"]

Gemini CLI

{"mcpServers":{"robots-txt-ai-crawler-audit":{"command":"npx","args":["-y","@readystack/robots-txt-ai-crawler-audit@0.1.3"]}}}

Cursor

{"mcpServers":{"robots-txt-ai-crawler-audit":{"command":"npx","args":["-y","@readystack/robots-txt-ai-crawler-audit@0.1.3"]}}}

VS Code

{"servers":{"robots-txt-ai-crawler-audit":{"type":"stdio","command":"npx","args":["-y","@readystack/robots-txt-ai-crawler-audit@0.1.3"]}}}

Runs on your computer with your permissions: check the package and its source before installing.

Browse

Related MCP servers

Storefront Check
Audit any storefront from outside: ad tags, dead-tag detection, AI crawler access, feed readiness.
Botcard
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Seegeo
Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check.
AIReady
Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
Link Followability
Check if a link on a live page is followable: rel tokens, meta robots and X-Robots-Tag
AI Crawler Access Check
Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.
Robots.txt & AI Crawler Checker
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
RobotsGate
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.jmshinhwa/robots-txt-ai-crawler-audit.svg)](https://aptlynx.net/servers/io.github.jmshinhwa/robots-txt-ai-crawler-audit)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents