MCP server

Robots.txt & AI Crawler Checker

dev.workers.quietworklabs.robots-check/robots-check

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Website

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

3

Token cost

About 610 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.robots-check]
url = "https://robots-check.quietworklabs.workers.dev/mcp"

Gemini CLI

{"mcpServers":{"robots-check":{"httpUrl":"https://robots-check.quietworklabs.workers.dev/mcp"}}}

Cursor

{"mcpServers":{"robots-check":{"url":"https://robots-check.quietworklabs.workers.dev/mcp"}}}

VS Code

{"servers":{"robots-check":{"type":"http","url":"https://robots-check.quietworklabs.workers.dev/mcp"}}}

Antigravity

agy mcp add robots-check https://robots-check.quietworklabs.workers.dev/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"robots-check","server_url":"https://robots-check.quietworklabs.workers.dev/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"robots-check","server_url":"https://robots-check.quietworklabs.workers.dev/mcp"}

Tools

• ai_crawler_access ~200 tokens
  AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose,
• robots_check ~226 tokens
  Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line
• sitemap_check ~184 tokens
  Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Determini

Browse

• Claude tools
• Web scraping

Related MCP servers

• Spook SEO
  Free SEO tools + the Spook SEO autopilot: analyze domains, generate and publish SEO articles.
• stillindexed SEO checkers
  Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
• SEO Tools
  5 pay-per-call SEO tools over MCP. Free discovery, tool calls settle in USDC on Base via x402.
• Seegeo
  Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check.
• Webbersites X402 Data Api
  45 pay-per-call AI agent tools: scraping, SEO, crypto data, lint, agent memory. x402 USDC on Base.
• Rankwise
  AI search visibility: see if ChatGPT, Perplexity, Claude and Gemini cite a brand, then win it back.
• Agent Harbor pay-per-call tools
  Pay-per-call company, SEC, recall, package and domain lookups, PDF to markdown, site audits (x402)
• CitedBy
  Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/dev.workers.quietworklabs.robots-check/robots-check.svg)](https://aptlynx.net/servers/dev.workers.quietworklabs.robots-check/robots-check)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents