MCP server

Robots.txt Policy Lint

dev.workers.pathwren.www/robots-policy-lint

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Working at last checkAbout 3,861 tokens per request once connected9 tools

Website

https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots

At a glance

Type: MCP server
Publisher: Not verified
Status: Working at last check
Tools: 9
Token cost: About 3,861 tokens per request
Sets up in: Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok
Last checked: 26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to Cursor Add to VS Code VS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http robots-policy-lint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.robots-policy-lint]
url = "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"

Gemini CLI

{"mcpServers":{"robots-policy-lint":{"httpUrl":"https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"}}}

Cursor

{"mcpServers":{"robots-policy-lint":{"url":"https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"}}}

VS Code

{"servers":{"robots-policy-lint":{"type":"http","url":"https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"}}}

Antigravity

agy mcp add robots-policy-lint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"robots-policy-lint","server_url":"https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"robots-policy-lint","server_url":"https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots"}

Tools

am_i_allowed ~595 tokens
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"am_i_allowed","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and the answer is what this host's own robots.txt says to YOU — the group that matches your user-agent, the line that decided it, that
audit_ai_access ~368 tokens
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and categ
check_path_allowed ~438 tokens
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the ru
diff_robots_txt ~320 tokens
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers '
example ~395 tokens
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server'
lint_robots_txt ~361 tokens
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string
merge_policy ~276 tokens
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-to
no_arguments_lint_this_hosts_robots_txt ~632 tokens
TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} to https://www.pathwren.workers.dev/mcp/robots — the answer is the full RFC 9309 lint of this host's OWN robots.txt and ai.txt, run for real on the bytes
whoami ~476 tokens
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/robots and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only

Browse

Related MCP servers

AI Crawler Index
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Crawler Log Triage
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
MCP Endpoint Lint - handshake, tool schemas, JSON-RPC errors
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Agent Discovery Doctor - llms.txt, agent card, mcp.json
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Crawler IP Verifier - real Googlebot or fake
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Markdown Lane Check - llms.txt, .md twins, Accept: text/markdown
Six checks: llms.txt, llms-full.txt, .md twins, Accept: text/markdown, sitemap.md, alt link. Free.
Alchemy Eth
Alchemy (Ethereum + L2) MCP.
Ai Visibility
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/dev.workers.pathwren.www/robots-policy-lint.svg)](https://aptlynx.net/servers/dev.workers.pathwren.www/robots-policy-lint)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents