MCP server

polite-crawler

io.github.mark1russell7/polite-crawler

Checked 26 Oct 2026, 08:31 UTC · Still published on its registry; checked against OSV.dev; never run by us

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Runs on your computer

Website Source

At a glance

Type: MCP server
Publisher: Not verified
Status: Runs on your computer
Sets up in: Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code
Last checked: 26 Oct 2026, 08:31 UTC

Connect it to your agent

Add to Cursor Add to VS Code VS Code Insiders

Opens the app, which shows the server and asks you to confirm. It then runs the package on your computer with your permissions. For other apps, copy the setup below.

Claude Code

claude mcp add polite-crawler -- npx -y polite-crawler-mcp@0.2.4

Claude Desktop

{"mcpServers":{"polite-crawler":{"command":"npx","args":["-y","polite-crawler-mcp@0.2.4"]}}}

Codex (config.toml)

[mcp_servers.polite-crawler]
command = "npx"
args = ["-y","polite-crawler-mcp@0.2.4"]

Gemini CLI

{"mcpServers":{"polite-crawler":{"command":"npx","args":["-y","polite-crawler-mcp@0.2.4"]}}}

Cursor

{"mcpServers":{"polite-crawler":{"command":"npx","args":["-y","polite-crawler-mcp@0.2.4"]}}}

VS Code

{"servers":{"polite-crawler":{"type":"stdio","command":"npx","args":["-y","polite-crawler-mcp@0.2.4"]}}}

Runs on your computer with your permissions: check the package and its source before installing.

Browse

Related MCP servers

Botcard
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Crawler Log Triage
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
stillindexed SEO checkers
Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
read_url
One public web page as clean markdown, with title, language and a SHA-256. Obeys robots.txt.
Citehound
Scan any domain for AI readiness and generate llms.txt, robots.txt and schema. Free, read-only.
Kliqo AI
Free AI visibility checks, then see if ChatGPT and Google AI name you or your competitors instead.
AI Crawler Access Check
Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.
Convertlyft
Web analytics that names the fix: funnels, replays, errors, page speed and SEO for your site.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.mark1russell7/polite-crawler.svg)](https://aptlynx.net/servers/io.github.mark1russell7/polite-crawler)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents