MCP server

Crawl Census

io.github.taylorsmithgg/crawl-census

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Website Source

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

7

Token cost

About 1,253 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.crawl-census]
url = "https://crawlcensus.com/mcp"

Gemini CLI

{"mcpServers":{"crawl-census":{"httpUrl":"https://crawlcensus.com/mcp"}}}

Cursor

{"mcpServers":{"crawl-census":{"url":"https://crawlcensus.com/mcp"}}}

VS Code

{"servers":{"crawl-census":{"type":"http","url":"https://crawlcensus.com/mcp"}}}

Antigravity

agy mcp add crawl-census https://crawlcensus.com/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"crawl-census","server_url":"https://crawlcensus.com/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"crawl-census","server_url":"https://crawlcensus.com/mcp"}

Tools

• agent_profile ~195 tokens
  What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of tha
• census_facts ~157 tokens
  Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the w
• census_stats ~58 tokens
  Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
• crawl_preflight Browses the web · ~399 tokens
  Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits
• scan_site ~119 tokens
  Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
• site_report ~83 tokens
  Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
• submit_domains ~242 tokens
  Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a

Browse

• GitHub tools
• Web scraping
• Website monitoring
• Payments
• GitHub and code hosting

Related MCP servers

• AstraVerify Domain Trust Check
  Email security and AI discoverability scores for any domain, with the exact record or tag to fix.
• Fetchgate
  Fetch any URL as clean Markdown or metadata, and buy digital goods via x402 — for AI agents.
• Rightshold
  License checks, signed rights proof, keyless pre-call checks for MCP servers, and the agent store
• CrawlCheck
  Verification layer for the agentic web: a signed answer about any site before an agent acts.
• ContrastAPI
  55 tools, 7 Resources, Sigma rules, email SPF/DMARC, MITRE, CVE/KEV, risk_score. No key.
• Setu.Anantasutra: Agent Tools API
  Pay-per-call (x402) tools: Indian ID checks, AI-visibility audits, AI crawler and llms.txt checks
• TunnelMind Data API
  Tracker / Sigil / Cross-lens — every TunnelMind Data API operation as one MCP surface.
• Zenrows
  Zenrows MCP server: Fetch, Extract, Batch, and Browser Sessions for AI agents

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.taylorsmithgg/crawl-census.svg)](https://aptlynx.net/servers/io.github.taylorsmithgg/crawl-census)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents