MCP server

Commoncrawl

io.github.pipeworx-io/commoncrawl

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read the…

Website Source

At a glance

Type

MCP server

Publisher

pipeworx.io (verified by the MCP Registry)

Status

Working at last check

Tools

45

Token cost

About 27,512 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http commoncrawl https://gateway.pipeworx.io/commoncrawl/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.commoncrawl]
url = "https://gateway.pipeworx.io/commoncrawl/mcp"

Gemini CLI

{"mcpServers":{"commoncrawl":{"httpUrl":"https://gateway.pipeworx.io/commoncrawl/mcp"}}}

Cursor

{"mcpServers":{"commoncrawl":{"url":"https://gateway.pipeworx.io/commoncrawl/mcp"}}}

VS Code

{"servers":{"commoncrawl":{"type":"http","url":"https://gateway.pipeworx.io/commoncrawl/mcp"}}}

Antigravity

agy mcp add commoncrawl https://gateway.pipeworx.io/commoncrawl/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"commoncrawl","server_url":"https://gateway.pipeworx.io/commoncrawl/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"commoncrawl","server_url":"https://gateway.pipeworx.io/commoncrawl/mcp"}

Tools

• commoncrawl_crawls ~233 tokens
  List the Common Crawl monthly crawl collections (crawl ids like "CC-MAIN-2026-34") with the date range each one covers. AUTHORITATIVE for "which Common Crawl snapshot covers <date>" — call this first to pick the crawl id that commoncrawl_index_search needs. Newest first. Keyless.
• commoncrawl_fetch_record ~400 tokens
  Read one archived page back out of Common Crawl by byte range — pass the filename, offset and length from a commoncrawl_index_search result and get the WARC headers, the captured HTTP response headers, and the page body as it was crawled. AUTHORITATIVE for "what did this page actually say when it wa
• commoncrawl_index_search ~576 tokens
  Search one Common Crawl collection's CDX index for archived captures of a URL, host, or whole domain. PREFER OVER WEB SEARCH when the question is "what did this page look like in <month>", "did Common Crawl ever see this URL", or "list the URLs crawled under this domain" — it returns the crawl recor

Browse

• GitHub tools
• Web scraping
• Web search
• GitHub and code hosting

Related MCP servers

• 19hz
  19hz.info MCP.
• Arquivo Pt
  Arquivo.pt MCP — full-text search over the Portuguese web archive.
• Crawlgraph
  CrawlGraph MCP — backlink intelligence on the public Common Crawl webgraph
• Seoapi Ccbot
  Validate CommonCrawl CCBot IP addresses. Remote MCP validate_ip tool.
• ExchangeRebel — pay-per-call APIs for AI agents (x402 on Solana)
  166 pay-per-call APIs for AI agents: market, company, dev + verification tools — x402 on Solana.
• Rankparse
  SEO MCP server — backlinks, domain authority, tech stack, and 18+ tools via Common Crawl.
• Gapup
  271 agent-payable tools: competitive intel, finance, KYC, compliance, ESG. x402 per-call.
• The Agent Commons
  Open public message board where AI agents read and post conversations. Remote MCP.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.pipeworx-io/commoncrawl.svg)](https://aptlynx.net/servers/io.github.pipeworx-io/commoncrawl)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents