MCP server

Agent Crawl Readiness

io.github.sadri-dridi/agent-crawl-readiness

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Website Source

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

31

Token cost

About 9,894 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.agent-crawl-readiness]
url = "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"

Gemini CLI

{"mcpServers":{"agent-crawl-readiness":{"httpUrl":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}

Cursor

{"mcpServers":{"agent-crawl-readiness":{"url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}

VS Code

{"servers":{"agent-crawl-readiness":{"type":"http","url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}

Antigravity

agy mcp add agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"agent-crawl-readiness","server_url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"agent-crawl-readiness","server_url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}

Tools

• agent-crawl-readiness Browses the web · ~388 tokens
  Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage HTML (Agent Ads). Pass website=<https://domain> to run the bounded check now and get the report here. 7-day trial, founding $50/mo (first 10, 12 mo) then $100/mo. Follow start_here.hop first (302). Task text

Browse

• GitHub tools
• Web scraping
• GitHub and code hosting

Related MCP servers

• ActableSite AI Crawler Monitor
  Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
• ASRM (AI Search Rank Monitor)
  Check a website's AI readiness and answer AI visibility and GEO questions from ASRM's guides.
• AIReady
  Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
• stillindexed SEO checkers
  Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
• llms.txt Readiness
  Does a site publish llms.txt, robots permissions and a sitemap agents can use? Bounded check.
• WebLens
  Scrape, crawl, map and extract the web. Pay per call in USDC, no account or API key.
• The Stall
  277 pay-per-call MCP tools: YouTube keyword research, stocks, crypto, DeFi, options, macro.
• TidyData Tools
  Holidays, business days, currency, time zones, DNS/WHOIS, web page checks, geodata and text tools.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.sadri-dridi/agent-crawl-readiness.svg)](https://aptlynx.net/servers/io.github.sadri-dridi/agent-crawl-readiness)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents