MCP server

AI Crawler Access Check

io.github.hamzawamiqhq/ai-crawler-check

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.

Website

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

1

Token cost

About 186 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http ai-crawler-check https://check.citant.ai/mcp?src=registry

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.ai-crawler-check]
url = "https://check.citant.ai/mcp?src=registry"

Gemini CLI

{"mcpServers":{"ai-crawler-check":{"httpUrl":"https://check.citant.ai/mcp?src=registry"}}}

Cursor

{"mcpServers":{"ai-crawler-check":{"url":"https://check.citant.ai/mcp?src=registry"}}}

VS Code

{"servers":{"ai-crawler-check":{"type":"http","url":"https://check.citant.ai/mcp?src=registry"}}}

Antigravity

agy mcp add ai-crawler-check https://check.citant.ai/mcp?src=registry

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"ai-crawler-check","server_url":"https://check.citant.ai/mcp?src=registry","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"ai-crawler-check","server_url":"https://check.citant.ai/mcp?src=registry"}

Tools

• check_ai_crawler_access ~186 tokens
  Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and ll

Browse

• GitHub tools
• Cloudflare tools
• Web scraping
• GitHub and code hosting
• Cloud and hosting

Related MCP servers

• Seegeo
  Audit any website for AI visibility: graded report, findings with fixes, AI crawler access check.
• Siteiz
  Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
• Toolbox
  Small tools for agent builders: AI-writing tells, tools/list token cost, CLAUDE.md lint, and more.
• AIReady
  Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
• CitedBy
  Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.
• Ai Visibility
  AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
• RankRoot: AI visibility audit
  Audit a site's visibility to ChatGPT, Perplexity, Claude and AI agents. Generates llms.txt.
• CueScout
  Check if ChatGPT or Perplexity names a brand for a buying question, and if AI bots can read a site.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.hamzawamiqhq/ai-crawler-check.svg)](https://aptlynx.net/servers/io.github.hamzawamiqhq/ai-crawler-check)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents