MCP server

RobotsGate

dev.workers.mike-tusa.robotsgate/robotsgate

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Website

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

4

Token cost

About 1,292 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.robotsgate]
url = "https://robotsgate.mike-tusa.workers.dev/mcp"

Gemini CLI

{"mcpServers":{"robotsgate":{"httpUrl":"https://robotsgate.mike-tusa.workers.dev/mcp"}}}

Cursor

{"mcpServers":{"robotsgate":{"url":"https://robotsgate.mike-tusa.workers.dev/mcp"}}}

VS Code

{"servers":{"robotsgate":{"type":"http","url":"https://robotsgate.mike-tusa.workers.dev/mcp"}}}

Antigravity

agy mcp add robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"robotsgate","server_url":"https://robotsgate.mike-tusa.workers.dev/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"robotsgate","server_url":"https://robotsgate.mike-tusa.workers.dev/mcp"}

Tools

• check_site ~222 tokens
  Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one G
• generate_robots ~693 tokens
  Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns
• list_crawlers ~171 tokens
  Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access
• validate_robots ~206 tokens
  Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fi

Browse

• Web scraping

Related MCP servers

• AIReady
  Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
• Kliqo AI
  Free AI visibility checks, then see if ChatGPT and Google AI name you or your competitors instead.
• Stormap
  Brainstorm scored idea maps, check npm/PyPI packages and vendors, find the right paid API.
• ActableSite AI Crawler Monitor
  Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
• Ai Visibility
  AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
• Site SEO & AI Crawler Audit
  Check public pages for SEO tags, AI crawler rules, AI citation signals and landing page conversion.
• Spook SEO
  Free SEO tools + the Spook SEO autopilot: analyze domains, generate and publish SEO articles.
• Citehound
  Scan any domain for AI readiness and generate llms.txt, robots.txt and schema. Free, read-only.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/dev.workers.mike-tusa.robotsgate/robotsgate.svg)](https://aptlynx.net/servers/dev.workers.mike-tusa.robotsgate/robotsgate)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents