MCP server

Robots Txt Checker

io.github.ryuamarines/robots-txt-checker

Checked 26 Oct 2026, 08:29 UTC · It answered and asked for a sign-in

Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).

At a glance

Type

MCP server

Publisher

Not verified

Status

Needs an API key

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http robots-txt-checker https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.robots-txt-checker]
url = "https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp"

Gemini CLI

{"mcpServers":{"robots-txt-checker":{"httpUrl":"https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp"}}}

Cursor

{"mcpServers":{"robots-txt-checker":{"url":"https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp"}}}

VS Code

{"servers":{"robots-txt-checker":{"type":"http","url":"https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp"}}}

Antigravity

agy mcp add robots-txt-checker https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"robots-txt-checker","server_url":"https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"robots-txt-checker","server_url":"https://faceted-napped-debtless.ngrok-free.dev/robots-txt-checker/mcp"}

Needs an API key or token from its maker, usually added as a header in the setup.

Browse

• GitHub tools
• Payments
• GitHub and code hosting

Related MCP servers

• Micropay Test Suite
  Verify your agent's x402 payment flow for free on Base Sepolia testnet before real money moves.
• Industrial Platform
  x402 MCP with exact and batched payments for recurring wallet, trading, market and web workflows.
• x402 endpoint checker
  Test any x402 paid URL without paying: 402, PAYMENT-REQUIRED, USDC, amounts, payTo, EIP-712, CORS.
• stillindexed SEO checkers
  Indexability, robots.txt, sitemap, TLS, domain expiry and Googlebot-vs-browser checks. No account.
• Tools
  Doormat Capital's paid x402 safety tools (token scan, HTTP security headers) for AI agents.
• Fetcher
  The web capability layer for AI agents: render, extract, DNS, SSL, WHOIS & more via x402.
• Pulsefeed X402
  Verify x402 payment endpoints before an AI agent pays: scam scan, on-chain checks, trust scores.
• Webbersites X402 Data Api
  45 pay-per-call AI agent tools: scraping, SEO, crypto data, lint, agent memory. x402 USDC on Base.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.ryuamarines/robots-txt-checker.svg)](https://aptlynx.net/servers/io.github.ryuamarines/robots-txt-checker)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents