MCP server

IA-QA Eval & RAG — 38 tools

io.github.JcJamet/ia-qa-eval

Checked 26 Oct 2026, 08:29 UTC · Connected to it and listed its tools

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

Source

At a glance

Type

MCP server

Publisher

Not verified

Status

Working at last check

Tools

38

Token cost

About 13,186 tokens per request

Sets up in

Claude Code, Claude Desktop, Codex, Gemini CLI, Cursor, VS Code, Antigravity, ChatGPT, OpenAI API, Grok

Last checked

26 Oct 2026, 08:29 UTC

Connect it to your agent

Add to CursorAdd to VS CodeVS Code Insiders

Opens the app, which shows the server and asks you to confirm. For other apps, copy the setup below.

Claude Code

claude mcp add --transport http ia-qa-eval https://www.ia-qa.com/mcp/eval

Claude Desktop

Settings → Connectors → Add custom connector, and paste the URL.

Codex (config.toml)

[mcp_servers.ia-qa-eval]
url = "https://www.ia-qa.com/mcp/eval"

Gemini CLI

{"mcpServers":{"ia-qa-eval":{"httpUrl":"https://www.ia-qa.com/mcp/eval"}}}

Cursor

{"mcpServers":{"ia-qa-eval":{"url":"https://www.ia-qa.com/mcp/eval"}}}

VS Code

{"servers":{"ia-qa-eval":{"type":"http","url":"https://www.ia-qa.com/mcp/eval"}}}

Antigravity

agy mcp add ia-qa-eval https://www.ia-qa.com/mcp/eval

ChatGPT

Settings → Apps & Connectors → Advanced settings: turn on Developer mode, then Create a connector and paste the URL. Which plans can do this varies.

OpenAI API (Responses tools)

{"type":"mcp","server_label":"ia-qa-eval","server_url":"https://www.ia-qa.com/mcp/eval","require_approval":"always"}

Grok (xAI API tools)

{"type":"mcp","server_label":"ia-qa-eval","server_url":"https://www.ia-qa.com/mcp/eval"}

Tools

• ab_test_report ~250 tokens
  Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns statistical comparison: mean, median, std deviation, winner, and improvement percentage.
• analyze_responses ~377 tokens
  Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closen
• bm25_score ~217 tokens
  Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
• build_rag_prompt ~309 tokens
  Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
• compare_models ~171 tokens
  Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.
• compare_responses ~458 tokens
  Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer
• consistency_check ~241 tokens
  Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French
• context_window_check ~190 tokens
  Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
• conversation_analyze ~144 tokens
  Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
• cot_analyzer ~157 tokens
  Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.
• count_tokens ~129 tokens
  Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.
• embedding_similarity ~322 tokens
  Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pai
• estimate_llm_cost ~186 tokens
  Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.
• extract_json_from_text ~144 tokens
  Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.
• generate_ci_workflow ~868 tokens
  Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (LLM quality gate, needs a provider API key as a repo secret), and "cli_checks" runs deterministic primitives via npx @ia-
• generate_eval_yaml ~328 tokens
  Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, c
• get_testing_guidelines ~403 tokens
  Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testin
• hallucination_check ~343 tokens
  Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement (
• list_llm_models ~172 tokens
  List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.
• llm_format_check ~194 tokens
  Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
• llm_generate ~346 tokens
  Generate text using open-source LLM models hosted on Groq (ultra-fast) or HuggingFace Inference (serverless). No API key required — the server provides its own keys. Supported models: Qwen3 32B, Gemma 4 27B, Gemma 3 27B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, Mistral Small 24B, and more. Use lis
• llm_json_schema_check ~153 tokens
  Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
• llm_output_validator ~397 tokens
  Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
• metamorphic_check ~1210 tokens
  Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, reordering, translation). Catches the failure class no reference answer can expose — an assistant tha
• model_info ~138 tokens
  Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
• needle_haystack_generate ~262 tokens
  Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
• prompt_test_suite ~359 tokens
  Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
• rag_relevance_rank ~147 tokens
  Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
• rerank_evaluate ~559 tokens
  Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the last. (1) BYO scores — give each passage the `score` your own reranker produced (Cohere, Jina, a self-h
• response_quality_score ~305 tokens
  Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a
• run_eval_contract ~451 tokens
  Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.
• run_semantic_tests ~477 tokens
  Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pip
• run_vlm_test_suite ~620 tokens
  Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stat
• similarity_score ~310 tokens
  Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.
• test_skill ~924 tokens
  Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: the spec's frontmatter rules checked without a model (name 1-64 chars, lowercase/digits/single hyphens
• token_budget_calculator ~290 tokens
  Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

Browse

• Claude tools
• GitHub tools
• Converting to Markdown
• Notes and wikis
• GitHub and code hosting
• Errors and monitoring

Related MCP servers

• IA-QA — 152 QA & Dev Tools for AI Agents
  152 free QA tools for agents: injection scan, evals, RAG, web audits. Smaller packs: /mcp/{pack}.
• x711 — Universal Agent Gas Station
  Pay-per-use tool API for AI agents. Free tier, x402 USDC micropayments, or API key.
• Llm Output Quality Monitor
  Cloudflare Workers MCP server: llm-output-quality-monitor
• Agent Data Tools
  567 agent tools: chain data, prices, sanctions, web, PDF. 25 free calls, then from $0.002.
• Agent Data Tools: Chain data
  354 tools: balances, blocks, gas, txs and logs on 5 EVM chains, plus Bitcoin. 25 free calls.
• Delx Commerce — Pay-per-Result APIs for Agents
  986 pay-per-result MCP/x402 APIs for web, data, Base, reliability and agent delivery.
• Brainonbnb Bobai
  BNB Chain agents: Plaza Board, ERC-8183 escrow status + free settle/refund, BSC trade cost, V3 LP
• xmp4 — Semantic code knowledge for your stack
  OSS libs in your stack, really used: source, tests, callers. C#, Java, TS, Python, Rust, PHP+.

Badge for your README

[![AptLynx](https://aptlynx.net/badge/io.github.JcJamet/ia-qa-eval.svg)](https://aptlynx.net/servers/io.github.JcJamet/ia-qa-eval)

Listing text is written by the maker; AptLynx lists tools and does not vouch for them. Find another tool · Browse by topic · For agents