Quickstart

No server of your own yet? The bundled mcpeval-demo binary is a real MCP server with a clean personality plus --broken <aspect> personalities that reproduce specific defects. Scaffold a manifest from its live catalog and run the deterministic battery:

mcpeval init --server demo --confirm-read-only \
  --output demo.manifest.json -- mcpeval-demo
mcpeval probe --server demo \
  --manifest demo.manifest.json -- mcpeval-demo

init writes every probe kind that needs no domain input: catalog budgets, pagination, protocol negotiation, and surface listing, plus per-tool cases for each candidate whose call succeeds, with {} or values built from its schema, and names every tool it left out. Candidates are tools annotated readOnlyHint: true and, after the read-only attestation, unannotated tools. Tools annotated destructiveHint: true or readOnlyHint: false are never called, and --dry-run lists each tool's decision without calling it. See Scaffold a manifest for every case it writes.

On the clean demo the gate passes 25/25 and readiness scores 91/100 under mcpeval's standard, with each area's score and every lost point. Flip one flag to watch a fixed failure reason appear:

mcpeval probe --server demo --manifest demo.manifest.json \
  -- mcpeval-demo --broken stalled-cursor
# catalog-pagination fails with pagination-stalled-cursor

mcpeval-demo --help lists every aspect, and an unknown aspect exits 2. The aspects reproduce incoherent schemas, unfaithful results, unstable error codes, accepted invalid arguments, bloated catalogs, terse descriptions, broken pagination, slow and flaky calls, dropped cancellations, echoed protocol versions, answered unknown methods, malformed sampling and elicitation requests, missing subscription notifications, and malformed completions.

The same manifest fails --broken negotiation with a negotiation-* reason, --broken bloated with discovery-limit-exceeded and token-budget-exceeded, and --broken slow with latency-budget-exceeded. Error honesty, state recovery, instruction fidelity, cancellation, sampling, elicitation, resource subscription, and completion need inputs init cannot infer; add those cases by hand.

To evaluate your own server, point the same flow at it:

mcpeval init --server demo --confirm-read-only -- your-mcp-server --flags
mcpeval probe --server demo -- your-mcp-server --flags

After the captured sessions contain representative traffic, rebuild the index, promote supported recurring failures, and view the sanitized findings:

mcpeval index
mcpeval promote
mcpeval findings --format agent

Wrap a newline-delimited JSON-RPC MCP server and let the client connect through the shim:

mcpeval shim --server demo -- your-mcp-server --flags

Verify one promoted finding with exactly one declared case:

mcpeval verify --finding finding-0123456789abcdef \
  --case repeat-read --manifest mcp-eval.manifest.json \
  -- your-mcp-server --flags

Use identifiers emitted by your own mcpeval findings output. The example finding ID only demonstrates the CLI shape.