› cavi-ai/
GitHub ↗

Evaluation dimensions

MCP Eval 0.8.0 has exactly five headline evaluation dimensions. The CLI also exposes fifteen supplemental probes, documented separately below.

Headline evaluation dimensions#

discovery-cost

Measures the number of tools and encoded tools/list catalog bytes in memory. The case passes when both values stay within max_tools and max_schema_bytes. Catalog descriptions and schemas are not persisted or printed.

schema-guessability

Requires the selected tool to expose an object input schema. Every required field must exist in properties and in the manifest's naive arguments, and the resulting tool call must succeed.

error-honesty

Requires an initial error, a stable error code, and retryable metadata matching expect_retryable. A retryable error must recover within max_attempts; a non-retryable error must remain consistently non-retryable through the confirmation attempt.

state-recovery

Runs the declared failure, recovery, and validation calls in order. It passes only when failure is observed, recovery succeeds, and validation succeeds. Mutating recovery plans need a declared sandbox and --allow-mutation.

contention

Synchronizes two independently initialized MCP clients against the same tool and arguments. Both calls must succeed.

Supplemental probes#

These checks are available in the manifest and CLI, but are not additional headline evaluation dimensions in version 0.8.0.

token-cost

Estimates the context cost of the tools/list catalog. Each complete tool entry (name, description, and input schema) is measured in memory as encoded bytes, then divided by four and rounded up. The deterministic, model-independent estimate is checked against max_total_tokens and an optional max_tool_tokens ceiling. Only tool names and numeric measurements appear in the result; descriptions and schemas are never persisted or printed.

degradation-over-n

Repeats the declared tool call through max_attempts. The first error fails the case and records its attempt number; all calls must succeed to pass.

instruction-fidelity

Runs one call and compares its machine-readable outcome with the declared expect object. It can check required result fields and exact scalar values at literal top-level keys or nested JSON Pointer paths, or an error code. It does not invoke an external model.

latency-budget

Runs a declared read-only tool through a bounded number of attempts and fails when an observed response exceeds max_latency_ms. It reports only the numeric latency measurement and a fixed failure label.

pagination

Walks a read-only tools/list catalog through its declared cursor sequence. It fails on malformed pages, duplicate tools, repeated cursors, or an exceeded max_pages bound. Catalog descriptions and schemas stay in memory.

payload-bounds

Injects one exact-size argument — a deep copy of the declared arguments with a single oversized ASCII string field — and verifies the server neither crashes nor hangs.

  • A clean structured JSON-RPC rejection is honest bounded behavior, unless

expect_handled asserts the tool must actually handle the size; then a rejection fails the case.

  • A transport-level failure always fails with payload-unhandled.
  • Only the numeric latency and a fixed label are reported; the injected bytes

are never echoed.

surface-listing

Validates the envelopes of the server's declared resources and prompts surfaces: item arrays must exist, cursor sequences must terminate within max_pages, and listing errors are defects. Surfaces the server did not declare pass trivially; the probe only verifies what the server claims.

output-schema

For a tool that declares outputSchema, the response must carry structuredContent satisfying the complete schema: types, nested fields, required properties, and constraints. Local schema references are supported; external references are disabled. Tools without a declared output schema pass trivially. Validation happens in memory; result values and schema diagnostics are never persisted or printed.

cancellation

Issues a read-only call, cancels it with notifications/cancelled, and requires the server to acknowledge the cancellation. The probe first confirms the tool succeeds uncancelled, so an honoring server cannot pass by accident.

  • Passes: silence for the cancelled request id within grace_seconds, or the

structured "Request cancelled" error (-32800) that production servers return for cancelled requests.

  • Fails: a full result delivered after the cancellation, or an unrelated error

code, on both stdio and Streamable HTTP transports.

protocol-negotiation

Runs three fresh initialize handshakes:

  1. The handshake with the supported date-shaped version must echo it.
  2. The handshake with an unknown date-shaped version (bogus_version from the

manifest) must be answered with a date-shaped version the server actually supports — never the echoed request.

  1. The version the server claims must itself be echoable on a third handshake.

A server that parrots the unknown version, replies with a non-date-shaped value, or advertises support it does not have fails.

sampling

Declares the client sampling capability and calls the tool. Any sampling/createMessage sub-request the server issues mid-call is answered with a deterministic stub sample; the server may ask at most max_requests times per call, must send well-formed requests, and must complete the call once its sampling request is answered.

elicitation

Same shape as sampling for elicitation/create: sub-requests must carry a message and a requestedSchema object, are answered with the manifest's declared respond action (accept, decline, or cancel), and stay within max_requests.

resource-subscription

For a server that declares resources.subscribe: the declared URI must be readable, resources/subscribe must succeed, the manifest's trigger tool must fire, and notifications/resources/updated for the URI must arrive within max_wait_seconds (on stdio through the shared session stream, on Streamable HTTP through the session's standing GET stream). A clean resources/unsubscribe closes the case; undeclared subscription support passes trivially.

completion

For a server that declares the completions capability, one completion/complete request for the manifest's reference (ref_type/ref_uri) and argument must answer with a well-formed completion — completion.values, an array of strings, optionally with hasMore/total — and stay within max_values. Undeclared completion support passes trivially.

  • A structured error naming the argument is the argument-unknown defect: the

server's completion surface is out of sync with its own prompt declarations.

  • A malformed envelope, a transport failure, or a flood of values fails the

case.

workflow

Runs a declared read-only sequence in one fresh client session, with 2 through 32 steps and 1 through 20 repetitions. Each step has explicit arguments and an expect object. All steps must pass for the workflow to pass.

  • The first mismatch stops the case and identifies the call number.
  • A transport fault makes the entire case incomplete.
  • Tools annotated as writers are refused before any step executes.
  • The workflow does not reuse other cases' client sessions; server-global or

external state is not reset.

See the manifest example.

All tool calls pass through the normal sanitized synthetic-record boundary.