mcpjam in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.
GitHub Actions
Authentication
There are three ways to authenticate in CI, depending on your server setup.Option 1: Headless OAuth login
Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run. Secrets needed:Option 2: Refresh token
Best when you already have a refresh token from a previousoauth login. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.
Secrets needed:
Option 3: Static API key
Best when your server uses a non-expiring API key instead of OAuth. Secrets needed:Option 4: No auth
Some servers don’t require authentication at all. Secrets needed:Tool surface diffing
Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).OAuth conformance suite
Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.Protocol conformance suite
Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.MCP Apps conformance suite
Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.protocol conformance, oauth conformance, and apps conformance also accept --reporter junit-xml when you only need one target/check selection instead of a suite config file.
GitLab CI
The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and.gitlab-ci.yml syntax.
Authentication
Headless OAuth login
Refresh token
Static API key
Tool surface diffing
Snapshot your tool surface before and after a deploy to catch breaking changes.OAuth conformance suite
Evals in CI
There are two ways to wire MCPJam evals into a pipeline: trigger a hosted eval run with the CLI, or run evals locally with the SDK and upload the results. Both authenticate with an MCPJam API key (sk_…) from Settings → API keys.
Trigger a hosted eval suite
mcpjam cloud eval run starts an asynchronous run of a suite that lives in your MCPJam project. Without --wait, it prints a launch receipt and returns immediately. In CI, add --wait and --out to write a structured JSON report after every launched run reaches a terminal state.
Secrets needed:
eval run prints a View: line after the payload so you can open the run directly from the terminal:
--format json output is unchanged, so scripts that parse the JSON stream are unaffected.
Use --wait-timeout <ms> to replace the 10-minute default. --out defaults to the structured JSON format; add --reporter junit-xml to write JUnit XML instead, or --reporter html for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When --reporter is present, the same report is also written to stdout.
eval gate sets a verdict-based exit code, and writes its report before doing so: 0 passed or waived, 1 an eval verdict failed, 2 usage error, 3 incomplete or non-gateable. Infrastructure conditions never map to 1, so retrying on 3 is safe. This is a different, four-code contract from eval run --wait above — gate’s 3 means “incomplete”, not the six-code scheme’s 3 (“auth failed”), and the two are deliberately not unified (see the CLI reference for why). eval status also prints a View: line in human format, identical to the one eval run prints.Waiving a gate
A run whose gate failed can be overridden by an authorized user until an expiry they name, so a release is not blocked while a known regression is being fixed:eval gate then exits 0 and reports the outcome as waived. It is not
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a <skipped> element, so it neither fails the build nor renders as
a clean green row).
Only a real verdict failure is waivable. A cancelled run, a --wait timeout, or
a network failure still exits 3 with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.
Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform’s answer. mcpjam cloud eval gate unwaive --run "$RUN_ID" ends one early.
Decision summary
eval run --wait, eval status, eval gate and eval compare all read one versioned object — the run decision summary — and every output format restates it. Where each command puts it:
Two scoping rules that are easy to miss.
eval run --wait attaches a summary only when the invocation launched one run: a fan-out has several, and labelling a receipt about N runs with the decision of one would be a false claim rather than a partial one. eval compare reports the compare side’s decision only — the baseline’s failures are a different run’s diagnostics, and printing them here would read as this run’s.
measurementUnit is caseVariant under verdict policy v2 — one case under one provider/model, with repetitions as trials inside it — and trial on a legacy percent-threshold run. A 3-case suite with 5 repetitions is legitimately “3” under one unit and “15” under the other, so a count quoted without its unit is not a fact.
The summary explains the verdict; it never re-decides it. Under policy v2 the run’s own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on decision. The per-trial diagnostics sit underneath that: a case can pass with a failing trial in it, so tallying the diagnostics gives a different answer than the platform reached.
notEstablished is not a failure. It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. undecided.reason says which. It is also not inconclusive, which is a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.
A page of diagnostics says whether it is the whole story. diagnostics.complete is true only when the listed trials are the run’s entire non-passing set, and scannedIterations says how many were examined — so an empty list from a complete page (“nothing failed”) is distinguishable from an empty list from a partial one (“we did not look”).
Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage’s row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.
eval status prints the block only when a terminal run did not pass — a clean pass has nothing to diagnose. --format json stays exactly one parseable document in every case: the summary rides inside it, never as a second block appended after it. If the summary cannot be fetched, it is omitted rather than failing the command.
The human block leads with where the chain broke. Under the diagnostics headline, before any per-trial detail, a non-passing run gets one line naming the earliest stage at which a readable trial stopped, why, and how many trials stopped there:
connection → discovery → selection → call → response → userValue — never “most common”, so the count beside it is what tells you whether the run had one problem or several. When the breaks are spread the line says so (earliest of 3 stages that broke), and when some chains could not be read it says that too (1 more had no readable chain), because otherwise the denominator quietly shrinks to the trials that happened to validate. A run that reached no stage at all — a setup abort, an evaluator error — names its bucket instead of inventing a location.
The line is not a diagnosis. A first failed stage is a location and a failure category is a bucket; neither on its own says what to change.
eval status --stages expands each failing trial to all six chain rows with their states and reasons. It is off by default: six rows per trial is a lot of terminal on a run with twenty failures, and the first-break line above already carries the answer. Human output is not a stable contract — script against --format json, where the enums travel as enums.
Hosted runs execute LLM iterations on the platform and consume your organization’s credits or configured provider keys. See the cloud eval command reference for the full surface, including cloud eval judge (request LLM-as-judge grading on a finished run), cloud eval validate (offline suite-file validation), cloud eval export (write a hosted suite to a local file), cloud eval checks list/connect (GitHub Checks integration), and more.
Upload SDK eval results
If you instead run evals inside your own CI job with@mcpjam/sdk (EvalTest / EvalSuite), set MCPJAM_API_KEY and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):