Toolmark Check a package

How it works

Every check installs one published version, closes the box, calls what can be called with synthetic input, and writes down what happened next to what the tool claims.

The same rules apply to every tool and every customer.

Illustration

1. One exact version

Every check runs one exact version, never a range. You can name it (npm:name@1.2.3, pypi:name==1.2.3) or paste a name, a link or your config line, and we look up the latest version; the report names the version that ran. npm packages are installed with install scripts turned off. Python packages are installed from ready-built wheels only, so no build step runs. A dependency from outside the package registry means the package is marked not tested.

2. A closed sandbox

Each check gets a fresh cloud sandbox with an extra isolation layer between the package and the host. After the install the network is blocked. No environment variables, secrets, volumes or open ports are passed in. CPU, memory and time are fixed, and the sandbox is destroyed when the check ends, also on errors.

Third-party packages never run on Toolmark's own computers.

3. Synthetic data only

Toolmark creates its own data folder for every run and fills inputs from its own catalog of synthetic values. A provider's example value only tells us which type to use. If a server will not start without a key and says so, it gets an obviously fake placeholder key, and the report says so.

4. What is called and what never is

  • Called last: tools that send, pay, message or publish, or act on accounts or the network, only after a check proves the sandbox has no network, and only with reserved fake values (addresses that cannot exist, a fictional phone number, zero amounts). We record whether they admit the failure or claim success anyway.
  • Called before them: tools that write or delete. They can only touch our throwaway data.
  • Never called: tools with no description, and every tool when the network check fails or cannot be read.
  • Called first: tools that read.
  • Hints such as "read-only" are treated as claims. They decide nothing on their own.

5. The checks

Output against its own schema
A tool that declares an output shape must return it. Breaking it is a failure.
Wrong-type input
We send a value of the wrong type. A clear refusal is verified; quietly accepting it is only an observation.
A second valid input
A different valid input should give a result of the same shape.
Data-folder changes
The data folder is compared before and after every call. A "read-only" tool that changed files, or a non-destructive tool that removed one, becomes a review item with the paths.
Failures without the error flag
A normal result whose text reads like a network or sign-in failure is a review item, never verified.
Start-up and protocol
Does the server start, answer the protocol correctly and list its tools?
Package facts
Size of the tool list, packages with install scripts, and dependencies as the package declares them.

6. HTTP API documents, skills and plugins

Today Toolmark checks MCP servers and published OpenAPI documents. Skills and plugins are being built and are not available yet.

Agent skills and plugins are published bundles. We will check what a bundle contains against what it claims, which scripts and hooks run, and what they touch in our data folder. An MCP server inside a plugin will be checked like any other.

HTTP APIs run on servers we do not own, so Toolmark reads only the published OpenAPI 3.0 or 3.1 document, once, and never calls the API. It checks the structure against its own rules for the required parts of the specification, that every operation has a unique operationId and a response schema, that examples match their schemas, that every $ref resolves inside the document (a $ref to another file or address is never fetched), that operations name only declared security schemes, and what changed since a previous version. Everything about the API's live behaviour is marked not tested. Live calls would need a sandbox key the provider enters themselves, after they show they own the API; that is not built.

7. Limits

  • A report describes one version on one day. A new version needs a new check.
  • "No change seen" is never proof that a tool cannot change anything.
  • Review items can be false positives. That is why they are not failures.
  • The provider's hosted service, if any, is not part of an MCP package check.

One package at an exact version, a private link to the report.

Check a package