Toolmark Check a package

See what an agent tool actually does.

Toolmark checks published agent tools at an exact version, with no network and no secrets, and shows what they did next to what they claim. MCP servers and published OpenAPI documents today; agent skills and plugins are being built.

For example npm:name@1.2.3, @scope/name, an npm, PyPI or GitHub link, or npx -y name. Leave keys out.

  • Free
  • No account
  • Private report link

checks output against its own schema, wrong-type input, files changed by "read-only" tools, start-up without a key

What the tool says. What the tool did.

Every tool's claims sit next to what we observed in the sandbox. A mismatch is a review item with its evidence, not a grade.

Illustration
Declared by the tool
Observed in the sandbox
search
readOnlyHint · outputSchema: object
output matched its schema
verified
read_note
readOnlyHint: true
changed 1 file in the data folder
review item
get_status
no hints
returned an error as normal text, without the error flag
review item
send_digest
"sends a summary email"
called last with the network closed; still reported “sent”
review item

How a check runs

The same five steps for every package. The full method

1

Pin the exact version

We install the version you name, never a range, so the report describes one build.

Illustration
$ check npm:example-files-mcp@1.4.0
resolved 1.4.0 from the npm registry
installed, then the network was cut
2

Close the box

The server starts in a fresh cloud sandbox with no network, no secrets and only our synthetic data folder.

Illustration
network  none
secrets  none
data     synthetic, created for this run
3

Record what the tools claim

Names, input schemas and hints such as "read-only" are stored as claims, never taken as facts.

Illustration
read_note   readOnlyHint
search      readOnlyHintoutputSchema
send_digest no hints
4

Call with synthetic input

Tools that send, pay, message or publish are called last, only after a check proves the network is closed, and only with fake values. What we cannot call is marked "not tested", never held against the tool.

Illustration
search · output matched its schema
search · refused a wrong-type input with a clear error
send_digest · reported success with no network
5

Compare claim and behaviour

You get a private link. Each finding carries its evidence, its source and the date. There is no score.

Illustration
read_note declared readOnlyHint
observed: changed 1 file in the data folder · review item

From a real report

These numbers are computed from a stored report when the site is built, never typed by hand.

Read the full sample report

Tools declared
5
Tools called
4
Checks passed
1
Checks failed
1
Notes to review
3
Not called
1

Source: Toolmark's own example server with planted flaws, run 2026-11-18 in memory with the same checks as every package, except the network check, which needs the sandbox; without it, outward tools are not called. It is not a published package.

What we check

  • Output against its own schemaA tool that declares an output shape is held to it.
  • Wrong-type inputDoes the tool refuse it with an error an agent can act on?
  • Files changed by "read-only" toolsWatched in our synthetic data folder only.
  • Failures without the error flagAn error returned as a normal result is flagged for review.
  • Start-up without a keyDoes the server start and list its tools?

What we never do

  • No score, grade or badgeEvidence with sources and dates instead.
  • Nothing leaves the boxOutward tools run only after a check proves the network is closed, with fake values.
  • No penalty for "not tested"What we could not test is labelled, not held against the tool.
  • No pay-to-changePayment never changes a result. There is nothing to buy today.
  • No third-party code on our machinesPackages run only in the remote sandbox.

Who it is for

Teams that approve agent tools

Does this tool do what it says?

  • Behaviour evidence for one pinned version, not a listing.
  • "Read-only" claims checked against what changed.
  • Plain labels: verified, review item, not tested.
See a sample report
People who build agent tools

See what your users' agents will see.

  • Free, no account, no sales call.
  • What failed and why, with the evidence.
  • Check each release at its exact version.
Check your package

One method for every kind of agent tool

The same rules, the same labels and the same private report for each tool type. MCP servers can be checked today; the other two are being built.

MCP servers

Published on npm or PyPI and run at an exact version in a sandbox with no network. Every tool's claims next to its behaviour. Available now.

Agent skills and plugins

Being built. What a bundle contains versus what it claims, which scripts and hooks run and what they touch in our data folder. MCP servers inside a plugin are checked like any other.

HTTP APIs

The published OpenAPI 3.0 or 3.1 document, checked offline: its structure, operation ids, response schemas, examples against their schemas, every $ref, declared security schemes, and what changed between two versions. Available now. The API itself is not called; live calls would need a sandbox key from the provider and are not built.

Questions

What can I paste?

A package at an exact version (npm:name@1.2.3, pypi:name==1.2.3), a name without a version, an npm, PyPI or GitHub link, or the npx or uvx line from your MCP config. Without a version we look up the latest one, and the report names the exact version that ran. Only the package is read from what you paste; keys and everything else are thrown away.

Is this a security scanner?

No. We observe how a tool behaves when it is called. We do not scan for vulnerabilities, and a report never says a tool is free of risk.

Is there a score?

No score, grade or badge. Each finding is labelled verified, review item or not tested, with its evidence, source and date.

Where does the package run?

In a cloud sandbox started for one check. It is installed at the exact version, then run with no network, no secrets and only our synthetic data folder. The sandbox is thrown away afterwards.

Which tools do you call?

Every tool we can fill with synthetic input. Tools that write or delete can only touch our throwaway data. Tools that send, pay, message or publish are called last, only after the sandbox proves it has no network, and only with reserved fake values, so they cannot reach anyone. Hints such as "read-only" are treated as claims.

What does "not tested" mean?

We could not test that part, for example a tool with no description, an input we cannot fill with synthetic data, or a missing key. It is never counted against the tool.

Who can see my report?

Anyone with the private link. Reports are not listed or published, and they are deleted after 30 days.

Can a provider pay to change a result?

No. Nothing is for sale today, and payment will never change a result.

What do you store about me?

The package you entered, a daily-salted hash of your network address for rate limits, and the report, for 30 days. No cookies, no JavaScript, no analytics. Privacy

Put a tool in the box.

A package, a link or the line from your config. You get a private link to the report.

For example npm:name@1.2.3, @scope/name, an npm, PyPI or GitHub link, or npx -y name. Leave keys out.

An https link to an OpenAPI 3.0 or 3.1 document, JSON or YAML, without ?parameters. Only the document is read; the API is not called.