How to Judge an MCP Server in Five Minutes

In short

There are more MCP servers than anyone can evaluate properly, and most "best MCP servers" lists are vendor rosters with no selection criteria. This is a triage pass, not an audit: five minutes to decide whether a server is worth wiring into your agent at all. In order — name the task it removes, find the real repository behind the package name, check whether it is maintained on a scale of weeks rather than years, list its tools before installing with npx @modelcontextprotocol/inspector, and compare the credential it demands against the credential the task needs. A server fails triage if it exposes forty tools you will never call, if its only value is an HTTP call your agent could already make, if there is no way to scope it down, or if its last commit predates the current protocol revision. Passing triage is not the same as being safe.

The five-minute pass

Set a timer. If a server cannot survive five minutes of scrutiny, it does not get a slot in your config.

0:00–0:30 — Name the task. Write one sentence: "this removes X from my workflow." If the sentence comes out as "it lets Claude talk to Y", that is a description of the protocol, not a benefit. Servers that survive this step replace something you currently do by hand, several times a week.

0:30–1:30 — Find the real repository. Package name and repo name are not the same thing, and a listing directory is not the source of truth. For an npm package:

npm view @playwright/mcp repository.url time.modified versions dependencies

You want three things off that output: a repository URL that resolves, a modified date measured in weeks, and a dependency list you can read in one screen. Twenty dependencies to call one REST API tells you how the thing was built.

1:30–3:00 — Read the tools before installing. The step everyone skips, and the one that decides most cases. The MCP Inspector runs a server in isolation and shows its actual tool list:

npx @modelcontextprotocol/inspector npx -y some-mcp-server

Count the tools and read the descriptions the way your model will — they land in your context verbatim. Vague descriptions ("performs an action on the resource") mean your agent picks the wrong tool and you spend the savings debugging it.

3:00–4:00 — Check the credential ask. What token does it want, at what scope, and can you narrow it? See the permissions section below.

4:00–5:00 — Apply the wrapper test. What does this server do that your agent could not do with the tools it already has? If the honest answer is "nothing, but it is more convenient", that can still pass — but it should then be a three-tool server, not a forty-tool platform.

Signals that a server is maintained

Maintenance matters more here than in most dependencies, because the protocol itself moves: the MCP specification is versioned by date, and servers pinned to an old revision drift out of client compatibility quietly. Signals worth weighting:

• Commit recency, not commit count. Last push within the last month is the bar. A server untouched for a year was written against a protocol revision that no longer matches what your client speaks.
• Release cadence on the package registry. @playwright/mcp has published 401 versions since March 2025 (checked 28 July 2026). That is a project with an actual release process behind it.
• Issue behaviour, not issue count. Sort the issue list by recently updated. 353 open issues on a repo with daily commits is a busy project; 40 open issues with no maintainer comment in six months is an abandoned one.
• A real license. MIT or Apache-2.0 in a LICENSE file. "No license" means you legally have no right to use it, whatever the README implies.
• A verified namespace in the official registry. Proof that the publisher controls the name it claims, nothing more — the registry delegates security scanning upstream.

Two anti-signals that look like signals: star counts, and a slick landing page.

Signals that it is a wrapper

A large share of published servers are one HTTP client plus a tool schema. Not automatically bad — a thin wrapper around a well-documented API can be exactly right. It is bad when the wrapper adds no judgement and costs context anyway. The tells:

• Tool names that mirror API endpoints one-to-one. get_v2_users_list, post_v2_users_create. Nobody designed a tool surface here; a script generated one from an OpenAPI file, and your agent gets no help deciding what to call.
• No result shaping. The tool dumps the raw JSON payload, all 40KB of it, into your context. A server built for agent use returns the fields you needed, or paginates.
• No outputSchema. The spec supports declaring output structure so clients can validate results. Its absence is not fatal; its presence means someone read past the quickstart.
• Forty tools, no grouping. Every tool definition costs tokens on every request. A server with no toolset flags or capability gating has pushed its design problem onto you.
• README screenshots instead of a tool table. If the docs cannot say what the tools are, that is the answer.

The counter-test: does it hold state or credentials your agent cannot hold on its own? Browser sessions, database connections, OAuth flows, long-lived job handles. That is where a server earns its slot.

Permissions: what it asks for versus what it needs

Before pasting any token, compare two things: what the setup instructions ask for, and what your one-sentence task requires. They are often far apart.

A well-designed server lets you close that gap. The GitHub MCP server is the clearest public example: 20+ toolsets, a subset enabled with --toolsets repos,issues or GITHUB_TOOLSETS, individual tools with --tools, and a full read-only mode via --read-only or GITHUB_READ_ONLY=1. Playwright MCP is structurally the same: core browser tools on by default, network, storage, DevTools, PDF and coordinate tools opt-in behind --caps, plus --isolated and origin allow/block lists.

The absence of any such switch is itself a finding. A server whose only configuration is a token cannot be reduced to fit your task — you either accept its full blast radius or you do not run it.

Also note that the protocol lets servers attach behaviour annotations to tools, but the spec warns that clients MUST treat those annotations as untrusted unless the server itself is trusted (spec, Tools). A tool marked read-only is a claim by its author, not an enforced property. Scope the credential; do not scope the trust.

A worked example on two real servers

Both pass triage, and they pass it differently — the pass is about fit, not a leaderboard. Figures checked 28 July 2026 via the GitHub and npm APIs.

Note the asymmetry in the credential row. GitHub's server wants a token whose blast radius is your account; Playwright's wants none, but reuses a persistent browser profile by default, so an already-logged-in session is the credential. "No API key required" is not "no access granted".

Where this pass stops

Triage answers "is this worth my context and my time", not "is this safe". Five minutes cannot detect a sleeper update in a package that shipped clean for a year, and a tool description proves nothing about runtime behaviour. Before a server touches real credentials, pin the version, install without lifecycle scripts and give it the narrowest token you can create — the full argument is in MCP server security before you install. For the wiring, see the setup walkthrough and the MCP connection guide; the MCP arsenal section has per-server notes.

FAQ

How do I know if an MCP server is any good before installing it?

Run it in isolation with npx @modelcontextprotocol/inspector npx -y <server> and read the tool list it produces. That takes about a minute and answers most of the question: how many tools it injects into your context, whether the descriptions are written for a model to read, and whether the names look designed or generated from an API spec.

How many MCP servers should I have connected at once?

Fewer than you think. Every connected server's tool definitions go to the model on each request, so you pay whether you call the server or not, and a crowded tool list makes selection worse. Keep the two or three you use weekly, and enable the rest per-project.

Do stars and download counts mean an MCP server is trustworthy?

They mean it is popular, which mostly means discoverable. Both metrics lag by months and both can be manufactured. Commit recency, maintainer responses on recent issues and a readable dependency list predict far better whether it will still work next quarter.

What is the difference between vetting for quality and vetting for security?

Quality vetting asks whether the server earns its slot: does it remove real work, is the tool surface sane, is it maintained. Security vetting asks what it can reach if it turns hostile — credential scope, install-time code execution, update path. Run the quality pass first, because it is faster and eliminates most candidates.

Is a server in the official MCP registry automatically safe?

No. Listing verifies namespace ownership — that the publisher controls the name it publishes under. The registry's own documentation says security scanning is delegated to upstream registries such as npm and PyPI. Treat listing as an identity check, not a code review.