Review AI Tool Catalog Changes
Compare tool names, schemas, handlers, scopes, and results before a changed catalog reaches an agent.
A tool catalog is a permission surface. A new tool, a changed description, or a broader input schema can change what an agent attempts even when no application role changes. Review catalog changes as a release item, not as generated documentation that can publish by default.
Export the old and new catalogs from the exact server builds. Compare stable tool identifiers, descriptions, argument schemas, result shapes, declared scopes, and handler routes. A text diff alone can miss a handler remap or a newly optional parameter that opens a wider path. The MCP security guidance puts authorization and validation at the server boundary, so catalog review must include the code that enforces them.
Change record
| Change | Risk question | Evidence |
|---|---|---|
| Tool added | Who can call it and what can it reach? | Permission rule and denial test |
| Description changed | Does it imply a wider task? | Reviewed text and model fixture |
| Input broadened | Can caller choose a new tenant, host, or path? | Rejected-argument test |
| Result expanded | Does it reveal new private fields? | Schema and redaction diff |
| Name collision | Could another server's tool be selected? | Qualified registry check |
Keep a server-qualified name in the operator inventory even when the client displays a short name. If two tools have the same visible name, resolve the ambiguity before release. A model may choose the wrong one, but a server must still enforce each operation's permission. Do not rely on tool ordering to make a collision harmless.
Release gate
In an owned lab, run a positive and negative test for each changed capability. For a description-only change, include a synthetic prompt that asks the agent to choose among the affected tools, but treat model choice as usability evidence, not authorization proof. For a schema change, test old and new inputs at the handler.
Record the reviewer, server commit, catalog digest, policy version, and decision. Publish the catalog only after the new handlers are covered by the intended scopes. If the diff cannot identify which build generated a tool, stop the rollout. A broad new catalog should not appear merely because an upstream OpenAPI file gained endpoints.
Know what your agents can reach
Map data access, tool actions and approval boundaries before testing. Use the scope guide, or discuss a review of your AI application.
Sources
- MCP Security Best PracticesAccessed
- OWASP Top 10 for LLM ApplicationsAccessed