How do I run `claude plugin eval` on a Claude Code plugin?
On Claude Code 2.1.269+, run claude plugin eval to execute a plugin's eval suite and get JSON + HTML scored reports. Not the same as skill-creator evals.
On Claude Code 2.1.269+, run claude plugin eval to execute a plugin's eval suite against Claude Code and get scored, reproducible results as a JSON + HTML report. Check claude plugin eval --help on your install for the live flag list: this page only covers what the CHANGELOG states. For iterating on a single skill inside a conversation, use the skill-creator flow in the official skills docs instead. The two paths are related but not interchangeable.
claude plugin eval (see --help). You get JSON + HTML scores. Skill-creator's evals/evals.json loop is a different tool for skill iteration: don't mix the formats.Prerequisites
- Claude Code ≥ 2.1.269, confirmed in the anthropics/claude-code CHANGELOG entry for
2.1.269: "Addedclaude plugin eval: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); seeclaude plugin eval --help." - A plugin with an eval suite. The command runs that suite. An empty or missing suite won't give meaningful scores.
- CLI available in your shell.
claudeon PATH so you can invokeclaude plugin eval.
Not tested in the CLI (as of 2026-09-12). We haven't captured claude plugin eval --help yet, so flag names, positional args, and CI exit-code details below follow the CHANGELOG and code.claude.com/docs/en/skills only. Re-run --help on your machine before scripting CI.
Steps
High-level loop (exact flags: your --help):
- Confirm version:
claude --version(or equivalent) shows 2.1.269+. - Open a terminal in the context where the target plugin is installed or available to Claude Code.
- Run:
claude plugin eval --help
- Run
claude plugin evalwith whatever arguments--helpdocuments for selecting the plugin / suite. - Inspect the JSON and HTML report outputs the CHANGELOG promises.
- Optionally wire the command into CI. Official skills docs note it can exit non-zero below a threshold so you can gate merges. The docs don't name the threshold flag, so check
--helpfor it.
What "eval suite" means, per the docs: for a skill that ships in a plugin, claude plugin eval runs each prompt in an isolated session with and without the plugin, scores with graders you define (or that it writes for you), and produces comparable results. That is the plugin-level path described under Evaluate and iterate on a skill in the official skills docs. For authoring and testing a single Skill first, see Create a Claude Code skill and Claude Skills.
Outputs: JSON + HTML
Per CHANGELOG 2.1.269, successful runs produce:
| Artifact | Role |
|---|---|
| JSON report | Machine-readable scored results, useful for CI parsers and trend diffs |
| HTML report | Human-readable scored report for review |
Treat both as the reproducible score pack for that plugin eval run. Do not assume file paths or grader schema beyond what --help and your suite define.
The Claude moves worth stealing, by email
Workflows tested on real jobs, sent only when they are worth it. Free.
Not the same as skill-creator evals
Official docs draw a hard line (Extend Claude with skills · Evaluate and iterate):
| Path | Scope | Format / loop |
|---|---|---|
claude plugin eval | Plugin-shipped skills / plugin eval suite | Isolated with/without plugin; JSON + HTML; CI-friendly exit |
skill-creator plugin | Single skill inside a Claude Code conversation | Own evals/evals.json · grading · benchmark · HTML review viewer |
Install skill-creator (when you need the in-chat loop):
/plugin install skill-creator@claude-plugins-official
Then ask Claude something like: evaluate my summarize-changes skill with skill-creator. The docs state the two formats aren't interchangeable: don't point claude plugin eval at a skill-creator evals.json and expect it to work unchanged.
If you are still learning what Claude Code is, start with What is Claude Code?. Plugin packaging notes also appear under the official plugins docs.
Pitfalls
- Wrong version: pre-2.1.269 builds won't have the command.
- Empty or missing suite: "eval suite" implies the plugin actually ships cases; otherwise scores are noise.
- Conflating skills vs plugins: skill-creator is for iterating one skill in-session;
claude plugin evalis the CLI suite against Claude Code for a plugin. - Trusting secondhand flags: forum threads and blogs are not official docs. Use CHANGELOG +
--helpsame day. - Mixing eval formats: skill-creator's
evals/evals.json≠ plugin eval suite format.
Field note, checked against CHANGELOG 2.1.269 and the skills docs in September 2026 (we haven't captured the CLI --help output yet). Re-check the CHANGELOG and skills docs the day you wire CI.
FAQ
Does claude plugin eval replace skill-creator?
No. Docs present both: plugin CLI for suite/CI; skill-creator for conversational iteration. Formats differ.
What version added claude plugin eval?
Claude Code 2.1.269 (CHANGELOG).
Where do I see the live flags?
claude plugin eval --help on a 2.1.269+ install. This guide sticks to the CHANGELOG and docs, so --help is the place for exact flags.
Can I use it for a personal ~/.claude/skills/ skill that isn't in a plugin?
For a single skill in conversation, use skill-creator. Plugin eval is framed around a plugin's eval suite.
Where next
- Create a Claude Code skill for authoring and trigger tests before suite eval.
- Claude Skills for the product overview.
- What is Claude Code? if you are still onboarding the CLI.
- Only if you package marketing skills as plugins: Claude for marketing.
Sources (checked 2026-09-12)
- anthropics/claude-code CHANGELOG.md · 2.1.269:
claude plugin eval+ JSON/HTML - code.claude.com/docs/en/skills: Evaluate and iterate; plugin eval vs skill-creator; formats not interchangeable
- code.claude.com/docs/en/plugins: plugin packaging context