How do I run `claude plugin eval` on a Claude Code plugin?
On Claude Code 2.1.269+, run claude plugin eval to execute a plugin's eval suite and get JSON + HTML scored reports. Not the same as skill-creator evals.
On Claude Code 2.1.269+, run claude plugin eval to execute a plugin's eval suite against Claude Code and get scored, reproducible results as a JSON + HTML report. Check claude plugin eval --help on your install for the live flag list: this page does not invent flags beyond the CHANGELOG. For iterating on a single skill inside a conversation, use the skill-creator flow in the official skills docs instead. The two paths are related but not interchangeable.
claude plugin eval (see --help). You get JSON + HTML scores. Skill-creator's evals/evals.json loop is a different tool for skill iteration: don't mix the formats.Prerequisites
- Claude Code ≥ 2.1.269, confirmed in the anthropics/claude-code CHANGELOG entry for
2.1.269: "Addedclaude plugin eval: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); seeclaude plugin eval --help." - A plugin with an eval suite. The command runs that suite. An empty or missing suite won't give meaningful scores.
- CLI available in your shell.
claudeon PATH so you can invokeclaude plugin eval.
[GAP · 2026-09-12]: claude was not installed in the drafting environment, so claude plugin eval --help could not be captured. Flag names, positional args, and CI exit-code details below stick to the CHANGELOG + code.claude.com/docs/en/skills only. Re-run --help on your machine before scripting CI.
Steps
High-level loop (exact flags: your --help):
- Confirm version:
claude --version(or equivalent) shows 2.1.269+. - Open a terminal in the context where the target plugin is installed or available to Claude Code.
- Run:
claude plugin eval --help
- Run
claude plugin evalwith whatever arguments--helpdocuments for selecting the plugin / suite. - Inspect the JSON and HTML report outputs the CHANGELOG promises.
- Optionally wire the command into CI. Official skills docs note it can exit non-zero below a threshold so you can gate merges. Exact threshold flags: [GAP] until
--helpis verified locally.
What "eval suite" means here (docs-level, not invented schema): for a skill that ships in a plugin, claude plugin eval runs each prompt in an isolated session with and without the plugin, scores with graders you define (or that it writes for you), and produces comparable results. That is the plugin-level path described under Evaluate and iterate on a skill in the official skills docs. For authoring and testing a single Skill first, see Create a Claude Code skill and Claude Skills.
Outputs: JSON + HTML
Per CHANGELOG 2.1.269, successful runs produce:
| Artifact | Role |
|---|---|
| JSON report | Machine-readable scored results, useful for CI parsers and trend diffs |
| HTML report | Human-readable scored report for review |
Treat both as the reproducible score pack for that plugin eval run. Do not assume file paths or grader schema beyond what --help and your suite define.
Not the same as skill-creator evals
Official docs draw a hard line (Extend Claude with skills · Evaluate and iterate):
| Path | Scope | Format / loop |
|---|---|---|
claude plugin eval | Plugin-shipped skills / plugin eval suite | Isolated with/without plugin; JSON + HTML; CI-friendly exit |
skill-creator plugin | Single skill inside a Claude Code conversation | Own evals/evals.json · grading · benchmark · HTML review viewer |
Install skill-creator (when you need the in-chat loop):
/plugin install skill-creator@claude-plugins-official
Then ask Claude something like: evaluate my summarize-changes skill with skill-creator. The docs state the two formats aren't interchangeable: don't point claude plugin eval at a skill-creator evals.json and expect it to work unchanged.
If you are still learning what Claude Code is, start with What is Claude Code?. Plugin packaging notes also appear under the official plugins docs.
Pitfalls
- Wrong version: pre-2.1.269 builds won't have the command.
- Empty or missing suite: "eval suite" implies the plugin actually ships cases; otherwise scores are noise.
- Conflating skills vs plugins: skill-creator is for iterating one skill in-session;
claude plugin evalis the CLI suite against Claude Code for a plugin. - Inventing flags: Discourse threads and secondhand blogs are not product truth. Use CHANGELOG +
--helpsame day. - Mixing eval formats: skill-creator's
evals/evals.json≠ plugin eval suite format.
Field note, tested against CHANGELOG 2.1.269 and the skills docs on September 2026 (CLI --help not captured in the drafting environment: [GAP]). Re-check the CHANGELOG and skills docs the day you wire CI.
FAQ
Does claude plugin eval replace skill-creator?
No. Docs present both: plugin CLI for suite/CI; skill-creator for conversational iteration. Formats differ.
What version added claude plugin eval?
Claude Code 2.1.269 (CHANGELOG).
Where do I see the live flags?
claude plugin eval --help on a 2.1.269+ install. This guide marks flag details [GAP] when the CLI wasn't available at write time.
Can I use it for a personal ~/.claude/skills/ skill that isn't in a plugin?
For a single skill in conversation, use skill-creator. Plugin eval is framed around a plugin's eval suite.
Where next
- Create a Claude Code skill for authoring and trigger tests before suite eval.
- Claude Skills for the product overview.
- What is Claude Code? if you are still onboarding the CLI.
- Cross-cluster only if you package marketing skills as plugins: Claude for marketing.
- Claude Code / plugin-authoring guides on this site: planned (no live
/guides/sibling yet; do not invent URLs).
Sources (checked 2026-09-12)
- anthropics/claude-code CHANGELOG.md · 2.1.269:
claude plugin eval+ JSON/HTML - code.claude.com/docs/en/skills: Evaluate and iterate; plugin eval vs skill-creator; formats not interchangeable
- code.claude.com/docs/en/plugins: plugin packaging context
- Local attempt:
claude plugin eval --help→ CLI not found ([GAP])