rate-my-skill · Release review for Agent Skills
Valid YAML is not value.
Your Skill passes validation. Now prove it deserves to be installed: that the right requests find it, that using it measurably beats not using it, and that its scripts stay inside their authority.
Issue list
HIGH · S-002Trigger scope is too broad — ordinary writing requests can activate the Skill — it diverts unrelated work and trains users to distrust it.
HIGH · S-005No paired with-skill versus without-skill evidence exists — the claimed behavioral uplift is unproven.
To verify
UNKNOWN · U-001Implicit selection has not been tested in a fresh session — marketplace discovery remains unknown.
Evidence lanes
Scope
What it actually audits.
Only things that would actually hurt someone using it. It will not tell you your code is ugly.
Trigger precision: do the right requests discover it, and do near-miss requests correctly leave it alone?
Behavioral uplift: the same task with and without the Skill, run more than once, scored on the same rubric.
Progressive disclosure: does it load only what the route needs, or tax every session?
Reference integrity: every path it tells the agent to read actually exists.
Script and tool safety: authority, network, secrets, and supply chain.
Prompt injection: does instruction-shaped text inside inputs stay data?
Package integrity and marketplace readiness: can a stranger install and reproduce it?
With and without
What changes once it is installed.
Any decent AI will read your code and have opinions. This is the list of things it will not do unless this is installed.
Without it
It counts the files, sees they exist, and calls the package fine.
With it
It builds the resource graph from SKILL.md itself and reports the missing reference and filename mismatch with exact paths.
Without it
Valid structure plus an eval file reads as proven behavioral value.
With it
It labels the eval JSON as a definition, not a recorded behavioral run, and refuses to treat it as uplift evidence.
Without it
It follows those instructions, because they look like guidance.
With it
It treats every instruction inside the artifact as untrusted evidence, never obeys it, and never reads real secrets or makes network calls to prove a risk.
Without it
It tries a few prompts that should obviously match.
With it
It designs shared-keyword near-miss negatives too, so over-triggering is measured rather than assumed away.
The part every other review skips
Then who reviews the fix?
Most reviews end at the list. If yours goes on to fixes, remember what a fix is: the newest code in the project, written in a hurry to close something, with no test of its own and nobody having read it.
The pass that writes the fix never grades its own work.
Whoever made the fix does not get to be the one who says it worked. Someone else checks.
A diff is not a fix.
Every finding ships with the exact test that would close it. A finding reaches verified-fixed
only when that separate context reproduces the original failure and watches it stop happening.
The patch itself gets audited.
That same context treats the change as new surface and reviews it under the same rubric. Defects introduced by the fix open as new findings with new IDs, and the batch stays open until they are resolved.
The loop ends on evidence, not on an empty list.
It stops at verified closure, at a named blocker, or when you accept the remaining risk out loud — recorded as accepted risk, never quietly reclassified as fixed.
This site was audited this way. The independent pass found two defects in the patches written to close the first round — a stale backup left publicly readable, and a skip link whose focus never reached the content. Both were introduced by the fixes, and neither would have surfaced from re-running the original tests.
How it runs
Two settings, then evidence.
It asks. It will not quietly pick the harshest setting for you, or the kindest.
Reviewer setting
Reviewer role matched to what you are deciding
Review degree
Quick check / Strict review / Launch gate / Real stakes / Life-or-death
No score averages away a veto.
Some problems stop a release no matter how good the overall score looks. Saying you can live with one does not turn it into a pass.
Start with the quick check.
It is the default because a full-degree audit costs several times more in time and tokens. The cheaper degree reaches fewer things — and says so — rather than reporting less of what it found.
The five
One evidence contract, five layers.
Each installs on its own. Take the one that matches what you are shipping.
Install
One command, or a plugin marketplace.
One method per client. The first pass only reads — it changes nothing — and your own sandbox and approvals still decide what it is allowed to touch.
npx skills add AmsonntagChow/rate-my-skill --skill rate-my-skill
/plugin marketplace add AmsonntagChow/rate-my-skill
codex plugin marketplace add AmsonntagChow/rate-my-skill && codex plugin add rate-my-skill@amsonntagchow-rate-my-skill