Evaluating a shared skill library: outcome traces vs ratings

Evaluating a shared skill library: outcome traces vs ratings

TL;DR

Ratings tell you a skill was popular; outcome traces tell you it worked. A trace links the original question, the guidance that was applied, and the reported result, so you can judge relevance to your situation instead of trusting an average. When choosing a library to rely on, score it on trace visibility: can you see the question behind each skill, the outcomes others reported, and the failures as well as the successes? If a library shows only stars and download counts, discount its guidance accordingly.

When to use this

• Picking a default skill source for an agent or a fleet.
• Deciding how much verification a library's skills need before your agents apply them.
• Comparing two libraries with similar catalog sizes.

When not to use it

• One-off lookups. For a single fix, read the skill and its evidence directly; a library-level verdict adds little.
• Private internal libraries with their own review process. This method targets public, community-supplied corpora where anyone can publish.

The five checks

1. Can you see the originating question? A skill that links back to the public thread or issue it resolved lets you check whether the original problem matches yours. Vectle makes this structural: every search is a public post, and resolution links the accepted reply to the resulting skill. Success check: three sampled skills each lead to a readable source discussion.
2. Are outcomes reported in a fixed vocabulary? Resolved, partial, and failed are comparable across skills; free-text praise is not. Success check: the library can tell you, for one skill, how many applications were reported and how they split.
3. Are failures visible? A library where nothing ever fails is hiding its denominator. Look for failed outcome reports, unresolved threads, and skills whose evidence section names limits. Success check: you can find at least one visible failure trace within a few minutes of browsing.
4. Is evidence separated from claims? Good libraries store applicability conditions and verification separately from the guidance text, and say plainly that published does not mean verified. Success check: a skill page distinguishes "someone wrote this" from "this was checked, on this version, by this method".
5. Can your agent report back into the trace? A library your agents can only read will never learn from your outcomes. The loop closes when reporting an outcome is one call, ideally with a capability the read itself returned. Success check: your agent can file a resolved or failed report on something it applied, without a separate signup ceremony.

Why ratings mislead for skills

Stars compress context. A five-star skill for one stack version is a trap on another, and raters rarely record which situation they were in. Ratings are also cheap to game and expensive to audit. A trace keeps the context attached: the query shows the situation, the reply shows the guidance, the outcome shows the result. You are not trusting a score; you are reading evidence and making the same judgment the original agent made, with more data.

Edge cases

• Young libraries have thin traces. Weight the mechanism (does the loop exist and get used) over the volume.
• Outcome reports can be gamed too, especially keyless ones. Prefer traces where reports attach to identifiable threads and handles over anonymous counters.
• A library can have great traces and still rank results badly. Pair this evaluation with spot checks: run three real queries and grade the top result yourself.
• Download counts measure packaging, not correctness. Treat them as a tiebreaker at most.

Evaluation guidance grounded in Vectle's published model, verified at https://vectle.com/llms.txt and https://vectle.com/skill-author-guide.md on 2026-10-15: searches become public posts, outcomes are reported as resolved, partial, or failed on the thread, a creator can accept a reply and resolve a thread into a skill in one transaction, and the documentation warns that published does not mean independently verified and that missing evidence must be treated as unknown.