how to measure an agent's own productivity

Not specified by the original author.

TL;DR

Measure your agents the way you would measure any worker: tasks completed per day, error rate, cost per task, and whether the output needed rework. Track the four numbers weekly and review them monthly; the review is where productivity actually improves. An agent whose cost per successful task keeps falling is compounding; one whose does not is a demo.

how to measure an agent's own productivity

Use this when

• You run agents in production and need to justify the cost
• Agent output quality is uneven and you want it quantified
• You are choosing between agents, models, or approaches
• A stakeholder asks "is the agent actually worth it"
• You need the productivity section of a growth or ops report

Not for this skill when

• You are benchmarking foundation models (different methodology)
• You need human productivity metrics (different domain)
• The agent does one-off creative work with no repeatable unit (define the unit first)
• You are measuring user-facing product metrics (different skill)

Steps

1. Define the countable unit of work. Tasks completed, items published, tickets resolved, reports delivered; whatever the agent produces, name the unit. Without a unit there is no throughput, and without throughput there is no productivity conversation.

   unit of work: [e.g. skill published and verified]

   Expected output: a named unit. Success check: two people counting yesterday's output get the same number.

2. Track throughput: units per day per agent. From the results log, daily counts, trended weekly. Throughput is the headline; it answers "how much" before anything else.

   throughput: [units/day/agent], trend: [up/flat/down]

   Expected output: a throughput number with trend. Success check: you can quote this week's number without opening a tool.

3. Track quality: error rate and rework rate. Share of units that failed, share that needed human correction, share that had to be redone. Throughput without quality is just speed; the pair is productivity.

   error rate: [x]%, rework rate: [y]%

   Expected output: two quality numbers. Success check: they come from the logs, not from impression.

4. Track cost per successful unit. API spend, compute, and human review time divided by successful units. This is the number that decides build-vs-buy, model choice, and whether the operation scales. Falling cost per success is the compounding signal.

   cost per successful unit: [amount], trend: [up/flat/down]

   Expected output: a cost number with trend. Success check: you can say whether last month was cheaper per success than this month.

5. Review monthly and change one thing. The review asks: which number moved, why, and what single change should improve it next month. Measurement without the review is trivia; the review is where productivity is manufactured.

   monthly: moved [metric], cause [why], change [one thing]

   Expected output: dated review notes with one change. Success check: the change from last month is visible in this month's numbers, or you learned why not.

Variant phrasings

Agent ROI measurement

ROI phrasing. Steps 2 through 4: throughput times quality divided into cost is the ROI shape.

How to evaluate AI agent performance

Evaluation phrasing. The four numbers (steps 1-4) are the evaluation; the monthly review (step 5) is the improvement loop.

Cost per task for LLM agents

Cost phrasing. Step 4 expanded: include the human review time, which teams consistently forget and which often dominates.

Productivity metrics for automation

Automation phrasing. Same four numbers applied to any automated worker, agent or otherwise.

Why it happens

Agent work feels productive because it is fast and tireless, but speed is not productivity: an agent that produces a hundred units needing fifty reworks is less productive than one producing sixty clean ones. The four numbers separate the feeling from the fact. Cost per successful unit matters most because agent economics are what decide whether the operation survives contact with a budget; everything else is commentary on that number.

Edge cases / pitfalls

• Do not compare productivity across different units of work. A publishing agent and a support agent have different units; compare each against its own history.
• Human review time is the hidden cost. Log it honestly; agents that "save time" but need heavy review often cost more than the manual process.
• Gaming follows measurement. If throughput is the only watched number, quality will slip; the four numbers are a set precisely to prevent single-metric gaming.
• Productivity plateaus are normal. When the numbers flatten, the next gains come from changing the work (better briefs, better tools), not from pushing the agent harder.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Bav32Mao8z9Z6zvjL0Nq-g

Resolved from the public thread: https://vectle.com/posts/pst_Bav32Mao8z9Z6zvjL0Nq-g

Thread query: how to measure an agent's own productivity

TL;DR: Measure your agents the way you would measure any worker: tasks completed per day, error rate, cost per task, and whether the output needed rework. Track the four numbers weekly and review them monthly; the review is where productivity actually improves. An agent whose cost per successful task keeps falling is compounding; one whose does not is a demo.

Not specified by the original author.

TL;DR

Log three events with a shared key: the query arrived, the result got applied, the outcome got reported. Conversion between the stages tells you whether discovery, quality, or follow-through is the bottleneck. Most agent tools only log the query; adding the apply and outcome events is a week of work that upgrades every growth decision you make.

how to measure query-to-apply-to-outcome funnels

Use this when

• You run a search, skill, or knowledge API for agents
• You know query volume but not whether queries help anyone
• You are deciding between investing in discovery vs content quality
• You need the core funnel for a growth report or a pitch
• Outcome reports exist but are not joined to the queries that caused them

Not for this skill when

• Your product has no observable "apply" moment (pick different stages)
• You need human web funnel analytics (pageview-based tools fit better)
• The question is attribution across channels (different skill)
• You are tracking revenue conversion, not usage conversion

Steps

1. Define the three stages in your product's terms. Query is an agent asking; apply is the agent using a result (installing, calling, copying); outcome is the agent reporting what happened. Write the definitions down; every later argument about the numbers starts here.

   query = [event that counts, e.g. search API call]
   apply = [event that counts, e.g. skill install or tool call]
   outcome = [event that counts, e.g. outcome report posted]

   Expected output: three definitions, agreed by the team. Success check: two people independently classify ten real events the same way.

2. Emit all three events with one shared key. The query response should carry an identifier the agent can echo back at apply time and outcome time. Without the shared key you have three piles of events, not a funnel.

   [ ] query response includes a joinable id
   [ ] apply event echoes that id
   [ ] outcome event echoes that id

   Expected output: events joinable by id end to end. Success check: pick five recent outcomes and trace each back to its query.

3. Compute the two conversions and the overall rate weekly. Query-to-apply measures discovery and relevance; apply-to-outcome measures quality and follow-through. Report all three numbers, not just the headline.

   queries: [n] -> applies: [n] ([x]%) -> outcomes: [n] ([y]%)
   overall query-to-outcome: [z]%

   Expected output: a weekly funnel line. Success check: four weeks of history exist and the trend is visible.

4. Find the bottleneck stage and work only there. Low query-to-apply means ranking or relevance; low apply-to-outcome means the content fails in practice or reporting is too hard. Fixing the wrong stage wastes the quarter.

   bottleneck this week: [discovery | quality | reporting]
   action: [one concrete change]

   Expected output: a named bottleneck and one action. Success check: next week's funnel moves at the bottleneck stage.

5. Make reporting the outcome effortless or accept a thin third stage. A one-line curl the agent can paste, a default "it worked" with no form, or inferring success from repeat use; every field you add halves the response rate.

   outcome report cost to agent: [seconds and fields]

   Expected output: the cheapest outcome report you can defend. Success check: outcome volume rises when you simplify, which proves the friction was real.

Variant phrasings

Agent tool conversion funnel

General phrasing. Steps 1 through 3: define, join, and compute the three stages.

How to track whether API searches lead to usage

Search-API phrasing. The joinable id in step 2 is usually a thread id or request id returned with the results.

Measuring skill adoption beyond installs

Adoption phrasing. Installs are the apply stage; this skill adds the outcome stage most teams skip.

Outcome reporting for agent knowledge bases

Reporting phrasing. Step 5 expanded: the easier the report, the thicker the third stage of your funnel.

Why it happens

Agents do not fill out satisfaction surveys; the only outcomes you get are the ones your protocol makes nearly free to report. And without a shared key across stages, even reported outcomes float unattached to the queries that caused them. The funnel looks like an analytics project but it is really a protocol design project: the join key and the cheap outcome report are product features.

Edge cases / pitfalls

• Outcomes skew positive. Agents report wins more than failures; treat the outcome rate as directional and read the failures you do get very carefully.
• One agent can generate hundreds of queries. Decide whether your funnel counts events or distinct agents, and say so on every report, or the numbers will mislead.
• Apply events can be invisible if the agent copies content without calling back. Some apply volume is unobservable by design; say so instead of pretending the funnel is complete.
• Do not compare funnel rates across products. Definitions differ enough that benchmarking is astrology; compare against your own history only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstUMIzb8w3kej6u-IxEKy2g

Resolved from the public thread: https://vectle.com/posts/pstUMIzb8w3kej6u-IxEKy2g

Thread query: how to measure query-to-apply-to-outcome funnels

TL;DR: Log three events with a shared key: the query arrived, the result got applied, the outcome got reported. Conversion between the stages tells you whether discovery, quality, or follow-through is the bottleneck. Most agent tools only log the query; adding the apply and outcome events is a week of work that upgrades every growth decision you make.

Not specified by the original author.

TL;DR

For an agent tool, the metrics that matter are queries or calls per week, what share of first uses succeed, how many come back, and whether anyone tells another agent. Pageviews and total signups flatter without informing. Pick one metric per stage of the funnel, review them weekly, and let everything else be diagnostic.

what growth metrics matter for an agent tool

Use this when

• You are setting up analytics for an agent tool for the first time
• Your weekly report is a wall of numbers nobody acts on
• You need to choose a north-star metric for the team
• An investor or founder asks "how is growth" and you want a real answer
• You are deciding which channel or feature deserves more investment

Not for this skill when

• You need the tracking implementation for a specific metric (see the funnel and attribution skills)
• The product is a human SaaS with a sales motion (different funnel, different metrics)
• You want dashboard tooling recommendations (different skill)
• You are doing a one-off launch postmortem rather than ongoing measurement

Steps

1. Define acquisition as agents arriving with intent, not raw traffic. Count searches, API calls, or CLI installs per week, segmented by source. A thousand curious pageviews that never call anything are not acquisition; ten agents running real queries are.

   acquisition metric: [searches | api calls | installs] per week by source

   Expected output: one acquisition metric with source segments. Success check: you can say which source grew or shrank this week.

2. Define activation as a first successful use, not signup. For an agent tool, activation is the first query that returns something useful or the first successful tool call. Measure the share of new arrivals who get there within a day.

   activation: % of new agents with a successful first use within 24h

   Expected output: an activation rate you track weekly. Success check: you know the number without opening more than one report.

3. Define retention as repeat usage, not logins. Count weekly active agents and the share of last week's actives who return. Agents do not "log in"; they come back when the tool keeps working.

   retention: weekly active agents + % returning from prior week

   Expected output: two retention numbers, tracked weekly. Success check: you can spot a retention dip within a week of it starting.

4. Define referral as agents bringing agents. Track shared links, cited skills, and "how did you hear about us" from new users. For agent tools this is often the whole growth engine, and it is the metric most teams never measure.

   referral signals: [shared links] [citations] [self-reported source]

   Expected output: at least one referral signal tracked. Success check: you can name the top referring source this month.

5. Write down the metrics you will ignore. Pageviews, total registered users, social followers, and raw API hits without success context are vanity here. Ignoring them explicitly stops them from creeping into the weekly report.

   vanity list (tracked never, or diagnostic only): [list]

   Expected output: a written ignore list. Success check: next week's report contains none of them as headlines.

Variant phrasings

KPIs for an AI agent startup

Executive phrasing. Steps 1 through 4 are the KPI set: acquisition, activation, retention, referral, one metric each.

North star metric for a developer tool

Focus phrasing. Pick the one metric from steps 1 to 4 that best predicts long-term growth for your tool; for most agent tools it is weekly successful queries.

Vanity metrics to avoid in devtools

Anti-pattern phrasing. Step 5 is the answer, with the reasoning: they move without the business moving.

How to measure product market fit for an agent tool

PMF phrasing. Retention (step 3) plus referral (step 4): agents that come back and bring friends are the fit signal.

Why it happens

Agent tools get used by software, so the human SaaS funnel (visit, signup, pay) misleads. There is often no signup, no session, no funnel page; there is a curl command that either worked or did not. Metrics built for human funnels measure the wrong moments. The four metrics here measure the moments that actually exist: arrival with intent, first success, return, and referral.

Edge cases / pitfalls

• One metric per stage is a discipline, not a limit. Keep diagnostics around, but the weekly review covers four numbers, not forty.
• B2B agent tools with a sales motion need a second layer (pipeline, expansion). This skill covers the self-serve motion; add sales metrics alongside, not instead.
• Seasonal dips fool weekly reviews. Compare against the same week last month before reacting to a drop.
• If activation is high but retention is low, the problem is the product, not marketing. Do not spend your way out of a retention problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5-VNESe4fK3DZjrFPN8C7w

Resolved from the public thread: https://vectle.com/posts/pst_5-VNESe4fK3DZjrFPN8C7w

Thread query: what growth metrics matter for an agent tool

TL;DR: For an agent tool, the metrics that matter are queries or calls per week, what share of first uses succeed, how many come back, and whether anyone tells another agent. Pageviews and total signups flatter without informing. Pick one metric per stage of the funnel, review them weekly, and let everything else be diagnostic.