how to measure an agent's own productivity

TL;DR

Measure your agents the way you would measure any worker: tasks completed per day, error rate, cost per task, and whether the output needed rework. Track the four numbers weekly and review them monthly; the review is where productivity actually improves. An agent whose cost per successful task keeps falling is compounding; one whose does not is a demo.

how to measure an agent's own productivity

Use this when

• You run agents in production and need to justify the cost
• Agent output quality is uneven and you want it quantified
• You are choosing between agents, models, or approaches
• A stakeholder asks "is the agent actually worth it"
• You need the productivity section of a growth or ops report

Not for this skill when

• You are benchmarking foundation models (different methodology)
• You need human productivity metrics (different domain)
• The agent does one-off creative work with no repeatable unit (define the unit first)
• You are measuring user-facing product metrics (different skill)

Steps

1. Define the countable unit of work. Tasks completed, items published, tickets resolved, reports delivered; whatever the agent produces, name the unit. Without a unit there is no throughput, and without throughput there is no productivity conversation.

   unit of work: [e.g. skill published and verified]

Expected output: a named unit. Success check: two people counting yesterday's output get the same number.

1. Track throughput: units per day per agent. From the results log, daily counts, trended weekly. Throughput is the headline; it answers "how much" before anything else.

   throughput: [units/day/agent], trend: [up/flat/down]

Expected output: a throughput number with trend. Success check: you can quote this week's number without opening a tool.

1. Track quality: error rate and rework rate. Share of units that failed, share that needed human correction, share that had to be redone. Throughput without quality is just speed; the pair is productivity.

   error rate: [x]%, rework rate: [y]%

Expected output: two quality numbers. Success check: they come from the logs, not from impression.

1. Track cost per successful unit. API spend, compute, and human review time divided by successful units. This is the number that decides build-vs-buy, model choice, and whether the operation scales. Falling cost per success is the compounding signal.

   cost per successful unit: [amount], trend: [up/flat/down]

Expected output: a cost number with trend. Success check: you can say whether last month was cheaper per success than this month.

1. Review monthly and change one thing. The review asks: which number moved, why, and what single change should improve it next month. Measurement without the review is trivia; the review is where productivity is manufactured.

   monthly: moved [metric], cause [why], change [one thing]

Expected output: dated review notes with one change. Success check: the change from last month is visible in this month's numbers, or you learned why not.

Variant phrasings

Agent ROI measurement

ROI phrasing. Steps 2 through 4: throughput times quality divided into cost is the ROI shape.

How to evaluate AI agent performance

Evaluation phrasing. The four numbers (steps 1-4) are the evaluation; the monthly review (step 5) is the improvement loop.

Cost per task for LLM agents

Cost phrasing. Step 4 expanded: include the human review time, which teams consistently forget and which often dominates.

Productivity metrics for automation

Automation phrasing. Same four numbers applied to any automated worker, agent or otherwise.

Why it happens

Agent work feels productive because it is fast and tireless, but speed is not productivity: an agent that produces a hundred units needing fifty reworks is less productive than one producing sixty clean ones. The four numbers separate the feeling from the fact. Cost per successful unit matters most because agent economics are what decide whether the operation survives contact with a budget; everything else is commentary on that number.

Edge cases / pitfalls

• Do not compare productivity across different units of work. A publishing agent and a support agent have different units; compare each against its own history.
• Human review time is the hidden cost. Log it honestly; agents that "save time" but need heavy review often cost more than the manual process.
• Gaming follows measurement. If throughput is the only watched number, quality will slip; the four numbers are a set precisely to prevent single-metric gaming.
• Productivity plateaus are normal. When the numbers flatten, the next gains come from changing the work (better briefs, better tools), not from pushing the agent harder.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Bav32Mao8z9Z6zvjL0Nq-g