Audit units, attribution, and coverage before designing agent workflow analytics
Use when designing dashboards for agent tools, CLI adoption, knowledge discovery, or branching task outcomes. Establish metric units, attribution coverage, event deduplication, and outcome semantics before choosing charts.
Audit agent workflow analytics
When to use
Use before designing adoption, CLI version, knowledge discovery, or task lifecycle dashboards backed by agent telemetry.
Procedure
- Resolve the actual data source and inspect its live schema, current aggregates, and capture dates. Read the existing dashboard queries as well as the visual labels. Schema availability alone does not establish populated coverage.
- Define the unit for every metric: human account, installation, agent persona, task, request, skill, or skill version. Installation-scoped owner groups are not unique humans. Multiple personas can share one installation. Do not infer downloads from successful registrations.
- Define active with an observed action and a time window. Authorization state and a recently used credential measure different things. Preserve the distinction between tool execution and substantive contribution activity.
- Count completed requests once when computing call volume. Deduplicate retries by durable operation identity when computing logical outcomes. Do not combine transport events, lifecycle events, and operations into an unlabeled activity total.
- Verify attribution joins. Versioned requests without installation identity support call volume by version, but not distinct installations by version. Last-observed contribution versions support a narrower inventory estimate with a freshness timestamp and an unknown bucket.
- Distinguish a generated recommendation, a returned suggestion, a full read, and a reported application. A background similarity result proves only that a candidate was computed. Do not label it used or useful without the corresponding evidence.
- Model task progress as branching and repeatable. A new skill, an updated skill, and an explicit decision that no skill is needed can each be valid outcomes. Preserve server-confirmed publication alongside later activity and agent-declared assessments. Conversation access state is not execution state.
- For stage comparisons, choose one task cohort and count distinct tasks reaching each milestone. Label overlapping milestones. Keep current status separate from ever-reached milestones. Missing telemetry and elapsed silence remain unknown unless an explicit completion or failure observation exists.
- Compare published package versions, registry channel tags, and the service recommendation independently. Display their source and observation time.
Deliverable
For each proposed chart, record its unit, time window, source, join, deduplication rule, coverage start, and uncertainty. Mark whether it is supported now, derived from partial observations, or requires new instrumentation. Do not invent historical event detail to fill a chart.