Audit agent workflow analytics

When to use

Use before designing or troubleshooting adoption, CLI version, knowledge discovery, or task lifecycle dashboards backed by agent telemetry.

Establish the measurement contract

1. Resolve the live data source and inspect its populated records, aggregate queries, capture dates, and displayed labels. A schema or successful HTTP response does not prove a metric is populated or visible.
2. Define the unit for every metric: human account, installation, agent persona, task, request, skill, or skill version. Installation ownership does not identify unique humans. Multiple personas can share one installation. Registrations do not measure package downloads.
3. Define active with a named observed action and period. Authorization, credential use, completed requests, and published contributions measure different things.
4. Count each completed request once for request volume. Deduplicate logical outcomes by durable operation identity. Do not sum transport starts, finishes, lifecycle events, and operations into an unlabeled activity total.
5. Verify attribution joins. Versioned requests without installation identity support request volume, but not unique installations. Latest observed contribution versions support a narrower inventory estimate with freshness and an unknown bucket. Count installation-client memberships independently from the latest client observation; one client must not replace another in a membership breakdown.
6. Distinguish computed recommendations, response suggestions, full reads, and reported application. A background matching run proves only that candidates were computed. Do not call it guidance received, used, or useful without corresponding evidence.
7. Model workflow progress as branching and repeatable. Skill creation, skill updates, and an explicit no-skill conclusion can each be valid outcomes. Preserve confirmed publication milestones after subsequent replies. Keep current status separate from milestones ever reached.
8. Use one task cohort for stage comparisons, count distinct tasks, label overlapping milestones, and show the measured subset. A small observed subset must not imply conversion across a much larger historical cohort.
9. Treat complete zero, partial measurement, unmeasured history, and unavailable sources as different states. Missing fields must not silently become zero. A zero denominator is not a zero-percent success rate. A replay or delayed observation must not erase an attained milestone.
10. Compare published package versions, registry tags, and the signed service recommendation independently. Rollout thresholds express configured availability; adoption and effective eligibility require separate evidence. Multiple policy revisions can refer to one package version.

Verify what people can see

1. Inspect each rendered widget at normal desktop and narrow viewport sizes, including all supported time windows. Read text, inspect screenshots, and check the browser console.
2. Compare the database result, parsed metric model, and rendered value. Distinguish missing source data, dropped fields, misleading labels, and content hidden by layout.
3. For time series, confirm nonzero buckets are visible without an undisclosed horizontal scroll. Prefer a responsive chart over a fixed minimum width for every date. Preserve the full selected period and use readable ticks, legends, totals, and accessible values.
4. Keep calendar-date buckets in their declared timezone. Avoid interpreting a date-only bucket in the viewer's timezone. Format timestamps with a stable server and initial client representation, then localize intentionally if needed.
5. Verify that list limits have explicit continuation and that total counts are computed before pagination. Test more rows than the visible limit.
6. Validate source payloads before display. Degrade only affected metrics when a field or source fails. Preserve public-read controls and privacy rules when using cached results.

Regression and release evidence

Use deterministic fixtures for exact metric expectations and live read-only reconciliation for current behavior. Test sparse series, a final-day-only spike, empty measured periods, missing fields, partial discovery, overlapping outcomes, multiple clients per installation, replayed observations, source outages, and timezone differences. Assert that all intended widgets render their values and that there are no hydration errors. Headings-only checks and successful builds do not establish dashboard correctness.

Deliverable

For every widget, record its unit, period, source, join, deduplication rule, coverage, uncertainty, and acceptance check. Separate verified findings from planned repairs. Do not invent historical event detail to fill a chart.