Source conversation · open
Measuring CLI adoption and skill-creation reliability
I am investigating an observability gap in a developer-facing agent CLI. Conversations appear to start, but skill creation is not visible enough to establish whether the feature works across developer machines. I am reviewing the existing lifecycle, logging, persistence, and client health signals to identify what is measurable today, what is missing, and how to design a privacy-safe funnel with reliable delivery and failure attribution.
Verified finding: the current system has strong private setup and installation event telemetry plus a useful local recovery journal for knowledge mutations, but the knowledge HTTP routes do not emit an equivalent request and response event stream. Durable tables can show task openings, replies, submissions, and publication outcomes, yet they cannot distinguish a model omission from missing lifecycle context, a transport failure, a rejected request, or a later review outcome. A useful design is a privacy-safe funnel with separate opportunity, context-delivery, tool-call, admission, settlement, and host-health signals joined by bounded correlation identifiers.
Proposed observability plan: add a content-free conversation lifecycle state machine with bounded phases such as opened, exploring, solution_found, skill_candidate, proposal_submitted, review_pending, published, rejected, and no_skill_reasoned. Record immutable transition events with sequence, source, confidence, harness and observed model metadata, while exposing only a redacted current projection. Keep the agent's decision separate from server truth: a candidate or proposal is not a published skill until review and publication succeed. Add aggregate funnel metrics for eligible conversations, candidate decisions, submissions, review outcomes, publication latency, and conversations ending without a skill. This makes missing submissions distinguishable from pending review, rejection, or legitimate no-skill outcomes without storing prompts or raw content.