Usage charts describe curiosity, not value
A rising usage chart makes me ask what happened to the work. Did a review take less time? Were fewer answers sent back for correction? A user sending five prompts may be engaged, or may still be trying to get one usable result. I look at logins and prompt counts when reviewing adoption, but I need the workflow outcome before I can judge whether the program is worth expanding.
I treat usage as diagnostic context, nothing more. The measure that matters is a defined workflow producing a result the business owner already values, and that a CFO would accept as evidence.
Weak measurement is why programs stall between stage two and three
I describe enterprise AI maturity in five stages: experimentation, departmental wins, platform foundation, operationalized AI and an AI-native organization. Most organizations believe they are at stage three. Very few are. The gap between departmental wins and a platform foundation is where I have watched companies lose twelve to eighteen months.
The cause is alignment, not technology. A department ships something useful, leadership gets excited, and the next investment stalls because nobody can show what the first win produced. The CIO wants a platform, business units want more quick wins, and no one owns data governance or the evidence. Fixing measurement early is one of the cheapest ways to shorten that stall, because it gives every side a shared fact base.
Define the unit of work and measure the old process honestly
Define one completed unit before collecting a baseline: a weekly account review, a reconciled variance explanation, an approved answer to a recurring regulatory reporting request. State what completion requires, including human review. A draft that still needs an analyst to rebuild the calculation is not a decision-ready answer. Do not roll different units into one enterprise "AI productivity" number unless you could defend the aggregation in front of an audit committee.
Then observe real tasks done the current way: elapsed time, hands-on time, waiting time, rework and handoffs. An eleven-day queue with twenty minutes of analysis is a different problem from six hours of manual reconciliation. In banking and healthcare I have usually found waiting time to be the larger cost, and it never appears on a usage dashboard. Note the observation window and its limits. Never use a retrospective estimate chosen because it flatters the new process.
Count verification time and test with ordinary users
Start the clock when the user begins and stop it when the result is ready for the decision. Include source checks, clarification and correction. Track failures and escalations. If a fast first answer triggers a long review, the business has gained nothing.
Compare like with like. A pilot run by an expert with unusually broad permissions will not predict what a branch operations lead or a claims analyst will experience. I insist on at least one participant whose access and experience match the people who will use the workflow after expansion.
Measure in weeks and watch for voluntary reuse
I will take a ninety-day pilot with a small, explicit scorecard over a year of vendor demonstrations every time. Within a few weeks you can see whether people come back after the launch novelty fades. Pair that with a short note from each user: what did it help accomplish, and what still needed another tool or person?
Voluntary reuse counts only where users have a reasonable alternative, because mandated usage hides dissatisfaction. A practical scorecard tracks completed workflows, time to a reviewed answer, escalations, and the share of sampled answers meeting the agreed evidence standard, with numerators and denominators visible. Add cost when it is material.
Measure the work, not the people, then decide
Collect the minimum data needed. Team-level observations usually answer the adoption question without retaining individual prompts or ranking employees. Tell participants what is measured and why. If someone later wants the data for performance reviews, make that a separate, deliberate decision. In regulated industries this is not only good culture; it keeps the program on the right side of privacy and employment obligations.
At the review point, expand, revise or stop. Expansion needs named owners and a support plan. Revision names the limiting issue. Stopping preserves what was learned. That discipline is how a departmental win becomes the first proof point for a platform, and how an organization actually moves up a stage rather than claiming it has.