The best evaluation environment is already on your calendar
I like using the weekly operations meeting to evaluate an AI workflow. It already has recurring questions, people who know the work and decisions that need to be made. That gives us something more useful than a fresh set of demonstration prompts. We can compare preparation effort, look at which answers survive discussion and see whether unresolved questions come back the following week.
That meeting asks the same questions every week. What changed? Which exceptions need attention? Who owns the next action? Repetition gives you a baseline that no vendor demo can provide, and the same people return each week to judge results against real decisions.
Test one slice of the agenda
Do not replace the meeting. Pick one part of the agenda, such as preparing the exception list for one operational metric, and keep the existing report available. The first goal is to compare usefulness and evidence, not to make people abandon a process they understand.
Tell participants what is being tested and how their feedback will be used. An evaluation should never depend on hiding where an answer came from. Reviewers need to know when they are inspecting experimental output and which decisions still follow the normal approval path.
Choose a meeting whose owner actually wants better answers. In healthcare operations, banking service teams and high-technology support organizations alike, the strongest pilots I have seen were pulled by an operations leader with a recurring pain, not pushed by a central AI team looking for a use case. Workflows over features: the assistant earns adoption by fitting work people already do every week, under an owner who will tell you plainly when it does not help.
Prepare the comparison before the meeting
Collect the recurring questions and the reference material used to answer them. Record the source cutoff and known limits. If the assistant reads a different snapshot from the existing report, you will measure timing differences, not answer quality, so align the evidence or label the gap.
Ask the business owner to name expected exceptions in advance, including one ambiguous case and one normal case that should not trigger attention. That keeps one impressive explanation of an obvious issue from dominating the review.
Assign a note-taker. The meeting lead is busy making decisions. A simple log of question, answer, evidence, reviewer judgment and next action is enough to support a real retrospective.
Watch how the answer changes the conversation
Ask whether participants can inspect the basis for each output and whether it helps them decide what to do next. If it just restates the dashboard in better prose, say so. Fluency is not decision support.
Count hidden reconstruction work. If an analyst has to open three systems to confirm each statement, that effort belongs in the evaluation. A faster first response can coexist with a slower path to a trusted conclusion.
Record disagreements without forcing consensus. A disputed definition or surprising source result is often the most valuable finding of the week. Assign an owner to investigate after the meeting, and capture decisions deferred for lack of evidence. Those deferrals are outcomes, not blanks to tidy away. Keep the action register under the meeting owner's control, with open questions separate from approved follow-up.
Measure in weeks, then decide how to grow
Run this inside a 90-day pilot with metrics agreed up front: preparation effort, verification effort and clarity of assigned actions. That will tell you more than a year of demos. After several cycles, look for patterns. Which question types help? Which need constant correction? Which expose missing data or ownership? Those patterns are a better expansion plan than a sentiment survey, and they may argue for narrowing scope as readily as widening it.
Resist claiming business outcomes that have not happened yet. A better exception discussion may eventually improve operations, but that effect needs its own measurement and an honest account of everything else that changed.
Before the next meeting, choose one recurring question and ask its owner to prepare both the usual answer and the assisted version. Compare what each required and record the corrections made during discussion. Repeat with the same question over several meetings. That gives the team a practical record of whether the workflow is improving, including the weeks when the data or the answer is difficult.