Spectacular demos teach you very little
A credit memo makes an impressive AI demonstration. Whether it makes a useful first pilot depends on how often the team prepares one and how well it can judge the result. In selection meetings, I look for work we can observe repeatedly during the pilot. A less dramatic task that happens every week often gives us more to learn from than the demonstration everyone remembers.
I look for a decision that recurs every day or every week. That gives the team several chances to compare the new workflow with the old one inside a quarter, including weeks when the inputs are awkward or the usual reviewer is unavailable. The pilot can then tell us about routine operation as well as the best result the team can produce.
Inventory the decisions that recur
List the decisions teams make on a daily or weekly rhythm. In a bank that might be exception review on loan documentation, preparing a relationship review for a commercial client or explaining a movement in the deposit pipeline. In a life sciences company it might be triaging supplier deviations or summarizing a batch record for a quality reviewer. Estimate how often each occurs, how many people touch it, and whether the team can recognize a good answer without inventing a new measurement process.
Frequency is necessary, not sufficient. A frequent task with severe consequences and weak evidence is a poor first choice. Credit approval recurs constantly at a community bank, but it carries regulatory and fair lending exposure that a first pilot should not absorb. Preparing the analyst summary that feeds that decision is a far better candidate. Aim for recurrence, real friction and a manageable review boundary.
Score candidates against real constraints
For each candidate, document the data, the business owner, the reviewer and the current elapsed time. Separate waiting time from hands-on effort. If a question takes three days because the approval committee meets on Thursday, a faster answer will not remove the delay. I have watched teams promise cycle time reductions that the calendar, not the analysis, made impossible.
Compare an account review brief with an autonomous purchasing action. The brief draws on approved reports, and a manager checks it before a meeting. The purchasing action brings supplier terms, spending authority and execution risk. Both may be technically possible. The brief produces clearer early evidence at far lower organizational cost.
Check source stability. A workflow resting on a spreadsheet with shifting columns can consume the pilot in data repair. That work may be valuable, but fund it and name it as preparation. Do not let it hide inside the model's performance numbers.
Protect the pilot from selection bias
Pull evaluation questions from recent real work, not from a list of things the system already handles. Include routine cases, awkward exceptions and requests that should be refused. If reviewers pick examples after seeing answers, you have a demonstration with a score attached.
Hold back part of the question set for a later review, and keep reference answers under business ownership with a change record. An improved answer may expose an error in the old report. Investigate that possibility rather than using it to explain away every disagreement.
Count interventions as well as answers. If a specialist rewrites each question or repairs records during the session, record that effort. The workflow may still be worth it, but its operating cost differs sharply from an unattended one.
Write the pilot in one sentence, then size it
A good selection meeting ends with one sentence: “For this team, during this reporting period, we will test whether this assistant helps this owner make this recurring decision with acceptable evidence and effort.” If the sentence turns into a paragraph of exceptions, narrow the scope.
Set the expansion rule in advance, using thresholds drawn from the business baseline rather than a borrowed benchmark. Keep the rejected candidates and the reasons, so the next planning meeting does not reopen the same debate without new evidence. A pilot can legitimately end with a decision to improve data first.
Check reviewer capacity before approving parallel pilots. If several teams depend on the same finance specialist or security reviewer, their schedules are linked even when their technology is separate. I would rather finish one credible evaluation than divide the people needed to judge three. Keep the next candidate ready, with its prerequisites recorded, so the team can move on when the first decision is made.