Third-party risk taught me to distrust feature lists
When I evaluate a vendor, I want to know what happens after the demonstration. Who investigates an incorrect result? What evidence can our team retrieve? How do we leave if the service no longer fits? Banking procurement and third-party risk reviews made those questions part of my routine. An AI vendor should be able to answer them using the workflow we intend to buy, rather than a list of available features.
Enterprise AI procurement needs the same maturity. A slide can confirm that a vendor offers connectors, audit logs and model choices. It cannot show how those pieces work together on your business question. Define the decision you intend to support before you compare products.
Give every vendor the same real scenario
Provide each vendor with the same sanitized scenario, source assumptions and evaluation criteria. Include an ordinary case, an ambiguous case and a request outside the user's authority. Ask for the evidence behind each response and the steps required to configure the workflow. Consistent inputs make differences interpretable.
Keep it bounded. A procurement evaluation should not become unpaid production implementation. A focused demonstration is enough to show how a vendor approaches the problem and which claims need a pilot. Separate what was demonstrated from what was proposed.
Include the people who will live with the result. In a health system that means a clinical informatics lead and a privacy officer who can judge HIPAA handling. In a bank it means the line of business owner and someone from second-line risk. Their questions during the demonstration are usually more revealing than the vendor's answers, and they will catch an access problem that a procurement scorecard cannot.
Ask what normal operation really costs
Years ago I led a team that built an internal enterprise data integration platform, well before managed pipeline products were mainstream. That experience made me permanently curious about the people behind a product. Who updates source mappings? Who investigates a disputed answer? What happens after a source or model change? A low license price is less attractive if routine operation consumes scarce internal specialists.
Examine support boundaries. A vendor may own the application while the customer owns permissions and data quality. That can be reasonable, but the handoff must be explicit. Ask both parties to walk through a failed query or stale source incident and say who acts first.
Compare commercial terms with expected use. Seats, workload limits, model usage, hosting and support each affect total cost differently. Have your commercial and legal owners review the actual agreement for your deployment.
Demand evidence for consequential claims
When a vendor claims a business outcome, ask how it was measured and whether the context resembles yours. A percentage improvement without a baseline, workload or time period is unusable. A smaller, well-explained result is worth more than a large unqualified one.
Match technical evidence to consequence. An access control claim should include a denied request, not a settings screen. A reliability claim should explain failure handling. A portability claim, including support for bring-your-own LLM or MCP, should identify what actually moves between providers and what must be rebuilt.
Separate shipped capability from roadmap. Record the version or deployment you evaluated. A promising roadmap can shape a relationship, but it should never silently become part of the accepted present-day requirement.
Close with an agreement the service owner can use
Use a decision matrix, but tie every score to observed behavior, remaining uncertainty and the cost of resolving it. Weight criteria by the intended workflow, and bring business reviewers, the CISO's team and operators to the final comparison. Each sees risks the others miss.
Ask the vendor to explain what evidence is unavailable during procurement and how a 90-day pilot will obtain it. Unknowns should become explicit acceptance conditions, not optimistic assumptions written into a statement of work.
End procurement with something the implementation team can use: the capability purchased, the conditions under which it must work, the evidence required for acceptance and the owner of each open question. Have the eventual service owner review it before signature. They inherit the support boundaries the day the sales process ends, and in a regulated bank or health system, they will be the one explaining it to the examiner or auditor.