Start with the work, not the logo
When a CIO or CFO asks me to compare Claude and ChatGPT, I ask which piece of work they have in mind. A comparison is much easier to defend once the team has agreed on the task and what a satisfactory result looks like. I have seen selections revisited after the demonstration because those expectations were never made explicit. Testing the intended workflow gives procurement and the business owner something concrete to discuss.
So give the comparison a concrete job. An assistant for contract review has different requirements from a finance question-answering service, and even inside finance, explaining an approved report is a different risk from drafting a proposed journal entry. Name the workflow, the users, the authoritative sources and the decisions that stay with a human. Both Anthropic and OpenAI publish solid evaluation guidance. Treat it as a method for designing tests, not as evidence that either will meet your requirements.
Compare complete configurations, fairly
A model is never evaluated alone. You are testing a model version, a prompt, a set of tools, a retrieval setup and an output format together. If one candidate gets a curated source document and the other gets a noisy export, you are measuring preparation effort, not capability. Document each configuration and keep it with the results, the same way a model risk management team expects to see the inputs behind any validated model.
Allow provider-specific tuning, but make it visible. Forcing identical prompts penalizes a legitimate integration approach; unlimited undocumented tuning makes the result impossible to reproduce. Give each candidate a similar improvement budget, then score on held-out questions nobody tuned against. In the interest of transparency, dotSolved is an Anthropic Claude Partner Network services partner, which is exactly why our evaluations are built to be reproduced by a skeptical reviewer rather than to persuade in a meeting.
Score the failures that would hurt the business
Build the test set from the questions that cause trouble: ambiguous requests, permission boundaries, missing information. For a procurement assistant, include a vendor with two legal entities, an invoice without a purchase order and a request for data outside the user's region. Sometimes the right answer is a clarifying question, a restricted answer or a plain statement that the evidence is not there. A model that always answers is not the safer model.
Score correctness separately from polish. Does the answer cite the right evidence, preserve units and dates, honor access decisions and avoid invented explanations? A readable answer with the wrong reporting period fails, however much reviewers like the prose. In regulated industries this is not pedantry. An examiner or an audit committee will not grade style; they will ask whether the number was right for that customer on that day.
Put operations and contracts in the same review
Measure end-to-end latency, including retrieval and tool calls, and track cost per accepted task rather than cost per call. A cheap call that triggers several rounds of correction is expensive. A premium model on a narrow classification step may be waste. These tradeoffs belong in the decision record, because the CFO will ask about them at renewal whether or not you documented them.
Procurement should examine the service configuration, contract and data handling terms for the intended deployment. Record who owns credentials, usage limits, incident response and model upgrades. An API integration and an employee subscription are different purchases with different administration. Bring the third-party risk team in early and identify the evidence it needs, so a successful evaluation does not arrive at procurement with unanswered questions.
Approve a scope, not a permanent winner
The outcome rarely needs to be one enterprise winner. One configuration may be approved for document analysis, another for structured extraction and neither for autonomous changes to business records. Write down the approved scope, the known weaknesses and the triggers for re-evaluation. A material model update, a new data source or new action permissions should reopen the evidence rather than inherit an old approval.
Bring the review team an actual task, a few difficult examples and the constraints the deployment must meet. Compare candidates on accepted answers, verification effort and operating burden. I prefer a bounded evaluation with a named business owner and scheduled reviews, so the comparison leads to a decision. Document the uncertainty that remains and which result would cause the team to reconsider its choice.