Define the unit a CFO can evaluate
A GPU business case needs a unit the finance team can connect to useful work. I prefer the cost of a completed, acceptable task: for example, a document that passes review or an answer supported by the required evidence. Tokens per second still matters to capacity planning. It just does not tell a CFO how much useful work the proposed investment will deliver once retries and human review are included.
Use that unit to compare alternatives. A configuration that generates text quickly but needs repeated human correction can have a poor cost per accepted task, while one that uses more compute per request may cut rework. Record both effects. Do not present a throughput benchmark as a return-on-investment calculation unless the link between that benchmark and the actual workflow has been demonstrated.
Build the demand profile you will actually face
Collect the distribution of input length, output length, concurrency and arrival patterns. Averages mislead. Users arrive together after a meeting, and a scheduled document batch competes with interactive requests. In banking we learned this with month-end close and regulatory reporting cycles: capacity sized for an ordinary Tuesday fails on the day that matters most. Separate predictable batch work from time-sensitive work and protect the service expectations that matter.
For a finance assistant, test short metric questions alongside long variance explanations and requests that require several tool calls. The model server may sit idle while a source system responds, then absorb a burst of follow-ups. End-to-end measurement shows whether more GPUs would actually help, or whether the bottleneck sits in retrieval, authorization or the underlying application. Buying accelerators to fix a slow ERP query is an expensive way to learn that.
Validate the supported configuration
Hardware selection must match the actual model, runtime and supported deployment profile. NVIDIA publishes NIM documentation and a support matrix; check the intended package against it. A familiar GPU name does not establish compatibility, and a benchmark from another precision, sequence length or parallelism setting will not transfer unchanged to the planned service.
Preserve the configuration behind every performance result: model revision, serving software, precision, request distribution and concurrency. Record warm-up and sustained behavior, not a short best-case run. This makes the capacity conversation reproducible and lets future teams separate hardware limits from changes introduced by a model update or a serving adjustment.
Treat vendor reference numbers as a starting hypothesis, not a forecast. Published figures are usually measured under favorable, well-tuned conditions that your workload may never resemble. The question a board or CFO should ask is not how fast the hardware can go, but how fast it went on our representative requests, at our concurrency, with our model revision. If the team cannot answer that, the purchase decision is premature.
Count the costs that survive low utilization
The economic case must include hardware or rental, power where relevant, networking, storage, operations staff and replacement capacity. It must also include idle time. Capacity purchased for a short daily peak may sit underused most of the day. That can still be the right decision, but the availability requirement has to justify it, and the board or audit committee should see that reasoning rather than a utilization surprise.
Consolidation needs care. Shared capacity improves utilization but creates noisy-neighbor effects and complicates access control, which matters when a wealth management workload and a customer-facing service share hardware. Set workload priorities, admission limits and explicit behavior when capacity runs out. A queue is a product decision because users feel its delay. Deferring low-priority work deliberately beats letting everything degrade unpredictably.
Prove it at small scale before committing
Use the expected workload to compare hosted, reserved and owned capacity. An internal GPU platform needs enough useful demand to justify its idle time and operating staff. If several teams will share it, decide how they get capacity during contention and how costs are assigned. Put those assumptions in the business case so finance can see what changes if demand arrives more slowly than expected.
Whatever the size, approve an experiment before a large commitment. Name the workload, candidate configuration, evaluation period and thresholds for expansion, plus a stopping condition if quality or response time does not improve enough. Keep measured results separate from estimates, and current vendor quotes separate from reusable reasoning. Have infrastructure and application owners review the inputs together, then record the uncertain assumptions and the load test that will check them.