Local is a boundary decision, not a hardware purchase
Before approving inference hardware, I ask the team to explain the boundary it needs to preserve. I have seen concerns about an external API turn into a server purchase before that question was settled. A workstation, an internal data center and a dedicated cloud environment solve different problems. We need to know which requirement matters and who will keep the service running before choosing among them.
Start by naming the boundary that matters: network connectivity, physical custody, data residency, latency or independence from a hosted endpoint. In healthcare and life sciences, the honest driver is often HIPAA obligations and how a covered entity can demonstrate them. In a bank, it is usually what you can show an examiner. Either way, the architecture decision should specify the requirement rather than relying on the word local.
Trace every place the data goes
Running the model nearby does not contain the rest of the workflow. Documents may arrive from a SaaS application, embeddings may be generated elsewhere and monitoring may ship diagnostics outside the environment. Draw a data-flow diagram covering ingestion, storage, inference, tools, logs and backups, then verify each crossing against actual access and retention controls.
This lesson is not new to AI. When we modernized a bank's data platform, the questions that consumed the most leadership time were never about the target platform. They were about every replica, extract and log that had accumulated around the old one. Local inference inherits the same discipline. The model is the easy part to locate; its supporting services are where boundaries quietly leak.
Size the workload before the equipment
Describe who will use the service, when demand peaks, how long inputs are and how much output people need. Short classification requests behave nothing like long documents processed while several users wait for streaming responses. Measure a representative workload instead of multiplying a vendor throughput figure by an assumed headcount.
For an internal policy assistant, assemble ordinary questions plus a few long, difficult requests, and run them at the concurrency you expect during a busy hour. Record waiting time as well as generation time. If one large request makes everyone else wait, you need a scheduling or capacity decision, not a reassuring average latency chart.
Plan for growth honestly. Pilots attract users quickly once people find them useful, and a local deployment sized for a pilot team can become a bottleneck within a quarter. Write down the trigger for adding capacity, and the trigger for moving the workload elsewhere, before demand forces the decision.
Assign ownership and test access together
A successful installation is a milestone, not an operating model. Name who owns patching, model updates, certificates, hardware incidents and user support, and who can approve and reverse an update. A workstation that sleeps or travels is fine for experiments and wrong for a shared business dependency. A highly available cluster for an occasional offline task is waste. Match the environment to the consequence of interruption.
Hold the local deployment to the same evidence standards as a hosted one: factual support, access boundaries, missing information and refusals. Model placement does not enforce permissions. When departments share a serving environment, retrieval and tools must carry each user's authority, and tests should prove one user's context never appears in another's result. Review caches and temporary files as part of that boundary. Security first means this is settled during design, not during the pilot.
Right-size the decision to the organization
Compare the options using the stated boundary and the available operating team. A dedicated hosted environment may meet the requirement without an internal hardware estate; a workload with different custody or connectivity requirements may justify local infrastructure. Have the service owner explain the proposed choice, including support coverage and recovery. The architecture has to remain workable after the evaluation team returns to its other responsibilities.
Before expanding use, assemble a small acceptance package: reproducible configuration, representative load results, quality evaluations and a recovery exercise that measures the real time to restore service after a node failure or a reverted update. A deployment that only works while its original engineer is in the room is not ready. Then compare local, hosted and hybrid options against the stated requirement, and record which constraint the choice satisfies and what operating work you are accepting.