Answer the question regulated buyers actually ask
In conversations with CISOs and healthcare technology leaders, the practical question about open models is often whether the organization can use them without sending data to an external API. I want to answer that directly, then examine the license and operating requirements. Calling a model open leaves too much unstated. The decision depends on which artifacts the team receives, what it may do with them and what it must operate itself.
A downloadable model is not automatically an open-source AI system. The Open Source Initiative's definition addresses the freedoms and materials needed to exercise them, and weights are one part of that. Inventory what you actually get: source of the weights, license, tokenizer, inference requirements, evaluation information and use restrictions. Note which artifacts you can retain internally and which still depend on a hosted service. This inventory is useful even if you end up buying commercially.
Name the capability you need to own
Teams pursue open weights for different reasons: a deployment boundary they control, a narrow task to adapt, reduced dependence on one endpoint, or operation where connectivity is limited. Write the requirement down first. Controlling where data lives is not the same as needing to modify training behavior or redistribute a derivative. In genomics and diagnostics work, the driving need is usually custody and auditability of sensitive data, not fine-tuning.
The model's practical traits matter more than its benchmark rank. Long context matters when the work is contracts, multi-document review or reconstructing an audit trail. Mixture-of-experts architectures can deliver useful capability at lower serving cost. For a support team classifying tickets, a smaller model that consistently applies the right categories beats a large general model that produces elaborate, inconsistent explanations. Select for the task boundary you defined, not for headlines.
Evaluate the application, not the model card
Test representative input lengths, languages, formatting and incomplete records, and include cases where the correct outcome is to abstain or escalate. Weight errors by consequence: misrouting an urgent incident is not the same as a topic label a person fixes in seconds. One aggregate score can hide the weakness that dominates your risk. Anyone who has sat through model risk management validation in a bank will recognize the discipline, and it applies here too.
Keep the exact model revision, quantization format, serving engine and prompt together with each result. A score from one package does not transfer to another because the family name matches. Rerun acceptance tests after changing runtime or precision settings. Those changes are often operationally attractive, and they are also exactly where quiet regressions appear.
Assign ownership of that evaluation record from the start. It is the evidence your CISO, your model risk team and any examiner or auditor will ask for later, and it is far harder to reconstruct than to keep.
Budget for the service around the weights
Owning deployment means owning far more than a weights file: the serving environment, capacity, patching, secrets, update validation and incident handling. A hosted service has its own costs and constraints, but moving infrastructure in-house does not remove responsibility. It changes which team carries it and where the expense appears. Boards and CFOs deserve to see that tradeoff stated plainly rather than discovered in next year's budget.
Draw the full data path before calling anything contained. A local model may still call external search, embedding, telemetry or document-processing services, and diagnostics and backups travel too. During a petabyte-scale cloud migration, the hard work was never the storage; it was proving where every copy went. Enforce the intended boundary with network and application controls, not a deployment label.
Keep the decision reversible and sized to the institution
Choose a first deployment you can compare with an alternative using the same task contract. Keep evaluation cases, output schemas and business rules outside provider-specific prompt code where practical. Compare a managed deployment with self-hosting using the staff and controls actually available. Owning the weights gives the organization options, but someone still has to manage dependencies, patches, capacity and recovery.
The selection record should state what was chosen, why it met the task, what remains unverified and who reviews the next update. Include an exit condition, such as rising escalation rates or unsupported dependencies. Have the operational owner review it alongside the license and deployment plan. Leadership can then approve a bounded evaluation with clear accountability, which is the honest version of “open.”