Work done at Embedded LLM Embedded LLM
← Research Details
COLM 2026 DAIH workshop San Francisco, CA, USA · Deployment case report · poster

Deploying Clinical Language and Vision-Language Models Where the Data Lives: An Energy-Aware, Cross-Vendor Protocol for On-Premises Healthcare AI

Author & affiliation Manpreet Singh · Embedded LLM, Singapore

1.64×
faster encoding
on-premises
0.50×
energy per
whole-slide image
2.03×
slides cleared
per shift
≤ 0.001
ΔAUC vs. baseline
accuracy preserved

Clinical foundation models are increasingly capable, but the usual way to run them, shipping the data to a cloud endpoint, is exactly what a hospital cannot do. Whole-slide images, radiology volumes, and the notes attached to them are protected health information (PHI); uploading them to an external service moves regulated data outside the institution's control. The practical answer is to run the model where the data already lives, on the hardware the hospital already owns, without a per-vendor rewrite for whatever GPU happens to be in the rack.

This is a deployment case report, not a new-architecture paper. It documents an energy-aware, cross-vendor protocol for encoding clinical language and vision-language foundation models on-premises: a small set of vendor-portable fused operators (flash attention, fused LayerNorm + linear, fused patch embedding) that run unmodified on both NVIDIA H100 and AMD MI300X. The result is 1.64× faster encoding at half the energy per slide, doubling the slides a pathologist's queue clears per shift, with downstream classification accuracy unchanged (ΔAUC ≤ 0.001).

Deployment architecture: inside the hospital, a slide scanner produces a whole-slide image of 10^4 to 10^5 tiles, sent to the pathology PACS server. Our on-prem foundation-model encoding block, built from vendor-portable fused operators (flash attention, LayerNorm+linear, patch embedding), runs on NVIDIA H100 and AMD MI300X, delivering 1.64× faster, 2.03× slides/shift, 0.50× energy, ΔAUC ≤ 0.001, feeds a MIL classifier and a draft clinical report. The traditional path instead uploads PHI to an external cloud encoder, so PHI leaves the hospital.
ARCHITECTURE PHI never leaves the building. The whole-slide image flows from scanner to PACS to an on-prem encoder built from vendor-portable fused operators that run identically on NVIDIA H100 and AMD MI300X, then into a MIL classifier and a draft report. The traditional path (right, dashed) ships PHI to an external cloud encoder, the upload this protocol removes.

Why on-premises, and why cross-vendor

Two constraints define the deployment. First, compliance: a whole-slide image is 104 to 105 tiles of identifiable patient tissue, and the accompanying notes are PHI. The traditional cloud-encoding path requires uploading that data off-site; the on-prem path keeps it behind the hospital firewall and never transmits it. Second, hardware heterogeneity: hospitals do not standardize on one accelerator. Some racks are NVIDIA, some are AMD (increasingly attractive for the 192 GB of unified HBM3 on MI300X), and a protocol that only runs on one vendor is not deployable in practice. The fused operators here are written once and run on both, with no per-vendor forks or ifdef'd kernels.

The energy argument

Latency is the number people quote, but in a hospital the binding constraint is often energy and throughput per shift. Encoding a slide at half the energy is not just a sustainability line; it directly changes how many slides the on-prem cluster can clear before it saturates its power and cooling budget. Here the fused encoder halves energy per whole-slide image (0.50×) while running 1.64× faster, and the two combine to 2.03× the slides cleared per shift. The accuracy tax for all of it is within measurement noise: downstream MIL classification AUC moves by ≤ 0.001.

0 uploads

PHI stays on-premises

The encoder runs inside the hospital on hardware it already owns. No whole-slide image, radiology volume, or clinical note is transmitted to an external service, so the compliance boundary is never crossed.

2 vendors

Write-once, run on both

The same fused-operator source runs unmodified on NVIDIA H100 and AMD MI300X. No per-vendor rewrite, so the protocol is deployable on whatever accelerator the rack already has.

0.50×

Half the energy per slide

Fusing attention, LayerNorm + linear, and patch embedding cuts energy per whole-slide image in half, extending how many slides fit inside a fixed power and cooling budget.

≤ 0.001

Accuracy held constant

Downstream classification AUC changes by at most 0.001 versus the unfused baseline. The speed and energy wins are free of a diagnostic-quality cost.

Where the speedup comes from

The gains are additive across three fused operators, each closing a distinct source of unfused overhead. Fusing attention removes the largest slice; fusing LayerNorm with the following linear layer removes a second memory round-trip; fusing patch embedding trims what remains. Together they compound to the full 1.64× over the unfused baseline, a +0.64× (38.9%) total gain, as the ablation below shows.

Cumulative speedup ablation bar chart over the unfused baseline. Unfused baseline 1.00×; +Fused Attention 1.33× (incremental +0.33×, 20.1%); +Fused LayerNorm+Linear 1.58× (incremental +0.25×, 15.2%); +Fused Patch Embedding (full) 1.64× (incremental +0.06×, 3.7%). Total gain versus baseline +0.64×, 38.9%.
ABLATION Cumulative speedup over the unfused baseline. Fused attention contributes the largest step (+0.33×, 20.1%), fused LayerNorm + linear adds +0.25× (15.2%), and fused patch embedding closes the last +0.06× (3.7%), for a full-stack 1.64×, a +0.64× (38.9%) gain, with no per-vendor code paths.

The thesis in one line: clinical foundation models can be deployed where the regulated data already lives, on the accelerators hospitals already own, faster and at half the energy, with no PHI leaving the building, no per-vendor rewrite, and no measurable loss in diagnostic accuracy.

Honest limitations

This is a deployment case report on the encoding stage: it optimizes the foundation-model encoder and measures the downstream MIL classifier's AUC, but it does not retrain or co-design the classifier. Energy figures are measured on the two accelerators studied (H100, MI300X) and will not transfer unchanged to other GPUs. The evaluation centers on the whole-slide pathology pipeline; radiology-volume and text-only clinical workloads share the operator set but have different tile counts and memory profiles, and their per-shift economics remain to be measured directly.

Part of an ongoing line of work on portable, energy-aware GPU kernel acceleration for biological and medical AI, benchmarked across NVIDIA H100 and AMD MI300X. See more on the research page.