Deploying Clinical Language and Vision-Language Models Where the Data Lives

“Benchmark performance is a poor proxy for real-world readiness.”DAIH 2026 call for papers

So we stopped benchmarking the model and benchmarked the deployment.

Three years of inference work went to LLM serving: vLLM, SGLang, paged attention, continuous batching. Almost none of it reaches a pathology encoder, a protein or genomic model, or a bedside time-series monitor, because none of them generate tokens. They ship as research code, built to prove accuracy, and stay that way. The main lever a hospital has left is buying more hardware. The fix belongs upstream, with the lab that ships the model.

01What the leaderboard leaves out

Accuracy is necessary, not sufficient. From paper to hospital or R&D lab, speed and cost per case decide whether a model gets used. DAIH asks whether clinical AI works, is equitable, and is safe. We turned those questions into four deployment metrics and measured them next to accuracy.

02Why chatbot speedups never reach these models

Chat-serving engines optimize the token-generation loop and the KV cache. These models never generate a token: a pathology encoder outputs embeddings, a binding model a score, an ICU model a risk. Their bottlenecks are domain-specific, and nothing built for chatbots touches them.

03One file, two vendors, one gate

These models are memory-bound. Unfused, every step writes its result to GPU memory (HBM) and the next step reads it back. We fuse the slow steps into single kernels that keep intermediates in on-chip SRAM, written once in OpenAI Triton and compiled unmodified for NVIDIA and AMD, with weights and maths held fixed.

The fidelity gate. A speedup is reported only if outputs match the reference: exact, or cosine ≥ 0.99999, and for pathology an AUC change of at most 0.001. Kernels that fail are excluded. Speed is never bought with silent quality loss.

This belongs upstream: the lab that releases a model should ship it fast, the same way it ships it accurate. Hospitals can't write GPU kernels, and shouldn't have to. If the lab ships the fast version, every hospital gets it for free.

04A shift's worth of slides on one GPU

On the UNI pathology encoder, fusing attention, LayerNorm + Linear and patch embedding adds up to 1.33×, 1.58× and 1.64× in order of where the time is spent: 129.6 s → 79.1 s per slide. Bigger backbones gain more: ViT-S 1.42×, UNI 1.64×, Virchow 1.71×.

05Inside the 50 ms bedside window

The old pipeline took one whole heartbeat. More than 85% of that delay was cleaning up irregularly sampled vital signs (over 30% missing), not running the model. We fused the whole time-aware interpolation into one kernel; its outputs match the unfused path within 5 × 10⁻⁷, so the model's predictions are unchanged.

06Same source, six models, three GPUs

Models from Meta, DeepMind, NVIDIA and the Baker lab. All accurate. All shipped as research code. The same Triton source, with no per-vendor forks, and every result passes the fidelity gate.

07Ask what it does on the card you will actually run it on

A negative result. One model, identical source, not one line changed between vendors.

Deployment evidence obtained at one site does not automatically transfer to another. The cause: DualBind's reference code copied results to the CPU for a NumPy score, every sample, then returned to the GPU. NVIDIA's runtime and prefetcher hid most of that cost; AMD exposed it (800 samples: 41.3 s reference vs 1.06 s fused). Corrected, the two vendors land within 1.4× of each other.

So the 38.8× is not evidence that AMD is faster: the reference code hid a cost that only one vendor exposed. A milder case shows the same thing: Enformer, same source, gets 1.82× on H100 and 1.43× on MI300X. This is one model in one implementation, and a simple GPU-resident rewrite would likely fix most of it. The claim is narrow: single-vendor testing can hide this.

Methods and limits

1. Energy: UNI encoder on a single on-site GPU; board power sampled every 100 ms across the full encode, idle not subtracted, host CPU and cooling excluded.
2. Baselines: speedups are against the reference PyTorch implementation, so they measure the headroom in code as shipped, which is what labs release and hospitals download.
3. Data: public, retrospective cohorts (PhysioNet 2012 for ICU, batch 32, model held fixed). Not a clinical validation study; site validation is still needed.
4. Kept separate: the paper also reports that swapping GRU-D for a state-space model adds +0.037 AUROC. That is a model change, not the kernel, and it is not counted here.

BibTeX

@inproceedings{singh2026deploying,
  title     = {Deploying Clinical Language and Vision-Language Models
               Where the Data Lives: An Energy-Aware, Cross-Vendor
               Protocol for On-Premises Healthcare {AI}},
  author    = {Manpreet Singh},
  booktitle = {LLM/VLM Deployment Opportunities and Risks in Healthcare},
  year      = {2026},
  url       = {https://openreview.net/forum?id=l5owp8UMxu}
}

More research

This is one piece of a broader line of work at Embedded LLM on portable, fused GPU kernels for biological and clinical foundation models, measured on both NVIDIA and AMD.

• MOML 2026From 13 Hours to 4.6: Profiling-Driven Kernel Fusion for TensorNet in Molecular Simulation
• ICPP 2026Error-Bounded Fused Attention Compression for Long-Context Genomic Foundation Models Across Heterogeneous GPUs
• ICML 2026From 805ms to 23ms: Accelerating State-Space Models for Real-Time ICU Monitoring
• ISCA 2026When the LLM-Tuned Stack Misses: An Infrastructure View of Biological Foundation Model Inference Across NVIDIA and AMD
• ISC 2026Portable GPU Kernel Acceleration for Biological Foundation Models & Algorithms using OpenAI Triton
• RECOMB 2026Hardware-Portable Fused GPU Kernels for High-Throughput Biological Foundation Models
• MLSys 2026BioTriton: Portable Cross-Vendor GPU Kernels for High-Throughput Bioinformatics via OpenAI Triton

All research →