When the LLM-Tuned Stack Misses

AI inference infrastructure is tuned for one workload shape: dense, large-batch, NVIDIA-resident transformer decoding. Biological foundation models do not fit it. They use short alphabets (4 to 20 tokens, not 100K+), attend over 100K+ token sequences at batch one, mix transformer blocks with geometric operations, and increasingly deploy on whatever GPU is available, including AMD MI300X for its 192 GB of HBM3.

This is a measurement study framed as an infrastructure paper. Six widely used biological models (ESM-2, ProtBERT, ProteinMPNN, AlphaGenome, Enformer, DualBind) are benchmarked on NVIDIA L4, H100 and AMD MI300X. 60–80% of reference runtime is spent outside dense compute, and a single Triton source tree closes most of the gap on both vendors with no per-vendor forks.

1Where the time actually goes

Across all three GPUs the reference PyTorch implementations spend most of their time in memory-bound element-wise operations, fragmented kernel launches and small-batch overhead. Vendor LLM kernels do not target these paths, because LLM serving amortizes them over large batches and long decode loops. Three properties explain it: short alphabets make general-vocabulary GEMMs wasteful, very long sequences at batch one make attention memory-bound, and non-transformer geometry (RBF encodings, virtual-Cβ, pairwise distances) becomes long chains of small launches with no vendor-library equivalent.

2Three observations

1. Different bottlenecks than LLMs. Bio workloads stress memory-bound element-wise work, fragmented launches and small-batch overhead, not the dense matmul path vendor libraries are tuned for.
2. Vendor asymmetry amplifies bugs. The biggest win is not kernel engineering: removing one per-sample host–device round-trip in DualBind gives 38.8× on MI300X but 4.22× on H100.
3. Write-once portability works today. One Triton source runs unmodified on L4, H100 and MI300X across all six models, at 1.51× to 43.1× with cosine similarity ≥ 0.99999.
4. Parity is closer than it looks. Once the code is GPU-resident, NVIDIA and AMD land within about 1.4× of each other in absolute throughput.

3Per-model results

Speedups over the reference PyTorch implementation, gated by a strict fidelity requirement: cosine ≥ 0.99999, exact top-1 predictions, or zero difference to six decimal places.

4The DualBind lesson

The kernel files are identical between vendors. Profiling showed the upstream DualBind reference was not GPU-resident end to end: it made a per-sample host–device round-trip between the transformer pass and the binding-energy computation. On H100, the PCIe controller, unified-memory prefetcher and CUDA runtime hide most of that cost. On MI300X it is exposed, and 800 samples take 41.3 s instead of 1.06 s.

The lesson is not that AMD is slower. The same upstream code can differ by an order of magnitude across vendors because the host–device path differs in how forgivingly it hides bad code. Biological reference code is overwhelmingly written on NVIDIA; deployed on AMD, latent systems bugs become first-order.

5Portability, measured cleanly

Enformer is the cleanest evidence: the same Triton files run on H100 and MI300X at the full 196K-base length, 25 trials each, with no per-vendor branches or tiles. The asymmetry (1.82× vs 1.43×) is a property of the baseline, not the kernel: H100 had less unfused overhead to remove. This is what write-once cross-vendor performance looks like on a non-LLM workload today: not identical speedups, but the same kernel winning on both.

Today's AI infrastructure has a workload shape it was not built for. The gap on biological models is large, vendor-stack asymmetry turns trivial bugs into order-of-magnitude regressions, and closing it with off-the-shelf compiler infrastructure costs far less than the field assumes.

6Limitations

1. The six models lean toward transformer-derived architectures; iterative structure predictors (AlphaFold, ESMFold) and diffusion-based designers are absent and likely have different bottlenecks.
2. The study covers single-GPU inference only; multi-GPU sharding and tensor parallelism, which dominate production pipelines at scale, are out of scope, and training-time portability remains open.

More research

Part of a broader line of work at Embedded LLM on portable, fused GPU kernels for biological and clinical foundation models, measured on both NVIDIA and AMD.

• NeurIPS 2026Change the Metric, Change the Winner: Auditing How We Decide Which Clinical Foundation Model to Deploy in Cancer Pathology
• MOML 2026From 13 Hours to 4.6: Profiling-Driven Kernel Fusion for TensorNet in Molecular Simulation
• COLM 2026Deploying Clinical Language and Vision-Language Models Where the Data Lives
• ICML 2026From 805 ms to 23 ms: Accelerating State-Space Models for Real-Time ICU Monitoring
• ICPP 2026Error-Bounded Fused Attention Compression for Long-Context Genomic Foundation Models Across Heterogeneous GPUs
• MLSys 2026BioTriton: Portable Cross-Vendor GPU Kernels for High-Throughput Bioinformatics via OpenAI Triton
• RECOMB 2026Hardware-Portable Fused GPU Kernels for High-Throughput Biological Foundation Models
• ISC HPC 2026Portable GPU Kernel Acceleration for Biological Foundation Models & Algorithms using OpenAI Triton

All research →