Change the Metric, Change the Winner

Change the metric and the winner changes. We audit an evaluation protocol rather than a model: the recipe the efficient-adaptation literature uses to recommend a fine-tuning strategy for a clinical foundation model, which fixes a benchmark, adapts a backbone under several strategies, reports accuracy over a few seeds, and takes the best. Run faithfully on a metastasis-detection grid in histopathology, five adaptation strategies by three annotation budgets by three seeds, that recipe admits twelve defensible specifications, and every one of the five strategies is selected as best under at least one of them.

Ranking by accuracy and ranking by calibration name different winners in all three regimes, and where labels are scarce the accuracy ranking does not reproduce even itself: redraw the seeds and its winner survives 53% of the time. A proper scoring rule (Brier) ranks with accuracy at every budget, so adding one would have concealed the disagreement where labels are plentiful. Turning the same checks on ourselves, two of the six conclusions our own protocol licensed do not survive. We propose a claims ledger, recording for each conclusion what its evidence resolves and which population it covers, as what evaluations of medical foundation models should report in place of a leaderboard row.

1The protocol under audit

Efficient-adaptation papers (LoRA, QLoRA, IA³, adapters) have converged on one recipe: fix a benchmark, adapt one backbone under several strategies, report accuracy, F1 or AUC over at most three seeds, pick the argmax, and declare parity with full fine-tuning when a paired t-test fails to reject. That recipe now underwrites deployment guidance in medicine, where the model decides whether a tissue patch contains metastasis.

Fitness for clinical decision support has two parts: how often the model is right, and whether its confidence can be acted on. The accuracy protocol measures only the first. We compare it with a protocol identical in every component except the metric (calibration error instead of accuracy), so any disagreement between their verdicts is attributable to the metric and nothing else. The gap is not one we invented: TRIPOD+AI, the reporting standard for clinical prediction models, already requires calibration, and PROBAST+AI treats its absence as a risk-of-bias signal.

2The audit instrument

A controlled grid, fixed in every respect except the factor under study. A DINO-pretrained pathology ViT-S/8 (21.7M parameters) with a fresh linear head, on PatchCamelyon: binary metastasis detection in lymph-node histopathology, 262,144 training and 32,768 test patches with slide-disjoint official splits. Five strategies, from a 770-parameter linear probe through IA³ (0.13%), LoRA and QLoRA (2.0%) to full fine-tuning (100%). Three annotation budgets (1%, 10%, 100%), stratified and seed-locked so every strategy sees identical data, and three seeds, with an identical training schedule throughout.

3The metric, not the model, decides the winner

Accuracy rank and calibration rank are uncorrelated to anti-correlated (Spearman ρ = +0.10, −0.21, −0.60 across budgets), and the two protocols pick a different top strategy at all three budgets. In the scarce-label regime the medical-adaptation literature most often addresses, the accuracy protocol selects QLoRA (Macro-F1 0.921), which sits at ECE 0.049, roughly 0.041 above the estimator's noise floor. A practitioner following the standard protocol to the letter would deploy a model close to worst on the axis their application depends on, with no signal that this had happened.

The obvious objection is that binned ECE is a poor estimator. Brier settles it: a proper scoring rule computed without bins, its ranking tracks accuracy (ρ = +0.70, +0.90, +1.00), so the disagreement is specific to calibration, not an artifact of binning. The uncomfortable corollary is that adding a proper scoring rule, the usual remedy, would have hidden the disagreement at two budgets of three.

The accuracy winner is not even stable. Redrawing each cell from its reported mean and seed spread over 200,000 replicates, the 1%-label accuracy winner wins again only 52.9% of the time (70.7% at 10%), while the calibration winner holds 86.3% and 99.5%.

4The verdict does not transfer across budgets

Full fine-tuning ranks first on accuracy at 100% labels and last on both accuracy and calibration at 1% (ECE 0.067, an order of magnitude above the noise floor). Annotation budget is a protocol parameter that inverts the conclusion, and it is almost never reported as one. A paper that evaluates at full data and recommends full fine-tuning has not evaluated the regime where the recommendation will be applied.

5A failure mode the protocol cannot see

QLoRA and LoRA share identical adapters and differ only in backbone precision. On accuracy the pair is unresolvable: the Macro-F1 gap is 0.002 at every budget, below the minimum detectable effect. On calibration it is not: QLoRA's ECE is higher at every budget and its fitted temperature is higher throughout (2.39 vs 2.19). 4-bit quantization shifts confidence while an accuracy-only protocol certifies "no change", and that blind spot lines up exactly with the intervention practitioners reach for to save memory.

6Applying the audit to ourselves

A paper arguing that protocols overclaim owes itself the same scrutiny. Two checks, run in minutes on numbers already reported, removed two of our own conclusions.

1. Statistical resolution. With three seeds, the paired t-test this literature relies on has a minimum detectable effect of 0.029 Macro-F1 at 1% labels, larger than nearly every gap it is used to dismiss. Our "LoRA matches full fine-tuning" (p = 0.16) is withdrawn as an equivalence claim: the protocol simply cannot resolve it. Four seeds would have been enough at full data.
2. Estimator resolution. Simulating the 15-bin ECE estimator on models that are perfectly calibrated by construction, it still reports 0.002 to 0.005 on average. The IA³ vs linear-probe gap (0.002) sits inside that noise, so "IA³ is the best-calibrated strategy" is withdrawn. Full fine-tuning's 0.067 is far outside it and survives.

7A reporting checklist

Five requirements that cost nothing but discipline, and need no GPU:

1. Report a calibration-specific metric alongside accuracy, or state that the protocol makes no claim about confidence.
2. State the estimator's configuration and resolution before interpreting a gap, with a null simulation to bound what is interpretable.
3. Report the minimum detectable effect and the stability of the argmax, not only the p-value.
4. Name the regime in the claim: annotation budget inverts the verdict here.
5. Do not treat a proper scoring rule as sufficient: Brier tracked accuracy at every budget.

8Limitations

One backbone family, one binary patch-level benchmark, one task: this bounds the paper to existence claims about the protocol, not how often it fails. No domain shift, external cohorts, slide-level evaluation or rare-event detection, and budgets are carved at the patch level, which is optimistic relative to annotating whole slides. The calibration analysis rests on ECE and Brier, not selective prediction.

More research

My other work at Embedded LLM builds portable, fused GPU kernels for biological and clinical foundation models, measured on both NVIDIA and AMD.

• MOML 2026From 13 Hours to 4.6: Profiling-Driven Kernel Fusion for TensorNet in Molecular Simulation
• COLM 2026Deploying Clinical Language and Vision-Language Models Where the Data Lives
• ICML 2026From 805 ms to 23 ms: Accelerating State-Space Models for Real-Time ICU Monitoring
• ICPP 2026Error-Bounded Fused Attention Compression for Long-Context Genomic Foundation Models Across Heterogeneous GPUs
• ISCA 2026When the LLM-Tuned Stack Misses: An Infrastructure View of Biological Foundation Model Inference Across NVIDIA and AMD
• MLSys 2026BioTriton: Portable Cross-Vendor GPU Kernels for High-Throughput Bioinformatics via OpenAI Triton

All research →