Merlin: a computed tomography vision-language foundation model and dataset.

📝 Paper Summary

Medical Vision-Language Models 3D Medical Imaging Foundation Models

Merlin is a 3D foundation model for abdominal CT scans trained on paired images, text reports, and EHR diagnosis codes to enable versatile downstream clinical tasks without manual labels.

Core Problem

Existing medical vision-language models are primarily limited to 2D images and short text, failing to capture the 3D volumetric nature of CT scans and the rich context of long radiology reports.

Why it matters:

Abdominal CTs are high-volume (25% of all CTs) and complex (300+ slices), contributing to radiologist burnout
Current 2D approaches (slice-by-slice aggregation) are inefficient and miss 3D anatomical correlations essential for identifying pathologies
Supervised ML requires expensive manual labeling, whereas hospitals have vast amounts of unlabelled paired data (images, reports, EHR codes) unused by current methods

Concrete Example: A single abdominal CT scan contains 300+ slices and corresponding reports exceed 300 words. A standard 2D CLIP model processes slices independently, missing the 3D shape of a tumor or organ, while Merlin processes the entire volume at once to capture spatial continuity.

Key Novelty

Merlin: 3D-native Vision-Language Pretraining with Multi-Granularity Supervision

Processes full 3D CT volumes natively rather than aggregating 2D slices, preserving volumetric spatial relationships
Uses 'radiology report splitting' to align specific anatomical image regions with their corresponding text sections (e.g., matching liver voxels to 'liver: normal' text)
Integrates structured EHR diagnosis codes as an additional supervision signal alongside unstructured text reports during pretraining

Architecture

Overview of Merlin's training pipeline and downstream tasks.

Evaluation Highlights

+16.0% improvement in F1 score (0.741 vs 0.641) for zero-shot findings classification compared to supervised training on the same data
Outperforms state-of-the-art 2D-to-3D lifted VLMs (like BioMedCLIP) by 32.1% F1 on findings classification and 39.8% AUROC on phenotype classification
Achieves equivalent segmentation performance to nnUNet using only 10% of training data, demonstrating high label efficiency

Breakthrough Assessment

9/10

Significant leap in medical AI by successfully scaling VLM pretraining to true 3D volumes with diverse supervision (text + EHR). Strong empirical results across many tasks and release of a large-scale dataset/model make it a major resource.

⚙️ Technical Details

Problem Definition

Setting: Multi-modal pretraining on paired 3D CT volumes, radiology reports, and EHR codes

Inputs: 3D CT volume V, Radiology Report T, EHR Diagnosis Codes C

Outputs: Joint embeddings for images and text; downstream predictions for classification, segmentation, or retrieval

Pipeline Flow

Input Processing: 3D CT Volume + Text (Reports/Codes)
Feature Extraction: 3D Image Encoder + Text Encoder
Contrastive Alignment: Projecting both into shared space
Task-Specific Heads: Classification / Segmentation / Generation

System Modules

3D Image Encoder (Feature Extraction)

Extract visual features from 3D CT volumes

Model or implementation: ResNet-152 (Inflated from 2D to 3D via I3D initialization)

Text Encoder (Feature Extraction)

Extract semantic features from radiology reports and diagnosis descriptions

Model or implementation: Clinical Longformer

Contrastive Loss

Align image and text representations

Model or implementation: InfoNCE / CLIP loss

Novel Architectural Elements

Multi-task supervision combining unstructured radiology report text AND structured EHR diagnosis codes
Radiology report splitting strategy: dividing reports into anatomical sections (e.g., 'liver', 'pancreas') to create fine-grained image-text pairs during pretraining

Modeling

Base Model: ResNet-152 (3D inflated) for vision, Clinical Longformer for text

Training Method: Contrastive Learning (CLIP-style) followed by task-specific adaptation

Objective Functions:

Purpose: Align 3D images with corresponding text descriptions.

Formally: Contrastive Loss (InfoNCE) maximizing cosine similarity of correct pairs in a batch.
Purpose: Align 3D images with EHR diagnosis codes.

Formally: Contrastive Loss between image embeddings and text descriptions of ICD codes.

Adaptation: Linear probing (classification), UNet decoder (segmentation), or Transformer decoder (report generation)

Trainable Parameters: Not reported in the paper

Training Data:

15,331 CTs (6M+ images)
1.8M+ EHR diagnosis codes
6M+ radiology report tokens
Internal validation: 5,137 CTs
External validation: 44,098 CTs

Key Hyperparameters:

learning_rate: Not explicitly reported in the paper
batch_size: Not explicitly reported in the paper
epochs: Not explicitly reported in the paper

Compute: Single NVIDIA A6000 GPU, approx 160 hours training time

Comparison to Prior Work

vs. BioMedCLIP: Merlin is native 3D and uses EHR codes + full reports, whereas BioMedCLIP is 2D and uses shorter captions
vs. RadFM: Merlin uses contrastive pretraining for better embeddings, while RadFM focuses on generation
vs. CT-CLIP: Merlin focuses on Abdominal CT (more complex anatomy) vs Chest CT
+ 1 more
vs. Med-PaLM M: Merlin is open-weights and specifically optimized for 3D volumetric understanding

Limitations

Under-reporting of positive findings in generated radiology reports
Slightly lower segmentation performance than nnUNet when 100% labeled data is available
Computational cost of 3D processing is higher than 2D slice-based methods
Requires paired text/EHR data which may not be available in all datasets

Reproducibility

Code: https://github.com/StanfordMIMI/Merlin

Models, code, and a de-identified dataset (Merlin Abdominal CT dataset) are publicly released at https://github.com/StanfordMIMI/Merlin. The dataset contains 25,494 CT images and reports.

📊 Experiments & Results

Evaluation Setup

Comprehensive evaluation across 6 task types and 752 individual tasks

Benchmarks:

Zero-shot findings classification (Classification) [New]
Phenotype classification (PheWAS) (Classification) [New]
Zero-shot cross-modal retrieval (Retrieval) [New]
5-year disease prediction (Prognosis / Risk Prediction) [New]
3D Semantic Segmentation (Segmentation)
Radiology Report Generation (Text Generation)

Metrics:

F1 Score
AUROC (Area Under Receiver Operating Characteristic)
AUPRC (Area Under Precision-Recall Curve)
Dice Score
RadGraph-F1
BERT Score
Statistical methodology: 95% Confidence Intervals reported; p-values reported for comparisons

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Merlin demonstrates strong zero-shot capabilities on findings classification, significantly outperforming 2D baselines.
Internal Validation Set (31 findings)	F1 Score	0.424	0.741	+0.317
External Validation Set	F1 Score	0.334	0.647	+0.313
In data-scarce segmentation tasks, Merlin's pretraining offers significant advantages over standard supervised methods.
20 Organs Segmentation (10% training data)	Dice Score	0.793	0.830	+0.037
Merlin effectively predicts future disease risk from baseline CT scans.
6 Chronic Diseases	AUROC	0.707	0.757	+0.050
Ablation studies reveal the importance of report splitting for classification performance.
Zero-shot Classification	F1 Score	0.662	0.741	+0.079

Experiment Figures

Zero-shot findings classification performance across internal and external datasets.

Comparison of Merlin against various baseline architectures (2D VLMs, 2D-to-3D lifted, 3D Vision-only) across data regimes.

Main Takeaways

Native 3D processing combined with Vision-Language pretraining significantly outperforms 2D and 2D-to-3D lifted approaches.
Utilizing both unstructured radiology reports and structured EHR codes via multi-task learning yields the most robust representations.
Radiology report splitting (anatomical sectioning) is critical for fine-grained discriminative tasks but less important for global retrieval.
Merlin shows strong generalization to external datasets and even chest CTs (despite being trained on abdominal CTs), outperforming domain-specific chest models.

📚 Prerequisite Knowledge

Prerequisites

Contrastive Learning (CLIP)
3D Convolutional Neural Networks (ResNet)
Medical Imaging formats (DICOM, NIfTI)
Electronic Health Records (EHR) structure

Key Terms

VLM: Vision-Language Model—a model trained to understand images and text jointly

EHR: Electronic Health Records—digital version of a patient's paper chart, including diagnosis codes

I3D: Inflated 3D ConvNet—a technique to expand 2D convolutional weights into 3D to initialize 3D networks

Zero-shot: Evaluating a model on a task it wasn't explicitly trained for, using only its pre-trained knowledge

PheWAS: Phenome-Wide Association Study—a method to scan for associations between genetic variants and a wide range of phenotypic traits (diseases)

Linear probe: Training a simple linear classifier on top of a frozen pre-trained model to evaluate feature quality

nnUNet: No-New-UNet—a self-configuring framework for medical image segmentation that adapts to dataset properties

AUROC: Area Under the Receiver Operating Characteristic curve—a metric for binary classification performance

Dice score: A metric measuring the overlap between predicted and ground-truth segmentation masks (0 to 1)