Adaptive Clinical-Aware Latent Diffusion for Multimodal Brain Image Generation and Missing Modality Imputation

📝 Paper Summary

Medical Image Synthesis Multimodal Learning Missing Data Imputation

ACADiff synthesizes missing brain imaging modalities by conditioning a latent diffusion model on available scans via adaptive fusion and clinical metadata encoded by GPT-4o.

Core Problem

Multimodal Alzheimer's disease datasets frequently suffer from missing imaging modalities (MRI, PET) due to cost or patient dropout, which limits diagnostic accuracy and research utility.

Why it matters:

Single modalities (like just MRI) miss complementary pathological markers available in PET scans (glucose metabolism, amyloid deposition)
Existing generative methods (GANs, standard diffusion) lack adaptive mechanisms to handle varying combinations of missing inputs (e.g., handling both 1→1 and 2→1 generation with one model)
Current approaches rarely integrate rich semantic clinical metadata (diagnosis, cognitive scores) to guide the generation process toward biologically plausible results

Concrete Example: A patient may have an MRI and clinical scores but lack FDG-PET due to cost. A standard diffusion model might generate a generic PET scan that looks realistic but ignores the patient's specific cognitive decline (e.g., MMSE=22), whereas ACADiff uses the clinical scores to generate a PET scan reflecting the appropriate hypometabolism patterns.

Key Novelty

Adaptive Clinical-Aware Diffusion (ACADiff)

Adaptive Image Conditioning: A dynamic fusion module switches between cross-attention (for multiple inputs) and projection (for single input) based on availability masks, allowing one model to handle any combination of missing modalities.
Semantic Clinical Guidance: Instead of simple class labels, patient metadata (MMSE, ADAS13) is converted into natural language prompts and encoded by GPT-4o to guide the diffusion process toward disease-consistent patterns.
Hierarchical Conditioning: Combines early fusion of image features with late fusion of text embeddings in the diffusion U-Net to preserve both structural details and high-level semantic disease information.

Architecture

The ACADiff framework illustrating the latent diffusion process with hierarchical adaptive conditioning.

Evaluation Highlights

Achieves 89.4% diagnostic accuracy with 20% missing data (97.2% of the performance of complete real data).
Maintains 77.5% diagnostic accuracy even under extreme 80% missing data scenarios, outperforming the best baseline (LDM) which achieves 76.4%.
Outperforms state-of-the-art baselines (LDM, PASTA, Pix2Pix) across all generation metrics, achieving PSNR of 27.9 and SSIM of 0.911.

Breakthrough Assessment

8/10

Strong practical contribution solving the specific problem of variable missing modalities in medical imaging. The integration of LLM-encoded clinical text for guidance is a novel and effective addition to standard latent diffusion.

⚙️ Technical Details

Problem Definition

Setting: Missing modality imputation for 3D brain volumes (sMRI, FDG-PET, AV45-PET)

Inputs: Set of available 3D modalities Z_available, binary availability mask z_avail, and clinical text description z_text

Outputs: Synthesized target modality X_target

Pipeline Flow

Preprocessing (VAE Encoding)
Conditioning Preparation (Adaptive Fusion + GPT-4o Encoding)
Latent Diffusion Process (Denoising U-Net)
Reconstruction (VAE Decoding)

System Modules

3D VAE Encoder

Compresses high-resolution 3D brain volumes into compact latent representations

Model or implementation: Pretrained 3D Autoencoder-KL

GPT-4o Text Encoder (Conditioning Preparation)

Encodes structured text prompts containing clinical metadata into semantic embeddings

Model or implementation: GPT-4o (text encoder)

Adaptive Fusion Module (Conditioning Preparation)

Fuses available image modalities based on dynamic availability

Model or implementation: Learnable Cross-Attention + 3D Convolution Projection

Denoising U-Net

Predicts noise to remove from the target latent at timestep t, conditioned on fused images and text

Model or implementation: 3D U-Net with Cross-Attention and FiLM

Novel Architectural Elements

Hierarchical Adaptive Conditioning: Combines availability-aware image fusion (concatenated input) with semantic clinical guidance (cross-attention layers) in a single diffusion backbone
Dynamic fusion switch: Mechanism that structurally changes between cross-attention and projection based on input count (1 vs 2 available modalities)

Modeling

Base Model: Latent Diffusion Model with 3D U-Net backbone

Training Method: Supervised training of diffusion model on paired multimodal data

Objective Functions:

Purpose: Minimize difference between predicted and actual noise.

Formally: L_diff = E[||epsilon - epsilon_theta(...)||^2]
Purpose: Ensure generated latent aligns with real distribution (Consistency Regularization).

Formally: L_cons = ||Z_hat_0 - Z_0||
Purpose: Total loss combines denoising and consistency.

Formally: L = L_diff + lambda * L_cons

Training Data:

ADNI cohort: 1,028 subjects (198 AD, 495 MCI, 335 HC)
Split: 600 for generator training (10% val, 10% test)
Split: 428 for classification downstream tasks

Key Hyperparameters:

learning_rate: 1e-4
optimizer: AdamW
timesteps_T: 1000
+ 3 more
image_size_input: 160x180x160
latent_size: 20x22x20
text_embedding_dim: Not explicitly reported in the paper (implied by GPT-4o/ACADiff-emb comparison)

Compute: 4 NVIDIA A100 GPUs

Comparison to Prior Work

vs. LDM: Adds adaptive fusion for variable inputs and GPT-4o semantic guidance vs. standard concatenation
vs. Pix2Pix/DS-GAN: Uses diffusion for stability vs. GAN adversarial training; handles missing modalities adaptively
vs. SynDiff [not cited in paper]: SynDiff uses adversarial diffusion for synthesis but typically lacks the specific LLM-based clinical text encoding for guidance

Limitations

Relies on proprietary GPT-4o API for text encoding, which may limit offline reproducibility or incur costs.
Evaluation limited to ADNI dataset; generalization to other datasets/diseases not tested.
Requires training separate generators for each target modality (Any→MRI, Any→FDG, Any→AV45) rather than a single unified model for all outputs.

Reproducibility

Code: https://github.com/rongzhou7/ACADiff

Code available at https://github.com/rongzhou7/ACADiff. Uses ADNI dataset which requires access approval. GPT-4o usage implies API dependency.

📊 Experiments & Results

Evaluation Setup

Generate missing modalities from available ones, then use completed data for AD diagnosis classification

Benchmarks:

ADNI (Alzheimer's Disease Neuroimaging Initiative) (Image Synthesis & Classification)

Metrics:

PSNR (Peak Signal-to-Noise Ratio)
SSIM (Structural Similarity Index)
MAE (Mean Absolute Error)
Classification Accuracy (ACC)
AUC (Area Under Curve)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Generation quality metrics show ACADiff outperforms all baselines in synthesizing realistic brain volumes.
ADNI Generation	PSNR	26.1	27.9	+1.8
ADNI Generation	SSIM	Not reported in the paper	0.911	Not reported in the paper
Classification performance on the downstream AD vs HC task using synthesized data demonstrates robustness to missing modalities.
ADNI Classification (20% missing)	Accuracy	92.0	89.4	-2.6
ADNI Classification (80% missing)	Accuracy	76.4	77.5	+1.1

Experiment Figures

Visual comparison of generated brain images and quantitative bar charts for generation metrics (PSNR, SSIM, etc.).

Main Takeaways

ACADiff consistently outperforms GAN-based and diffusion-based baselines in generation quality (PSNR, SSIM).
Clinical guidance matters: The ~1.8 PSNR gap between ACADiff and ACADiff-emb (using learnable embeddings instead of GPT-4o) proves the value of semantic LLM encoding.
Robustness: Even with 80% missing data, ACADiff maintains 77.5% diagnostic accuracy, whereas simple imputation fails and other generative methods perform worse.
Hierarchical conditioning effectively handles variable input scenarios (2→1 and 1→1 generation) within a single framework.

📚 Prerequisite Knowledge

Prerequisites

Denoising Diffusion Probabilistic Models (DDPM)
Latent Diffusion Models (LDM)
Variational Autoencoders (VAE)
Multimodal fusion strategies (Cross-attention)

Key Terms

sMRI: Structural Magnetic Resonance Imaging—provides anatomical information about brain structure and atrophy

FDG-PET: Fluorodeoxyglucose Positron Emission Tomography—measures glucose metabolism in the brain

AV45-PET: Florbetapir Positron Emission Tomography—visualizes amyloid plaques associated with Alzheimer's disease

Latent Diffusion: A generative model that performs the diffusion (noise removal) process in a compressed latent space rather than pixel space to save compute

MMSE: Mini-Mental State Examination—a widely used test of cognitive function

ADAS13: Alzheimer's Disease Assessment Scale-Cognitive Subscale (13 items)—measures severity of cognitive and noncognitive dysfunction

CDR-SOB: Clinical Dementia Rating Sum of Boxes—stages severity of dementia

VAE: Variational Autoencoder—neural network that learns to compress data into a lower-dimensional latent space

FiLM: Feature-wise Linear Modulation—a technique to influence network activations by applying an affine transformation based on conditioning information