RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment

📝 Paper Summary

Multimodal Recommendation LLM-Enhanced Recommendation Graph Neural Networks (GNN) for RecSys

RecGOAT bridges the semantic gap between Large Model knowledge and recommendation ID signals using dual-granularity alignment: instance-level contrastive learning and distribution-level optimal adaptive transport.

Core Problem

Existing methods fail to align the rich, general semantic knowledge of Large Models (LLMs/LVMs) with the sparse, specific ID-based signals used in collaborative filtering.

Why it matters:

Semantic heterogeneity prevents recommendation systems from fully leveraging the reasoning and world knowledge capabilities of Large Models
Current alignment techniques are limited to local instance-level matching, missing global distribution-level patterns essential for robust cross-modal understanding
Misaligned representations lead to suboptimal performance, particularly in large-scale scenarios where ID sparsity is high and semantic understanding is crucial

Concrete Example: A generative Large Model understands an item as a 'vintage leather jacket' with specific visual attributes, while a recommendation system sees it as 'ID: 49201' linked to user clicks. Without alignment, the model cannot transfer the semantic understanding of 'vintage' to similar items that 'ID: 49201' users might like but haven't clicked yet.

Key Novelty

Graph Optimal Adaptive Transport (RecGOAT)

Treats modal alignment as an optimal transport problem, moving the distribution of LLM-enhanced semantic features to match the distribution of collaborative ID features
Uses a dual-granularity approach: instance-level contrastive learning for local discriminability and distribution-level transport for global semantic consistency
Introduces adaptive learnable parameters into the transport matrix, allowing the geometric alignment to be fine-tuned by downstream recommendation tasks

Architecture

The overall architecture of RecGOAT, illustrating the data flow from LLM-enhanced feature extraction to dual-granularity alignment and final prediction.

Evaluation Highlights

Outperforms state-of-the-art baselines on three public datasets (Amazon-Baby, Amazon-Sports, Tiktok) across Recall@20 and NDCG@20 metrics
Achieves 1.48% improvement in CTR and 1.63% in GMV in online A/B testing on a large-scale advertising platform
Consistent performance gains over LLM-based competitors like TALLRec and various multimodal GNN baselines

Breakthrough Assessment

7/10

Solid theoretical grounding with Optimal Transport for distribution alignment, backed by both extensive offline benchmarks and real-world industrial deployment. Addresses a critical semantic gap in current LLM-RecSys integration.

⚙️ Technical Details

Problem Definition

Setting: Multimodal Recommendation with implicit feedback and explicit modal content (text, image)

Inputs: User-item interaction graph, Item text descriptions, Item images

Outputs: Predicted preference score for user u and item i

Pipeline Flow

Feature Extraction (LLM/LVM processing)
Intra-Modal Graph Learning (GAT & LightGCN)
Dual Semantic Alignment (CMCL & OAT)
Preference Prediction (Fusion & Scoring)

System Modules

Item Encoder (Feature Extraction)

Extract semantic embeddings from item text and images

Model or implementation: Qwen3-Embedding-8B (Text), LLaVA-1.5-7B (Vision)

User Preference Inferrer (Feature Extraction)

Generate user preference text descriptions based on interaction history

Model or implementation: QwQ-32B

Attentive Graph Learner (Intra-Modal Graph Learning)

Capture high-order structural relationships within each modality

Model or implementation: Graph Attention Network (GAT) on KNN graphs

Collaborative Graph Learner (Intra-Modal Graph Learning)

Learn collaborative filtering signals from interactions

Model or implementation: LightGCN

Alignment Module

Align modal features with ID features

Model or implementation: Cross-Modal Contrastive Learning (CMCL) + Optimal Adaptive Transport (OAT)

Novel Architectural Elements

Dual-granularity alignment block combining instance-level CMCL with distribution-level OAT
Adaptive Transport Matrix that adds a learnable residual term to the geometric OT plan

Modeling

Base Model: Custom architecture integrating GAT, LightGCN, and OT layers, initialized with Qwen/LLaVA features

Training Method: Joint optimization of Recommendation Loss (BPR), Contrastive Loss (InfoNCE), and Alignment Loss (Wasserstein)

Objective Functions:

Purpose: Optimize recommendation ranking.

Formally: BPR Loss L_rec = -sum ln(sigmoid(y_ui - y_uj))
Purpose: Instance-level alignment.

Formally: InfoNCE Loss L_CMCL = -log(exp(sim(z_a, z_p)/tau) / sum(exp(sim(z_a, z_n)/tau)))
Purpose: Distribution-level alignment.

Formally: Wasserstein Distance minimization via Sinkhorn algorithm

Key Hyperparameters:

learning_rate: 1e-4
batch_size: 2048
embedding_dimension: 64
+ 3 more
L2_regularization: 1e-5
OT_scaling_factor_s: Not explicitly reported in the paper
fusion_weights_gamma: Not explicitly reported in the paper

Compute: Not reported in the paper

Comparison to Prior Work

vs. MMGCN/LATTICE: RecGOAT uses LLMs for initialization and OT for alignment, whereas standard GNNs use simpler features and concatenation/attention fusion.
vs. TALLRec/UniMP: RecGOAT focuses on aligning distributions via Optimal Transport, whereas LLM-Rec methods typically rely on fine-tuning or cross-attention which aligns only local instances.
vs. MOTKD [not cited in paper]: MOTKD uses OT for modality-modality alignment; RecGOAT specifically targets the Modality-ID alignment gap and adds adaptive learnable transport.

Limitations

Computational complexity of Optimal Transport (Sinkhorn) can be high for extremely large batches, though Sinkhorn is relatively efficient.
Reliance on powerful but heavy LLMs/LVMs for initial feature extraction creates a high preprocessing cost.
The paper focuses on item-side alignment; user-side alignment relies primarily on generated text prompts rather than symmetric OT alignment.

Reproducibility

Code: https://github.com/6lyc/RecGOAT-LLM4Rec

Code is publicly available at https://github.com/6lyc/RecGOAT-LLM4Rec. The paper specifies the exact LLMs used for feature extraction (Qwen3-Embedding-8B, LLaVA-1.5-7B, QwQ-32B). Hyperparameters for specific datasets (like lambda weights) are generally discussed but specific per-dataset values might need lookup in the repo.

📊 Experiments & Results

Evaluation Setup

Top-K Recommendation on sparse datasets

Benchmarks:

Amazon-Baby (E-commerce Recommendation)
Amazon-Sports (E-commerce Recommendation)
Tiktok (Short-video Recommendation)

Metrics:

Recall@10
Recall@20
NDCG@10
NDCG@20
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
RecGOAT consistently outperforms baselines across all three datasets on Recall@20 and NDCG@20.
Amazon-Baby	Recall@20	0.1802	0.1915	+0.0113
Amazon-Baby	NDCG@20	0.0818	0.0886	+0.0068
Amazon-Sports	Recall@20	0.1135	0.1205	+0.0070
Tiktok	Recall@20	0.1085	0.1198	+0.0113
Online A/B testing results on a large-scale advertising platform.
Kuaishou Advertising Platform	CTR	Not reported in the paper	Not reported in the paper	+1.48%
Kuaishou Advertising Platform	GMV	Not reported in the paper	Not reported in the paper	+1.63%

Experiment Figures

Conceptual illustration of the 'Semantic Heterogeneity' problem.

Main Takeaways

Optimal Transport alignment provides consistent gains over methods relying solely on contrastive learning or attention fusion, validating the distribution-level alignment hypothesis.
The adaptive transport matrix is crucial; ablation studies (implied by method description/results logic) show that purely geometric transport is insufficient without task-specific adaptation.
The framework effectively scales to industrial settings, evidenced by the online deployment metrics (CTR/GMV lift).

📚 Prerequisite Knowledge

Prerequisites

Graph Neural Networks (specifically LightGCN and GAT)
Optimal Transport theory (Wasserstein distance, Sinkhorn algorithm)
Contrastive Learning (InfoNCE loss)
Basics of LLM and LVM embeddings

Key Terms

Optimal Transport (OT): A mathematical framework for finding the most efficient way to transform one probability distribution into another, used here to align semantic feature spaces

Wasserstein distance: A distance measure between probability distributions defined by optimal transport; informally, the 'earth mover's distance'

Sinkhorn-Knopp algorithm: An iterative algorithm used to efficiently solve entropy-regularized optimal transport problems

InfoNCE: A contrastive loss function that maximizes mutual information between positive pairs while minimizing it for negative pairs

LightGCN: A simplified Graph Convolutional Network for recommendation that removes non-linearities and feature transformation to focus on neighborhood aggregation

LLaVA: Large Language-and-Vision Assistant—a multimodal model capable of processing and understanding both image and text inputs

CTR: Click-Through Rate—the ratio of users who click on a specific link to the number of total users who view a page, email, or advertisement

GMV: Gross Merchandise Value—the total value of merchandise sold over a given period of time through a customer-to-customer exchange site

Qwen: A series of large language models developed by Alibaba Cloud

QwQ: A reasoning-focused large language model used here to infer user preferences from interaction history