CoRA: Collaborative Information Perception by Large Language Model's Weights for Recommendation

📝 Paper Summary

LLM for Recommendation Collaborative Filtering integration in LLMs Parameter-efficient fine-tuning (PEFT)

CoRA injects collaborative filtering signals into Large Language Models by transforming user-item embeddings into temporary model weights (like LoRA) rather than modifying the text prompt.

Core Problem

Existing methods integrate collaborative information by adding tokens to the text prompt and fine-tuning, which degrades the LLM's general reasoning abilities and disrupts text semantics.

Why it matters:

Fine-tuning on recommendation data causes 'catastrophic forgetting,' weakening the LLM's ability to reason, summarize, and understand general text
Inserting collaborative embeddings as tokens into prompts breaks the natural language structure, confusing the LLM and leading to hallucinations or sub-optimal predictions
Current approaches force a trade-off between leveraging collaborative signals (crucial for recommendation accuracy) and maintaining the LLM's powerful text inference capabilities

Concrete Example: When Vicuna-7B is fine-tuned with standard recommendation prompts (like TALLRec), its performance on general knowledge tasks (WikiFact) drops significantly. Additionally, inserting user/item tokens into a prompt asking to 'repeat the description' causes the model to output gibberish instead of the original text.

Key Novelty

Collaborative LoRA (CoRA)

Instead of treating collaborative data as input text, CoRA treats it as a modifier to the model's processing logic (weights)
A generator converts user/item embeddings into 'collaborative weights' that temporarily merge with the LLM's attention weights, similar to how LoRA adds update matrices
This allows the LLM to 'perceive' user preferences without changing its input text or permanently altering its pre-trained general knowledge

Architecture

The overall framework of CoRA. It illustrates the flow from User/Item Embeddings -> Collaborative Query Generator -> Low-Rank Weight Generation -> Injection into LLM Backbone.

Evaluation Highlights

Outperforms state-of-the-art LLM-based method (BinLLM) by significant margins on Amazon-Book and Yelp datasets
Achieves higher recommendation accuracy than traditional Collaborative Filtering models (like SASRec and LightGCN) while retaining text reasoning capabilities
Preserves general LLM capabilities (evaluated on WikiFact, SocialIQA) where fine-tuning methods like TALLRec show significant degradation

Breakthrough Assessment

7/10

Novel approach applying vision-inspired weight-space injection (VLoRA) to recommendation. effectively solving the prompt-disruption problem. Strong results, though reliance on pre-trained CF embeddings limits it to a hybrid role.

⚙️ Technical Details

Problem Definition

Setting: Click-through rate prediction (binary classification) using both collaborative embeddings and textual descriptions

Inputs: User u, Item i, Textual descriptions t_u, t_i, Pre-trained CF embeddings e_u, e_i

Outputs: Prediction y_ui indicating likelihood of interaction (0 or 1)

Pipeline Flow

CF Embedding Extraction
Collaborative Query Generation
Weight Projection (Low-Rank)
LLM Inference

System Modules

CF Model

Extract latent collaborative representations for users and items

Model or implementation: Pre-trained CF model (e.g., Matrix Factorization or GNN)

Collaborative Query Generator (Weight Generation)

Inject collaborative information into learnable query embeddings via cross-attention

Model or implementation: Transformer Decoder blocks (N blocks)

Low-Rank Projector (Weight Generation)

Map query embeddings into delta weights for the LLM

Model or implementation: Two linear layers (W_FC and W_proj)

LLM Backbone

Generate recommendation prediction using text prompts and merged collaborative weights

Model or implementation: Vicuna-7B (Frozen)

Novel Architectural Elements

Parameter Space Alignment: Converting user/item embeddings into dynamic weight updates (ΔW) rather than input tokens
Collaborative Query Generator: A module specifically designed to translate CF signals into the weight space of a frozen LLM

Modeling

Base Model: Vicuna-7B

Training Method: Supervised Learning on the Query Generator and Projector (LLM is frozen)

Objective Functions:

Purpose: Minimize prediction error for recommendation.

Formally: Standard Cross-Entropy Loss on the generated 'Yes'/'No' token logits.

Adaptation: Dynamic Low-Rank Adaptation (weights generated per sample)

Trainable Parameters: Collaborative Query Generator and Linear Projectors (LLM backbone is frozen)

Training Data:

Amazon-Book
Yelp
MovieLens-1M (mentioned in baselines, implied usage)

Compute: Not reported in the paper

Comparison to Prior Work

vs. TALLRec: CoRA does not fine-tune the LLM backbone, preserving general knowledge
vs. CoLLM/LlaRA: CoRA injects signals into weights, not input tokens, preventing semantic disruption of the prompt
vs. BinLLM: CoRA uses continuous weight adaptation rather than discrete binary text encoding
+ 1 more
vs. VLoRA [not cited in paper]: VLoRA encodes visual features into weights; CoRA adapts this concept for collaborative filtering signals in recommendation

Limitations

Relies on an external pre-trained Collaborative Filtering model to provide initial embeddings
Inference cost may be higher than simple prompting due to the weight generation step for each user-item pair
Performance depends on the quality of the underlying CF model

Reproducibility

Code availability is not provided in the paper. The paper uses public datasets (Amazon-Book, Yelp). Hyperparameters for the generator (number of queries k, number of blocks N) are discussed.

📊 Experiments & Results

Evaluation Setup

Click-through rate (CTR) prediction on standard recommendation datasets

Benchmarks:

Amazon-Book (CTR Prediction)
Yelp (CTR Prediction)

Metrics:

AUC (Area Under Curve)
Accuracy
F1 Score
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
CoRA is compared against traditional CF methods and LLM-based methods. Note: Specific numeric values for baselines are not extractable from the provided text (e.g., 'See Table X'). Qualitative descriptions are provided in Main Takeaways.
WikiFact, SocialIQA, XSum	Performance Score	Significantly lower (visualized in Fig 1)	Comparable to Base LLM (visualized in Fig 1)	Positive (Qualitative)

Experiment Figures

Radar chart comparing General Knowledge (WikiFact, SocialIQA, XSum) and Recommendation Knowledge (UProfile, ITMatching) between Base LLM, TALLRec, and CoRA.

Qualitative example of semantic disruption. It shows Vicuna-7B's response to a prompt with and without inserted collaborative tokens.

Main Takeaways

CoRA effectively integrates collaborative information without fine-tuning the LLM, preventing catastrophic forgetting of general knowledge.
The method eliminates the issue of 'semantic disruption' where inserting embedding tokens confuses the LLM's understanding of text prompts.
Parameter-space alignment (injecting weights) allows the LLM to switch seamlessly between recommendation tasks (using CoRA weights) and general tasks (using frozen weights).
The approach generalizes the concept of LoRA to be input-dependent, generating specific adapters for each user-item pair.

📚 Prerequisite Knowledge

Prerequisites

Collaborative Filtering (CF) concepts
Low-Rank Adaptation (LoRA)
Transformer architecture (specifically Multi-Head Attention)
Large Language Models (LLMs)

Key Terms

LoRA: Low-Rank Adaptation—a technique to fine-tune models by adding small, trainable rank-decomposition matrices to existing weights

CF: Collaborative Filtering—recommendation methods relying on patterns of user-item interactions (e.g., 'users who bought X also bought Y')

SASRec: Self-Attentive Sequential Recommendation—a sequence-based recommendation model using self-attention

LightGCN: Light Graph Convolutional Network—a simplified graph neural network for recommendation that removes non-linearities

LLMRec: Large Language Model for Recommendation—the subfield of adapting LLMs to solve recommendation tasks

Instruction Tuning: Fine-tuning LLMs on datasets formatted as instructions to improve performance on specific tasks

Input Space Alignment: Mapping external data (like user IDs) into the same vector space as text tokens so the LLM can process them as input

Parameter Space Alignment: Mapping external data directly into the model's weights (e.g., attention matrices) rather than its input tokens