Laser: Parameter-Efficient LLM Bi-Tuning for Sequential Recommendation with Collaborative Information

📝 Paper Summary

Sequential Recommendation LLM Adaptation Parameter-Efficient Fine-Tuning

Laser adapts frozen LLMs for sequential recommendation by adding trainable prefix/suffix tokens and using a Mixture-of-Experts transformer to integrate user-specific collaborative signals.

Core Problem

Existing LLM-based recommenders are resource-heavy and integrate collaborative signals via simple linear projections, failing to capture diverse user characteristics.

Why it matters:

Traditional ID-based methods miss rich semantic information in item descriptions
Standard LLM fine-tuning is computationally expensive and struggles to align language space with recommendation space
Simple projections of collaborative signals ignore the complexity of different user types, leading to suboptimal personalization

Concrete Example: A user who buys 'Kaytee Aspen Bedding' and 'KONG Dog Toy' has specific pet-owner traits. A standard LLM might just see text, while a simple projection mixes all user signals uniformly. Laser uses specific 'experts' to process this pet-owner signal differently from a tech-buyer signal.

Key Novelty

Bi-Tuning with MoE-based Collaborative Integration (Laser)

Bi-Tuning: Freezes the LLM and tunes only added 'virtual tokens' at the start (prefix) and end (suffix) of the input to adapt the model to recommendation tasks efficiently
M-Former: An MoE-based module that selects specific 'query experts' to process ID-based collaborative signals, ensuring different user types are handled by specialized parameters before integration into the LLM

Architecture

The Laser framework overview, detailing the input processing, M-Former module, and Bi-Tuning mechanism.

Evaluation Highlights

+13.27% NDCG@10 improvement on Pet Supplies dataset compared to the second-best baseline (LlamaRec)
+17.65% MRR improvement on Scientific dataset compared to the best traditional method (SASRec)
Zero-shot performance on Scientific dataset (Recall@10 ~0.97) surpasses baselines trained on 100% of data using only 5% of training data

Breakthrough Assessment

7/10

Strong empirical gains and a sensible architecture for combining ID-based signals with LLMs. The Bi-Tuning and MoE integration are effective, though the core concept of soft prompting is established.

⚙️ Technical Details

Problem Definition

Setting: Sequential recommendation: Predict the next item i_{N+1} given a sequence of historical interactions S_u = {i_1, ..., i_N}

Inputs: User interaction sequence containing both item IDs and textual attributes (title, category, brand)

Outputs: Embedding of the predicted next item (used for similarity search against candidate items)

Pipeline Flow

Input Construction (Text Template + ID Sequence)
Collaborative Encoding (Frozen ID-based Model)
M-Former (MoE Router -> Query Expert Selection -> Fusion)
LLM Processing (Prefix + Text + Suffix)
Suffix Extraction & Recommendation

System Modules

ID-based Encoder

Encode user interaction history into ID-based embeddings to capture collaborative signals

Model or implementation: BERT4Rec (frozen)

M-Former

Integrate collaborative embeddings into query experts selected specifically for the user type

Model or implementation: MoE-based Transformer (Trainable)

LLM Backbone

Process the text sequence conditioned on the collaborative prefix to generate user representation

Model or implementation: ChatGLM2-6B (frozen)

Novel Architectural Elements

Bi-Tuning: Simultaneous optimization of prefix tokens (carrying collaborative signal) and a single suffix token (aggregating representation) on a frozen LLM
M-Former: Using a Mixture-of-Experts router to select specific 'query experts' that act as the prompt prefix, conditioning the LLM on user-type-specific collaborative signals

Modeling

Base Model: ChatGLM2-6B

Training Method: Multi-task learning with Two-Stage Training

Objective Functions:

Purpose: Maximize similarity between user embedding and ground-truth next item embedding.

Formally: Contrastive Loss L_IIC using cosine similarity and temperature tau.
Purpose: Ensure diverse usage of experts in M-Former.

Formally: Load Balancing Loss L_LB = K * sum(f_j * P_j) where f_j is fraction of items dispatched to expert j.

Adaptation: Bi-Tuning (tuning only virtual prefix/suffix tokens and M-Former parameters)

Trainable Parameters: 0.135M (approx. 0.07% of backbone, excluding M-Former overhead; total trainable ~183.3M with embeddings)

Key Hyperparameters:

batch_size: 8
learning_rate: 5e-4
num_epochs: 20
+ 4 more
K (experts): 8
L (virtual tokens): 32
temperature_tau: 1.0
loss_weight_lambda: 0.01

Compute: Not explicitly reported in the paper

Comparison to Prior Work

vs. LlamaRec: Laser uses LLM as an encoder for retrieval/embedding generation rather than just reranking, and integrates ID signals via MoE.
vs. Standard Soft Prompting: Laser uses a suffix token for representation extraction (converting language space to rec space) rather than [EOS] or average pooling.
vs. Simple Projection Methods: Laser uses M-Former (MoE) to handle diverse user types instead of a unified linear layer.

Limitations

Depends on a pre-trained ID-based recommender (BERT4Rec) for collaborative signals; performance scales linearly with this base model quality.
The unified template strategy, while robust, may still be sensitive to specific phrasing changes (as shown in ablation).
Requires two-stage training (embedding generation -> fine-tuning) which increases training complexity.

Reproducibility

Code availability is not provided. The paper details the backbone (ChatGLM2-6B) and baseline (BERT4Rec). Hyperparameters for the proposed method are listed in Appendix.

📊 Experiments & Results

Evaluation Setup

Sequential next-item prediction using leave-one-out splitting

Benchmarks:

Amazon Industrial and Scientific (Sequential Recommendation)
Amazon Arts, Crafts and Sewing (Sequential Recommendation)
Amazon Pet Supplies (Sequential Recommendation)

Metrics:

Recall@10
NDCG@10
MRR
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Laser consistently outperforms both traditional and LLM-based baselines across all three datasets.
Scientific	NDCG@10	0.0764	0.0970	+0.0206
Pet	NDCG@10	0.0754	0.0898	+0.0144
Arts	Recall@10	0.1368	0.1489	+0.0121
Ablation studies confirm the necessity of the MoE module and the specific Bi-Tuning strategy.
Scientific	Recall@10	0.1261	0.1396	+0.0135
Scientific	Recall@10	0.1128	0.1396	+0.0268

Experiment Figures

Zero-shot and low-resource performance curves on the Scientific dataset.

Impact of the underlying ID-based recommender (SASRec vs BERT4Rec) on Laser's performance.

Main Takeaways

Laser significantly outperforms traditional ID-based methods (SASRec, BERT4Rec) and existing LLM-based methods (LlamaRec, LLM4REC) across all metrics.
The trainable suffix strategy is far superior to average pooling or using [EOS] for embedding generation (+29% NDCG vs [EOS]).
Zero-shot performance is strong: Laser trained on 'Pet' achieves high results on 'Scientific' without any fine-tuning, demonstrating strong generalization.
Performance is dependent on the frozen ID-model quality; better ID-models lead to better Laser performance.

📚 Prerequisite Knowledge

Prerequisites

Sequential Recommendation
Large Language Models (LLMs)
Prompt Tuning / Soft Prompts
Mixture of Experts (MoE)

Key Terms

Bi-Tuning: A parameter-efficient tuning method proposed in this paper that inserts trainable virtual tokens at both the prefix and suffix of the input text while freezing the LLM

M-Former: A lightweight MoE-based querying transformer designed to integrate ID-based collaborative information into the prefix tokens

Collaborative Information: Signals derived from user-item interactions (typically via ID embeddings) that capture latent user preferences

MoE: Mixture of Experts—a machine learning technique where different parts of the model (experts) specialize in different types of inputs, controlled by a router

Virtual Tokens: Learnable vectors inserted into the input sequence that don't correspond to actual vocabulary words but optimize the model for a specific task

NDCG: Normalized Discounted Cumulative Gain—a measure of ranking quality that accounts for the position of relevant items in the recommendation list

MRR: Mean Reciprocal Rank—a metric that evaluates the rank of the first correct recommendation

SASRec: Self-Attentive Sequential Recommendation—a baseline model using self-attention to capture sequential patterns from item IDs