Empowering Contrastive Federated Sequential Recommendation with LLMs

📝 Paper Summary

Federated Sequential Recommendation LLM-augmented Recommendation Contrastive Learning

LUMOS addresses data scarcity in federated sequential recommendation by using on-device LLMs to generate synthetic future, paraphrased, and counterfactual user behaviors for private contrastive learning.

Core Problem

Federated sequential recommendation (FedSeqRec) suffers from sparse, short interaction histories on individual client devices, making it hard to learn stable user representations without centralizing data.

Why it matters:

Standard federated learning (FL) fails on cold-start or infrequent users because local data is too scarce to train powerful sequence encoders like SASRec
Traditional data augmentations (masking, cropping) designed for large centralized datasets often destroy semantic structure when applied to short local sequences
Existing contrastive FL methods often compromise privacy by sharing embeddings or gradients, or require heavy server-side coordination

Concrete Example: A user might have only 5 interactions (e.g., buying a phone case). A standard federated model sees just this short sequence. LUMOS uses an LLM to hallucinate that the user *might* next buy a screen protector (future view) or rephrases the history as 'purchased mobile accessories' (paraphrased view), providing richer signals for training.

Key Novelty

LUMOS: Local LLM-driven Multi-view Optimization for Sequential recommendation

Treats the local LLM as a privacy-preserving 'behavioral generator' that creates synthetic data (plausible futures, paraphrased histories, and hard negatives) directly on the client device
Integrates these synthetic sequences into a tri-view contrastive learning objective that forces the recommendation model to align real user history with plausible variations while pushing away inconsistent behaviors

Architecture

The LUMOS framework illustrating the client-side workflow: LLM generation of three views (Future, Paraphrase, Counterfactual) from the raw sequence, followed by shared encoding and tri-view contrastive optimization.

Evaluation Highlights

Outperforms state-of-the-art federated baseline ConFedSRS(SASRec) by 6.2–7.6% HR@20 across Amazon and MIND datasets
Surpasses centralized SASRec training on all datasets, demonstrating that LLM-augmented local training can beat centralized training on raw data
Maintains robustness against adversarial noise, with only ~0.01 drop in HR@20 even when 90% of knowledge is exposed to attackers

Breakthrough Assessment

8/10

Significantly improves federated recommendation by effectively utilizing client-side LLMs for data augmentation, surpassing even centralized baselines. A strong practical contribution to privacy-preserving AI.

⚙️ Technical Details

Problem Definition

Setting: Federated Sequential Recommendation where N users hold private interaction sequences S_u

Inputs: User interaction sequence S_u = [v_{u,1}, ..., v_{u,T_u}]

Outputs: Next-item probability distribution p_theta over item set V

Pipeline Flow

Local Client: Interaction Sequence S_u
Local LLM Generator: Produces Future, Paraphrase, Counterfactual views
Shared Backbone Encoder: Encodes all 4 sequences (Real + 3 Synthetic)
Local Optimization: Minimizes Next-Item Prediction Loss + Tri-View Contrastive Loss
Server: Aggregates model parameters via FedAvg

System Modules

Local LLM Generator

Generate synthetic behavioral sequences conditioned on user history

Model or implementation: Pre-trained LLM (black-box to the recommender)

Sequential Encoder

Encode sequences into dense vector representations

Model or implementation: SASRec (Self-Attentive Sequential Recommendation) or GRU4Rec

Novel Architectural Elements

Integration of an on-device LLM strictly as a data generator (not an encoder) within the federated loop
Tri-view contrastive topology where gradients flow from synthetic views (future, paraphrase, negative) to the shared encoder without exposing the views themselves

Modeling

Base Model: SASRec (default) or GRU4Rec

Training Method: Federated Averaging (FedAvg) with Local Contrastive Optimization

Objective Functions:

Purpose: Predict the next item in the sequence.

Formally: L_sup = - sum(log p_theta(v_{u,t+1} | v_{u,1:t}))
Purpose: Align user representation with plausible variations and separate from negatives.

Formally: L_CL = - log ( exp(sim(anchor, pos)) / [exp(sim(anchor, pos)) + sum(exp(sim(anchor, neg)))] ) summed over positive views (future, paraphrase)
Purpose: Combine objectives.

Formally: L_u = L_sup + lambda_CL * L_CL

Key Hyperparameters:

learning_rate: 1e-3
weight_decay: 1e-5
batch_size: 128
+ 5 more
local_epochs: 5
contrastive_weight_lambda_CL: 0.1
temperature_tau: 0.07
client_sample_rate: 10%
rounds: 100

Compute: Trained on NVIDIA A100 GPUs (server-side simulation)

Comparison to Prior Work

vs. FedSeq: LUMOS adds local contrastive learning with LLM-generated views
vs. ConFedSRS: LUMOS uses semantic LLM augmentations (future, paraphrase, counterfactual) instead of random masking/cropping, yielding much stronger signals for short sequences
vs. FedCL: LUMOS performs contrastive learning entirely locally without sharing representations or gradients for negative sampling [not cited in paper]

Limitations

Relies on the availability of a capable LLM on client devices, which may be computationally prohibitive for mobile phones
Performance depends on the quality of the LLM's generations; hallucinations could introduce noise
No specific details provided on the LLM used (e.g., Llama-2, GPT-3.5) or its computational cost on the client
Experiments are simulated on a server (A100 GPU) rather than real edge devices

Reproducibility

Code availability is not provided in the paper. The paper specifies datasets (Amazon, MIND) and hyperparameters, but lacks specific details on which LLM was used (size, version) for the generation step or the exact prompts used.

📊 Experiments & Results

Evaluation Setup

Next-item prediction on sparse user sequences (filtered to min 5 interactions, max 50)

Benchmarks:

Amazon Cell Phone (Sequential Recommendation)
Amazon Baby (Sequential Recommendation)
MIND (News Recommendation)

Metrics:

HR@20 (Hit Ratio)
NDCG@20 (Normalized Discounted Cumulative Gain)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
LUMOS consistently outperforms both centralized and federated baselines across all datasets.
Amazon Cell Phone	HR@20	0.1118	0.1878	+0.0760
Amazon Baby	HR@20	0.0984	0.1604	+0.0620
MIND	HR@20	0.1345	0.2015	+0.0670
Amazon Cell Phone	NDCG@20	0.0612	0.0740	+0.0128
Ablation studies confirm the necessity of all three LLM-generated views.
Average (implied)	HR@20	Not reported in the paper	Not reported in the paper	Not reported in the paper

Experiment Figures

Performance (MRR) scaling with the number of users per epoch (32 to 512)

Robustness of LUMOS against varying levels of adversarial knowledge (30% to 90%)

Main Takeaways

LUMOS achieves superior performance compared to both standard FedSeqRec and centralized SASRec, proving that semantic augmentation compensates for data fragmentation
The 'Predictive Future' view is the most critical component, followed closely by Paraphrased and Counterfactual views
Performance scales positively with the number of participating clients (improving from 32 to 512 users per round)
Robustness experiments show negligible performance drops (HR@20 drops ~0.01) even under high levels of adversarial noise (90%)

📚 Prerequisite Knowledge

Prerequisites

Federated Learning (FedAvg protocol)
Sequential Recommendation (SASRec, GRU4Rec)
Contrastive Learning (InfoNCE loss)
Large Language Models (generative capabilities)

Key Terms

FedSeqRec: Federated Sequential Recommendation—training sequential recommenders across decentralized devices without sharing raw user data

SASRec: Self-Attentive Sequential Recommendation—a Transformer-based model that predicts the next item in a sequence using self-attention mechanisms

HR@20: Hit Ratio at 20—the percentage of times the correct next item appears in the top-20 predicted recommendations

NDCG@20: Normalized Discounted Cumulative Gain at 20—a ranking metric that credits the model more for placing the correct item higher in the top-20 list

Tri-view contrastive learning: A training objective that aligns an anchor representation with two positive views (future, paraphrase) and pushes it away from a negative view (counterfactual)

Counterfactual view: A synthetic data sample generated to be plausible but intentionally conflicting with the user's actual preferences (e.g., buying a totally unrelated item)

FedAvg: Federated Averaging—the standard algorithm for aggregating model updates from multiple clients in federated learning