FELLAS: Enhancing Federated Sequential Recommendation with LLM as External Services

📝 Paper Summary

Federated Sequential Recommendation Privacy-Preserving Recommendation LLM as a Service (LLMaaS)

FELLAS improves federated sequential recommendation by querying external LLMs for item and sequence embeddings, using privacy-preserving perturbation and contrastive learning to securely transfer knowledge to client models.

Core Problem

Federated sequential recommendation (FedSeqRec) models must be lightweight for edge devices, resulting in poor performance compared to centralized models, while powerful LLMs are too resource-intensive to run locally or fine-tune federatedly.

Why it matters:

Traditional centralized training poses significant data leakage risks and violates privacy regulations like GDPR and CCPA.
Existing solutions either sacrifice performance for privacy (lightweight models) or incur impractical communication/compute costs (fine-tuning LLMs on-device).
Directly using external LLM APIs with raw user data violates the core privacy principles of federated learning.

Concrete Example: A client wants to use a powerful external LLM to encode their shopping history [Laptop, Mouse, HDMI Cable] to get better recommendations. Sending this raw sequence to the LLM server exposes private interests. Existing methods force the client to use a weaker local model instead.

Key Novelty

Federated Sequential Recommendation with LLM as External Service (FELLAS)

Treats the LLM as a remote 'oracle' service rather than a local backbone, querying it for item-text and sequence-context embeddings to augment lightweight local models.
Implements a 'd_chi-privacy' perturbation mechanism that shuffles and alters user sequences before sending them to the LLM, ensuring the external server sees only noisy, plausible data.
Uses contrastive learning to align the client's local model representations with the semantic knowledge returned by the LLM from the noisy sequences.

Architecture

The FELLAS architecture, illustrating the interactions between Clients, the Central Server, and the External LLM Server.

Evaluation Highlights

Demonstrates effectiveness on three Amazon datasets (Beauty, Office Products, Patio Lawn and Garden) using SASRec and BERT4Rec backbones.
Validates privacy protection against two novel inference attacks (SIA and SIAUI) where the LLM server and central server act as adversaries.

Breakthrough Assessment

7/10

A practical architectural shift for FedRec that bridges the gap between lightweight edge models and powerful cloud LLMs without compromising the 'no-raw-data-sharing' rule of federated learning.

⚙️ Technical Details

Problem Definition

Setting: Federated Sequential Recommendation where N clients collaboratively train a model without sharing private data

Inputs: User interaction sequences (temporally ordered item lists)

Outputs: Next item prediction

Pipeline Flow

Item-Level Service: Central Server → Query LLM → Fuse Item Embeddings
Sequence-Level Service (Training): Client → Perturb Sequence → Query LLM → Contrastive Learning

System Modules

Item-Level Representation Service

Enrich ID-based item embeddings with semantic information

Model or implementation: External LLM (e.g., BERT, Llama)

Sequence Perturbation Module

Protect user privacy before external querying

Model or implementation: d_chi-privacy mechanism

Sequence-Level Representation Service

Provide semantic sequence encoding

Model or implementation: External LLM (e.g., Llama2, Llama3)

Local Sequential Recommender

Learn user preferences and predict next item

Model or implementation: Lightweight Sequence Model (SASRec or BERT4Rec)

Novel Architectural Elements

Dual-service architecture: Central server queries Item Service, Clients query Sequence Service
Privacy-preservation layer (d_chi-privacy) inserted specifically between Client and External LLM Service

Modeling

Base Model: Sequential Recommenders (SASRec, BERT4Rec)

Training Method: Federated Learning with auxiliary Contrastive Learning

Objective Functions:

Purpose: Minimize prediction error for next item.

Formally: Standard cross-entropy loss on recommendation task.
Purpose: Transfer knowledge from noisy LLM embeddings to local model.

Formally: Contrastive learning loss minimizing distance between local representation and LLM representation of the perturbed sequence.

Training Data:

Amazon Beauty
Amazon Office Products
Amazon Patio Lawn and Garden

Comparison to Prior Work

vs. FedSeqRec: FELLAS integrates external LLM knowledge for both items and sequences, whereas FedSeqRec relies solely on local ID-based learning.
vs. FedPA: FELLAS addresses sequential recommendation (not just collaborative filtering) and introduces a privacy-preserving mechanism for sharing user *sequences* with LLMs, which FedPA does not handle.
vs. Split-Learning LLMs: FELLAS uses LLMs as an API service, avoiding the massive communication overhead of split-training large models.

Limitations

Reliance on an external LLM service implies dependency on third-party availability and API costs.
The perturbation mechanism introduces noise which might limit the upper bound of knowledge transfer compared to accessing raw data.
Communication of embeddings between clients and the LLM server adds a new communication cost (though less than model transmission).

Reproducibility

Code availability is not provided in the paper text. The paper uses public datasets (Amazon) and standard base models (SASRec, BERT4Rec, Llama2, Llama3).

📊 Experiments & Results

Evaluation Setup

Federated training on partitioned user data with external LLM queries

Benchmarks:

Amazon Beauty (Sequential Recommendation)
Amazon Office Products (Sequential Recommendation)
Amazon Patio Lawn and Garden (Sequential Recommendation)

Metrics:

Hit Ratio (HR)
NDCG
Statistical methodology: Not explicitly reported in the paper

Main Takeaways

FELLAS aims to improve FedSeqRec performance by leveraging the semantic reasoning of LLMs without violating privacy.
The framework is generic and compatible with various base sequential models (e.g., SASRec, BERT4Rec) and various external LLMs (BERT, Llama2, Llama3).
Two proposed inference attacks (SIA and SIAUI) utilize similarity measures and update information to attempt to recover user items, serving as empirical validation for the privacy mechanism.

📚 Prerequisite Knowledge

Prerequisites

Federated Learning (FedAvg)
Sequential Recommendation (SASRec/BERT4Rec)
Local Differential Privacy
Contrastive Learning

Key Terms

FedSeqRec: Federated Sequential Recommendation—training sequential recommenders across decentralized devices while keeping data local

d_chi-privacy: A generalization of local differential privacy for metric spaces, ensuring that indistinguishable inputs produce similar output distributions based on a distance metric

LLM as a Service: Using large language models via API calls to obtain embeddings or text generation, rather than hosting or fine-tuning the model locally

SIA: Similarity-based Inference Attack—an adversarial method proposed in the paper to guess user interactions based on embedding similarities

Contrastive Learning: A learning paradigm where the model learns to pull representations of similar (positive) pairs together and push dissimilar (negative) pairs apart