Dual Personalization on Federated Recommendation

📝 Paper Summary

Federated Recommendation Systems Personalized Federated Learning On-device Recommendation

PFedRec is a federated recommendation framework that simultaneously learns a personalized local score function and fine-tunes item embeddings locally to capture user-specific preferences without exposing user data.

Core Problem

Existing federated recommendation systems typically learn a heavyweight global model on the server and share identical item embeddings across all users, failing to capture fine-grained user-specific views on item relationships.

Why it matters:

Standard federated approaches hinder the deployment of lightweight on-device models by relying on server-centric heavy models
Sharing exactly the same item embeddings ignores that different users may perceive the proximity and utility of items differently
Privacy concerns require minimizing data exposure, yet personalization is critical for effective recommendation

Concrete Example: In standard FedRec, if User A and User B both like 'Sci-Fi', they share the exact same embedding for the movie 'Matrix'. However, User A might view 'Matrix' as similar to 'Inception' (plot complexity), while User B views it as similar to 'Star Wars' (action). A global embedding forces a compromise, losing these specific personal views.

Key Novelty

Dual Personalization Mechanism (PFedRec)

Splits the recommendation model into a personalized score function (kept local) and item embeddings (shared but locally fine-tuned)
Uses a 'post-tuning' strategy where clients first update their score function using global embeddings, then fine-tune the item embeddings on their local data to create a personalized view before aggregation

Architecture

The conceptual framework of PFedRec compared to FedMF and FedNCF. It illustrates the 'Dual Personalization' approach where both the score function and item embeddings are personalized.

Evaluation Highlights

+13.53% HR@10 improvement on MovieLens-100K when integrating Dual Personalization into FedNCF
+18.29% NDCG@10 improvement on MovieLens-100K when integrating Dual Personalization into FedNCF
PFedRec outperforms state-of-the-art federated baselines (FedMF, FedNCF, FedRecon) across four datasets (MovieLens, Lastfm-2K, Amazon-Video)

Breakthrough Assessment

7/10

Offers a simple yet effective architectural shift for federated recommendation by personalizing item embeddings, a component usually kept global. Significant empirical gains, though the underlying technique is a straightforward application of fine-tuning.

⚙️ Technical Details

Problem Definition

Setting: Implicit feedback recommendation in a federated setting where user data is distributed across devices

Inputs: User-item interaction matrix (distributed locally), Global item embeddings

Outputs: Personalized recommendation scores for items

Pipeline Flow

Server Initialization: Initialize global item embeddings and score function parameters
Client Download: Selected clients download global item embeddings
Local Update (Client): Update local personalized score function while keeping item embeddings fixed
Local Post-tuning (Client): Fine-tune item embeddings using the updated score function
Upload: Client uploads updated item embeddings to server
Aggregation (Server): Server averages uploaded item embeddings to update global view

System Modules

Personalized Score Function (S_i)

Captures user-specific decision logic for ranking items

Model or implementation: Multi-layer Neural Network (MLP) or Dot Product

Item Embedding Module (E_i)

Maintains vector representations of items

Model or implementation: Embedding Lookup Table

Novel Architectural Elements

Deprecation of explicit user embeddings in favor of a personalized neural score function
Decoupled optimization: Score function is private/local-only, while item embeddings are shared-but-personalized via local fine-tuning cycles

Modeling

Base Model: Neural network-based score function (1-layer MLP used for simplification in main experiments)

Training Method: Federated Optimization with Bi-level Objective

Objective Functions:

Purpose: Minimize local prediction error.

Formally: Binary Cross-Entropy Loss L_i = - sum(log y_hat) for positive - sum(log(1-y_hat)) for negative samples.
Purpose: Bi-level optimization.

Formally: min over theta_m and {theta_i} sum(alpha_i * L_i), where theta_i includes personalized item embeddings obtained via gradient descent steps from global theta_m.

Adaptation: Fine-tuning global item embeddings on local data

Trainable Parameters: Item embeddings (theta_m), Score function weights (theta_s)

Training Data:

Implicit feedback (interaction matrix)
Negative sampling (4 negatives per positive instance)

Key Hyperparameters:

embedding_size: 32
batch_size: 256
negative_sampling_ratio: 4

Compute: Not reported in the paper

Comparison to Prior Work

vs. FedMF/FedNCF: PFedRec keeps the score function local and personalizes item embeddings via fine-tuning, whereas baselines typically share exact item embeddings or global score functions.
vs. FedRecon: PFedRec maintains persistent local score functions and fine-tunes item embeddings, rather than retraining local parameters from scratch.
vs. MetaMF: PFedRec does not require a central meta-network or upload model parameters that could leak interaction info (beyond embeddings).
+ 1 more
vs. FedFast [not cited in paper]: FedFast uses active aggregation to speed up convergence, but PFedRec focuses on dual personalization logic rather than aggregation speed.

Limitations

Communication cost of transmitting item embeddings can be high for very large item sets, though the paper suggests clients only need embeddings for interacted items.
Requires local training on client devices, which assumes sufficient local compute capability.
Implicitly assumes user preferences can be captured by modifying item embeddings from a global initialization, which might not hold for extremely divergent user groups.

Reproducibility

Code: https://github.com/Zhangcx19/IJCAI-23-PFedRec

Code is publicly available at https://github.com/Zhangcx19/IJCAI-23-PFedRec. Datasets (MovieLens, Lastfm-2K, Amazon-Video) are standard public benchmarks.

📊 Experiments & Results

Evaluation Setup

Leave-one-out evaluation on implicit feedback datasets

Benchmarks:

MovieLens-100K (Movie Recommendation)
MovieLens-1M (Movie Recommendation)
Lastfm-2K (Music Recommendation)
Amazon-Video (Product Recommendation)

Metrics:

Hit Ratio @ 10 (HR@10)
Normalized Discounted Cumulative Gain @ 10 (NDCG@10)
Statistical methodology: Reported mean and standard deviation over 5 repetitions

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
PFedRec outperforms both centralized and federated baselines on MovieLens datasets.
MovieLens-100K	HR@10	66.38	71.62	+5.24
MovieLens-1M	HR@10	68.45	73.26	+4.81
MovieLens-100K	HR@10	60.62	68.82	+8.20
MovieLens-100K	NDCG@10	33.25	39.33	+6.08
MovieLens-100K	HR@10	71.62	70.84	-0.78
Amazon-Video	NDCG@10	38.87	39.12	+0.25

Experiment Figures

t-SNE visualization of item embeddings learned by baselines (FedNCF, FedRecon, FedMF) vs. PFedRec for a specific user.

Impact of using different item embeddings (Own vs. Global vs. Random) during client inference.

Main Takeaways

Personalized item embeddings significantly improve recommendation performance compared to shared global embeddings, especially on dense datasets like MovieLens.
The proposed Dual Personalization mechanism is generalizable and can be integrated into existing methods like FedMF, FedNCF, and FedRecon to boost their performance.
Clients using their own personalized item embeddings perform much better than using global or random embeddings, confirming the embeddings capture specific user preferences.
The method maintains robustness even when Local Differential Privacy (Laplacian noise) is applied to the uploaded item embeddings.

📚 Prerequisite Knowledge

Prerequisites

Federated Learning (FedAvg)
Matrix Factorization (MF)
Neural Collaborative Filtering (NCF)
Implicit feedback recommendation

Key Terms

Federated Learning: A machine learning technique where multiple decentralized edge devices training a model without exchanging local data samples

Item Embedding: Vector representations of items in a latent space, typically shared across users in standard collaborative filtering

Score Function: A function (often a neural network or dot product) that predicts a user's preference score for an item based on embeddings

Dual Personalization: The proposed mechanism that personalizes both the user's decision logic (score function) and their view of items (item embeddings)

Post-tuning: The strategy of fine-tuning the item embeddings locally on a client's data after updating the score function, generating a personalized item representation

HR@10: Hit Ratio at 10—the percentage of test cases where the target item is present in the top-10 recommendations

NDCG@10: Normalized Discounted Cumulative Gain at 10—a measure of ranking quality that accounts for the position of the hit in the top-10 list

Local Differential Privacy (LDP): A privacy standard where noise is added to individual data (here, embeddings) before it leaves the client device