Enhancing Serendipity Recommendation System by Constructing Dynamic User Knowledge Graphs with Large Language Models

📝 Paper Summary

Serendipity Recommendation LLM-based User Profiling Industrial Recommender Systems

This paper breaks filter bubbles in industrial recommendation systems by using LLMs to infer potential interests via dynamic knowledge graphs and deploying a specialized retrieval model that aligns user embeddings with these interests.

Core Problem

Industrial recommender systems suffer from feedback loops where systems recommend what users clicked, and users click what is recommended, creating filter bubbles that lack novelty.

Why it matters:

Filter bubbles reduce user satisfaction and retention by confining users to homogeneous content.
Existing serendipity methods lack data or use small models, while direct LLM inference is too slow (high latency) and prone to hallucinations for real-time industrial use.

Concrete Example: A user searches for 'basketball shoes'. A traditional system keeps recommending basketball shoes (homogeneous). A direct LLM might hallucinate irrelevant items or take seconds to respond. This system reasons: 'User likes basketball shoes' -> (Hypernym) 'Sports' -> (Co-hyponym) 'Sports drink', finding a relevant but surprising interest.

Key Novelty

Dynamic User Knowledge Graph with Two-Hop Reasoning & Multi-Agent Debate

Constructs a temporary knowledge graph for each user to reason from 'History' -> 'Core Demand' -> 'Potential Interest' (two-hop), rather than simple similarity matching.
Uses a multi-agent debate mechanism where LLM instances critique each other's reasoning to reduce hallucinations and ensure the inferred interest is actually relevant.
Deploys via a 'nearline' cache and a multi-task dual-tower model that forces the user representation to align with the new interest, combining the relevance of item-to-item retrieval with the conversion rates of user-to-item retrieval.

Architecture

The overall two-stage framework: (1) Offline/Nearline generation of User Potential Interests using LLMs and (2) Online Serendipity Retrieval.

Evaluation Highlights

+4.62% Exposure Novelty Rate and +4.85% Click Novelty Rate in online A/B testing on the Dewu app (10M+ users).
+0.15% Average View Duration per person, indicating users actually engage with the novel content.
96% of generated potential interests received a relevance score of 2 (highest) in offline human evaluation.

Breakthrough Assessment

7/10

Solid industrial application. While using LLMs for profiling isn't new, the specific integration of dynamic KG construction with a nearline architecture and a custom loss retrieval model for a massive user base is a significant practical engineering contribution.

⚙️ Technical Details

Problem Definition

Setting: Serendipity Recommendation in an Industrial Setting

Inputs: User search behavior sequence Q_i = {q_1, ..., q_m} and user profile S_i

Outputs: List of l items relevant to potential novel interests P_i

Pipeline Flow

Nearline Group: User History -> Multi-Agent Debate (LLM) -> Potential Interests -> Cache
Online Group: User Request -> Cache Lookup -> Retrieval Model -> Candidate Items

System Modules

Dynamic KG Builder & Reasoner (Nearline Interest Generation)

Constructs a graph from user history and reasons to find latent interests using DeepSeek-R1/QWQ-32B

Model or implementation: DeepSeek-R1 (Teacher) / QWQ-32B (Student/InterestGPT)

Nearline Cache (Nearline Interest Generation)

Stores generated interest keywords to decouple slow LLM inference from fast online retrieval

Model or implementation: Key-Value Store (implied)

Text Encoder (Online Retrieval)

Encodes the retrieved potential interest keyword into a dense vector

Model or implementation: Not specified (likely BERT-based or internal encoder)

Dual-Tower Retrieval Model (Online Retrieval)

Retrieves items by matching user representation (augmented with interest) to item representations

Model or implementation: Dual-Tower DNN

Novel Architectural Elements

Nearline 'Surprise Channel' architecture that caches LLM-derived interests to bypass online latency constraints.
Hybrid retrieval mechanism: A u2i (user-to-item) model that explicitly incorporates an 'interest-aligned contrastive learning loss' to mimic the relevance precision of i2i (item-to-item) retrieval while maintaining u2i conversion capabilities.

Modeling

Base Model: QWQ-32B (fine-tuned as InterestGPT), distilled from DeepSeek-R1

Training Method: Supervised Fine-Tuning (SFT) + Distillation

Objective Functions:

Purpose: Train the retrieval model to predict clicks.

Formally: Standard Binary Cross Entropy (BCE) Loss.
Purpose: Force the user tower's embedding to align with the semantic embedding of the generated potential interest (Contrastive Learning).

Formally: Interest-aligned contrastive learning loss (details not fully expanded in text, but described as auxiliary task).

Adaptation: Full fine-tuning (implied for SFT of QWQ-32B)

Trainable Parameters: Not reported in the paper

Training Data:

SFT dataset D constructed by distilling 'thought process and potential interests' from DeepSeek-R1 inference on user profiles.

Key Hyperparameters:

cache_window_days: 7

Compute: Nearline latency requirement: <100ms for online retrieval (achieved by caching LLM output). LLM inference done asynchronously.

Comparison to Prior Work

vs. SERAL: SERAL uses a pipeline approach; this paper introduces 'Near-line adaptation' with a specific dual-tower training loss (contrastive alignment) to solve the u2i/i2i trade-off.
vs. Standard u2i Retrieval: Standard u2i fails on new interests (missing behavior data); this method injects LLM-derived interest embeddings directly into the user tower training via contrastive loss.
vs. Standard i2i Retrieval: i2i ignores user history/profile; this method combines user profile (u2i) with the specific interest vector.
+ 1 more
vs. KAR (Knowledge-Augmented Reasoning) [not cited in paper]: KAR uses external KGs; this paper builds *dynamic* user-specific KGs on the fly using LLMs.

Limitations

Relies on the capabilities of the underlying LLM (DeepSeek-R1/QWQ-32B); if the LLM fails to reason, recommendations fail.
The approach adds system complexity (maintenance of nearline cache and separate LLM inference pipeline).
Evaluation is primarily on a single e-commerce platform (Dewu); generalization to other domains (e.g., movies, news) is not tested.
No statistical significance tests (p-values) reported for the online metrics.

Reproducibility

Code: https://github.com/DouYuoD/LLM-Two-hop-Reasoning-Recommendation

Prompt templates and SFT data samples are available at https://github.com/DouYuoD/LLM-Two-hop-Reasoning-Recommendation. Full production model weights and the proprietary Dewu dataset are not released.

📊 Experiments & Results

Evaluation Setup

Online A/B testing on the Dewu app and offline human evaluation of reasoning quality.

Benchmarks:

Dewu App Online A/B Test (Live Recommendation)
Offline Interest Scoring (Human Evaluation) [New]

Metrics:

Exposure Novelty Rate
Click Novelty Rate
Average View Duration
UV CTR (Unique Visitor Click-Through Rate)
UV Interaction Penetration
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Online A/B testing results on the Dewu platform showing improvements in novelty and engagement metrics.
Dewu App	Exposure Novelty Rate	Not reported in the paper	Not reported in the paper	+4.62%
Dewu App	Click Novelty Rate	Not reported in the paper	Not reported in the paper	+4.85%
Dewu App	Average View Duration per person	Not reported in the paper	Not reported in the paper	+0.15%
Dewu App	UV Interaction Penetration	Not reported in the paper	Not reported in the paper	+0.30%
Offline evaluation of the potential interests generated by the LLM.
Interest Quality	Score 2 Distribution (High Quality)	0	96	96

Main Takeaways

The method successfully breaks the filter bubble without hurting user engagement, as evidenced by simultaneous increases in Novelty Rates (+4.62%) and View Duration (+0.15%).
The 'Near-line' architecture effectively bridges the gap between powerful but slow LLMs and the low-latency requirements of industrial recommendation systems.
Multi-agent debate combined with two-hop reasoning produces highly rational and useful interest keywords (96% high-quality score).

📚 Prerequisite Knowledge

Prerequisites

Dual-Tower Retrieval Models (DSSM)
Knowledge Graphs (Nodes, Edges, Reasoning)
Large Language Models (Prompting, SFT)
Contrastive Learning

Key Terms

Serendipity: Recommendations that are both unexpected (surprising) and relevant (useful) to the user.

Filter Bubble: A state of intellectual isolation where a user is only exposed to content that aligns with their existing preferences.

Two-hop Reasoning: A reasoning process that connects a starting node to a target node via an intermediate node (e.g., Item -> Underlying Need -> New Item Category).

Near-line: A processing mode between real-time (online) and batch (offline), typically involving caching results that are updated periodically (e.g., every 7 days).

SFT: Supervised Fine-Tuning—training a pre-trained model on a specific smaller dataset to adapt it to a particular task.

u2i retrieval: User-to-Item retrieval; matching a user's embedding directly to item embeddings.

i2i retrieval: Item-to-Item retrieval; finding items similar to a trigger item the user interacted with.

BCE Loss: Binary Cross-Entropy Loss; a standard loss function for binary classification tasks (click vs. no-click).

InterestGPT: The specific LLM fine-tuned in this paper (based on QWQ-32B) to generate potential user interests.

Hypernym: A word with a broad meaning that more specific words fall under (e.g., 'Fruit' is a hypernym of 'Apple').