Simulating Filter Bubble on Short-video Recommender System with Large Language Model Agents

📝 Paper Summary

Agent-based simulation Recommender System Simulation

SimTok employs LLM-based user agents to simulate the feedback loop in short-video recommender systems, enabling the analysis of filter bubble formation and the testing of mitigation strategies without live user testing.

Core Problem

The formation of filter bubbles on short-video platforms involves complex dynamics between algorithms and user feedback that are difficult to study due to the risks and costs of online testing.

Why it matters:

Filter bubbles can trap users in echo chambers, potentially leading to political polarization and threatening democratic information flow
Short-video platforms (TikTok, Kuaishou) lack the 'friend loops' of traditional social media, relying entirely on algorithmic recommendation, making them more susceptible to rapid filter bubble formation
Existing research often lacks a realistic simulation of the user-recommender feedback loop, relying instead on static datasets or simplistic user models

Concrete Example: On a platform like TikTok, a user interacting with a few sports videos might be locked into a feed consisting exclusively of sports content. Studying this requires simulating the user's sequential decisions (skip, like, comment) and the system's subsequent updates over many iterations.

Key Novelty

LLM-driven User-Recommender Feedback Loop Simulation

Replaces static user datasets with LLM agents that possess distinct personalities (OCEAN) and motivations (uses and gratifications) to generate dynamic feedback
Converts video items into text-based prompts (titles, tags, popularity) allowing the LLM agent to 'watch' and reason about content without processing raw video
Integrates a specific feedback weighting mechanism where different agent actions (Like vs. Comment vs. Skip) exert varying influence on the recommender's loss function

Architecture

The overall SimTok simulation framework showing the closed loop between the LLM Module and Recommender Module

Breakthrough Assessment

7/10

Offers a novel framework for simulating the specific dynamics of short-video platforms using LLM agents. While agent-based simulation exists, applying it specifically to the filter bubble mechanism with feedback weighting is a valuable methodological contribution.

⚙️ Technical Details

Problem Definition

Setting: Closed-loop simulation of a recommender system (RS) and user agents, where agents provide feedback on items, updating the RS model iteratively

Inputs: Video metadata (tags, categories, popularity) and User Profiles (demographics, personality traits)

Outputs: User feedback actions (Watch, Like, Comment, Collect, Skip/Dislike) and updated recommender states

Pipeline Flow

LLM Module (Profile Generation)
Recommender Module (Item Selection)
User-Agent Interaction (Feedback Generation)
Recommender Training (Model Update)

System Modules

LLM Module

Generates personalized agent profiles and simulates user decision-making

Model or implementation: Large Language Model (Specific model name not reported in the provided text)

Recommender Module

Selects video items to present to the user agent based on learned preferences

Model or implementation: Matrix Factorization (MF) or Factorization Machines (FM)

Novel Architectural Elements

Integration of feedback weights (w_i) directly into the Binary Cross-Entropy loss of the recommender based on LLM-generated discrete actions
Closed-loop architecture where the LLM agent's output immediately becomes training data for the next iteration of the Recommender Module

Modeling

Base Model: Large Language Model (Specific model name not reported in the provided text)

Training Method: Iterative Recommender System Training (Supervised Learning on generated feedback)

Objective Functions:

Purpose: Optimize recommender to predict user preference while accounting for the intensity of different feedback types.

Formally: L = - (1/N) * sum(w_i * [y_i * log(y_hat_i) + (1-y_i) * log(1-y_hat_i)]) where w_i is the feedback weight.

Compute: Not reported in the provided text

Comparison to Prior Work

vs. Traditional simulation: SimTok uses LLMs to generate nuanced, psychologically-grounded (OCEAN) feedback rather than probabilistic clicks
vs. Static Datasets: SimTok allows the system to evolve dynamically, capturing the *formation* process of filter bubbles rather than just observing them ex-post

Limitations

Relies on text summaries of videos rather than visual content, which may miss visual cues important for short-video engagement
Simulation validity depends heavily on the alignment between the LLM's reasoning and real human psychology
Computational cost of using LLMs for every user interaction limits the scale of simulation compared to lightweight stochastic agents

Reproducibility

The paper uses an open-source real-world dataset from a Chinese short-video platform (4000+ videos). Specific code URLs, LLM model versions, and hyperparameters are not provided in the text snippet.

📊 Experiments & Results

Evaluation Setup

Simulation of user-agent interactions with a short-video platform recommender over multiple iterations

Benchmarks:

Real-world short-video dataset (Recommendation Simulation)

Metrics:

Overall Coverage (ratio of unique categories watched to total categories)
Overall Entropy (Shannon entropy of watched video categories)
Satisfaction (ratio of positive responses to total videos watched)
Statistical methodology: Not explicitly reported in the provided text

Main Takeaways

The framework successfully integrates LLM agents with MF/FM recommenders to simulate the recommendation-feedback cycle
Filter bubbles are defined and detected using a coverage-based threshold (user coverage < median coverage)
The simulation enables the testing of interventions such as feedback weighting strategies to mitigate content homogenization
LLM-inherent biases are analyzed for their propagation through recommendations (bias analysis methodology mentioned, specific results not in text)

📚 Prerequisite Knowledge

Prerequisites

Collaborative Filtering (Matrix Factorization)
Large Language Models (LLMs) as agents
Recommender Systems evaluation metrics (Entropy, Coverage)

Key Terms

Filter Bubble: A state where a user is exposed to a narrow scope of content categories, isolated from diverse information

OCEAN: The Big Five personality traits model: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism; used here to parameterize user agents

Uses and Gratifications: A theory describing user motivations for media usage (e.g., Social Interaction, Escapism), used here to drive agent behavior

Matrix Factorization (MF): A collaborative filtering algorithm that decomposes the user-item interaction matrix into lower-dimensional latent vectors

Factorization Machines (FM): A generalized model for feature-based recommendation that captures interactions between variables (like user demographics and item categories)

Feedback Weighting: Assigning scalar values to different user actions (e.g., Comment > Like > Watch) in the loss function to reflect their varying significance

BCE Loss: Binary Cross-Entropy loss, a standard loss function for classification tasks, modified here to include feedback weights