TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds

📝 Paper Summary

Recommendation system auditing Algorithm transparency

The authors propose a framework to audit social media feeds by labeling recommendations as either 'exploration' or 'exploitation' based on user history, revealing that TikTok exploits user interests in 30-50% of videos.

Core Problem

Social media recommendation algorithms, particularly TikTok's short-video feed, operate as black boxes, making it difficult to measure how much they personalize content versus exploring new topics.

Why it matters:

Excessive personalization can trap users in filter bubbles and echo chambers
Rapid-fire short videos may amplify adverse effects like radicalization or depression loops
New regulations like the EU Digital Services Act require algorithmic transparency and auditing, but tools to measure personalization in the wild are lacking

Concrete Example: A user watches several depression-related videos. Without a way to audit the feed, it is unclear if subsequent recommendations are 'exploit' (deliberately showing more depression content based on watch history) or 'explore' (random/popular content). This framework distinguishes the two.

Key Novelty

Exploration/Exploitation Labeling Framework for Black-Box Feeds

Models a user's feed as a timeline of item-user pairs and defines a set of 'features' (e.g., shared hashtags with previously liked videos) that act as activation conditions
If a recommendation satisfies a feature condition linking it to prior engagement (like, follow, watch time), it is labeled 'Exploit'; otherwise, it is 'Explore'
Introduces a 'personalization score' calculated by comparing a specific user's recommendation label against how that same video would be labeled in other users' timelines

Evaluation Highlights

TikTok's algorithm exploits user interests in 30% to 50% of recommended videos within the first 1,000 videos of a user's tenure
Following a creator is the primary driver of personalization, followed by liking videos
Personalization scores effectively distinguish real user feeds (highly personalized) from randomized baselines

Breakthrough Assessment

7/10

Provides a practical, applicable framework for the difficult task of auditing black-box recommenders in the wild using real user data, though the technical method is relatively straightforward rule-based matching.

⚙️ Technical Details

Problem Definition

Setting: Auditing a sequence of chronologically ordered content recommendations R = {r1, ..., rN} for a specific user u

Inputs: A user's recommendation timeline containing content attributes (hashtags, creator), user attributes (follows, interests), and engagement attributes (likes, watch time)

Outputs: Binary label for each recommendation r_i: 'Exploit' (result of personalization) or 'Explore' (not result of personalization)

Pipeline Flow

Data Ingestion (User Timeline) → Feature Extraction → Activation Check (Local/Global Features) → Label Assignment (Exploit/Explore) → Metric Calculation

System Modules

Feature Extractor (Analysis)

Extracts content (hashtags, creator), user (follows), and engagement (likes, watch time) attributes

Model or implementation: Rule-based extraction

Labeler (Analysis)

Applies activation conditions to determine if an item is an 'Exploit' recommendation

Model or implementation: Deterministic logic (if feature_condition == True then Exploit)

Scorer (Analysis)

Calculates personalization score by cross-referencing other user timelines

Model or implementation: Statistical aggregation

Novel Architectural Elements

Generalizable framework defining personalization as the satisfaction of 'activation conditions' based on historic user-item interactions (likes, follows, watch time) within variable temporal windows

Modeling

Base Model: None (Analytical Framework applied to data)

Compute: Not reported in the paper

Comparison to Prior Work

vs. Boeker/WSJ: Focuses on real user traces (N=347) rather than just bots, capturing authentic/diverse behavior
vs. Previous Audits: Explicitly models the 'Exploration vs. Exploitation' dichotomy rather than just correlating features to outcomes
vs. Self-report studies (Lee et al.): Uses objective data traces instead of relying on potentially biased user perceptions
+ 1 more
Novelty: Introduces quantitative metrics (User exploit fraction, Personalization score) to measure the *extent* of personalization over time

Limitations

Relies on a defined set of features (hashtags, creators) which may not capture complex semantic personalization (e.g., visual similarity)
Dataset relies on GDPR data donation which misses deleted videos or private accounts (only 4.1M of 4.9M videos had metadata)
Bot baseline behavior is simplistic and may not fully mimic complex human scrolling patterns
Analysis is post-hoc on collected traces; cannot casually intervene in the live algorithm

Reproducibility

Data availability: The analysis uses a dataset of real traces from 347 TikTok users (collected by Zannettou et al., 2023) and a custom dataset of 5 automated bots. Code availability is not provided in the paper text.

📊 Experiments & Results

Evaluation Setup

Analysis of TikTok recommendation traces from real users and automated bots

Benchmarks:

Real Users Dataset (Real-world trace analysis)
Simulated-Bot Dataset (Controlled environment analysis) [New]
Randomized Baseline (Statistical baseline) [New]

Metrics:

User Exploit Fraction (percentage of feed labeled as exploit)
Personalization Score (degree of tailoring to specific user)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Real Users Dataset	User Exploit Fraction	0	0.50	+0.50
Real Users Dataset	Personalization Driver	N/A	Following & Liking	N/A

Main Takeaways

The framework successfully labels 30-50% of a user's feed as 'exploitation' (personalized) within their first 1,000 videos.
Validates that explicit signals (Like, Follow) are stronger drivers of personalization than passive signals in this specific framework analysis.
Personalization Score metric aligns with intuition: real user feeds show higher personalization scores than randomized baselines.
Watch time (duration) alone is a factor, but explicit following provides a very strong 'global' personalization signal.

📚 Prerequisite Knowledge

Prerequisites

Basic understanding of recommendation systems (exploration vs. exploitation)
Familiarity with social media engagement metrics (likes, follows, watch time)

Key Terms

Exploitation recommendation: A recommendation personalized based on the user's inferred interests or previous actions (e.g., showing a video with a hashtag the user previously liked)

Exploration recommendation: A recommendation NOT resulting from user personalization, but rather the algorithm trying to test if a user likes a new or different topic

Personalization score: A metric estimating how personalized a specific item is for a user by comparing its label (exploit/explore) in that user's timeline vs. when hypothetically inserted into other users' timelines

User exploit fraction: The fraction of items in a specific temporal window W that are labeled as 'exploit' recommendations

Local features: Features modeling relationships between items within a specific short-term temporal window (e.g., matching a hashtag from the last W videos)

Global features: Features capturing personalization at a macroscopic scale (e.g., matching a creator the user followed at any point in the past)

DSA: Digital Services Act—EU legislation emphasizing algorithmic transparency and calling for audits of algorithmic feeds

Activation condition: A rule based on features (like matching hashtags or creators) that, if met, triggers a recommendation to be labeled as 'Exploit'