Transparent and Scrutable Recommendations Using Natural Language User Profiles

📝 Paper Summary

Explainable Recommendation User Modeling LLM-based Recommendation

UPR generates natural language user profiles from past reviews to enable transparent, editable recommendations in warm-start settings without relying on uninterpretable embeddings.

Core Problem

Traditional recommender systems rely on uninterpretable latent embeddings, preventing users from understanding why items are suggested or effectively modifying their preferences.

Why it matters:

Users cannot easily scrutinize or correct the internal representation of their preferences when they receive poor recommendations
Modifying embeddings requires significant changes to interaction history, which is inefficient and indirect
Lack of transparency reduces user trust and decision-making confidence in the system

Concrete Example: In collaborative filtering, a user represented by a latent vector cannot simply tell the system 'I dislike romance movies now' to update their recommendations; they would have to artificially interact with many non-romance items to shift the vector.

Key Novelty

User Profile Recommendation (UPR)

Simulates natural language profiles by extracting feature sentiments from past reviews and prompting an LLM to summarize them into a coherent description
Replaces user ID embeddings with these text profiles during the fine-tuning of an LLM for rating prediction, making the input fully transparent
Enables users to directly edit the natural language text of their profile to immediately alter recommendation outcomes

Architecture

Overview of the User Profile Recommendation (UPR) framework, contrasting traditional embedding-based methods with the proposed language-based approach

Evaluation Highlights

Achieves RMSE of 0.9856 on Amazon Movies & TV, outperforming collaborative filtering baselines like NeuMF (1.0264) and item-level explainable models like PETER+ (1.0069)
Maintains competitive performance with standard matrix factorization methods while offering full transparency and scrutability
Demonstrates that editing the NL profile (e.g., adding a specific genre preference) successfully steers recommendations toward the desired category

Breakthrough Assessment

7/10

Offers a significant step toward truly scrutable recommendation by replacing embeddings with text. While performance is comparable rather than superior to baselines, the added utility of direct profile editing is a strong contribution.

⚙️ Technical Details

Problem Definition

Setting: Rating prediction in a warm-start setting using natural language inputs

Inputs: Natural language user profile P_u and candidate item information

Outputs: Predicted rating score r_ui

Pipeline Flow

Feature Extraction (Extract features/sentiment from reviews)
Profile Generation (LLM summarizes features into NL profile)
Recommendation (LLM predicts rating based on NL profile)

System Modules

Feature Extractor

Identify significant features and sentiment from historical user reviews

Model or implementation: Phrase-level sentiment analysis toolkit

Profile Generator

Synthesize a coherent natural language user profile from extracted features and sample reviews

Model or implementation: Instruction-tuned LLM (gpt-3.5-turbo)

Recommender

Predict item ratings based on the generated profile

Model or implementation: Fine-tuned LLM (Flan-T5-Base)

Novel Architectural Elements

Substitution of latent user embeddings with LLM-generated Natural Language Profiles as the sole user representation during fine-tuning

Modeling

Base Model: Flan-T5-Base (for recommendation task)

Training Method: Fine-tuning for regression

Objective Functions:

Purpose: Minimize the error between predicted and actual ratings.

Formally: Mean Squared Error (MSE) loss during fine-tuning.

Training Data:

Amazon Movies and TV dataset
TripAdvisor dataset
Split: 8:1:1 train/validation/test
Warm-start constraint: at least 5 reviews per user in training set

Key Hyperparameters:

learning_rate: 1e-3
batch_size: 8

Compute: Not reported in the paper

Comparison to Prior Work

vs. DeepCoNN: UPR uses LLM-summarized profiles as interpretable input rather than learning latent vectors from raw review text [not cited in paper]
vs. Sanner et al. (2023): Validates language-based profiles in a warm-start setting (vs. near cold-start) and automates profile creation via LLMs (vs. crowdsourced profiles)
vs. PETER+/PEPLER: Focuses on explaining the user input (holistic preferences) rather than explaining the item output (individual recommendation justification)

Limitations

Relies on the availability of user reviews to generate profiles; cannot handle pure cold-start users without text history
Profile generation cost scales with the number of users if using commercial LLM APIs
The approach is tested on rating prediction (regression), not top-k ranking optimization
No statistical significance tests reported for the performance comparisons

Reproducibility

Code: https://github.com/jeromeramos70/user-profile-recommendation

Publicly available code (GitHub). Datasets are standard public benchmarks (Amazon, TripAdvisor). Specific prompts used for profile generation and recommendation are provided in the Appendix.

📊 Experiments & Results

Evaluation Setup

Rating prediction on standard explainable recommendation datasets

Benchmarks:

Amazon Movies and TV (Rating Prediction)
TripAdvisor (Rating Prediction)

Metrics:

RMSE (Root Mean Squared Error)
MAE (Mean Absolute Error)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
UPR achieves competitive error rates compared to strong baselines on the Amazon Movies dataset, outperforming neural and explainable baselines.
Amazon Movies and TV	RMSE	1.0264	0.9856	-0.0408
Amazon Movies and TV	RMSE	1.0069	0.9856	-0.0213
Results on TripAdvisor show UPR remains competitive, though slightly behind the strongest baselines compared to the Amazon dataset.
TripAdvisor	RMSE	0.8255	0.8329	+0.0074

Experiment Figures

Impact of editing user profiles on recommendation rankings (Change in Rank vs. Item Genre)

Main Takeaways

Natural language profiles can replace latent embeddings in warm-start settings without significant loss in recommendation accuracy
Qualitative analysis confirms LLM-generated profiles are fluent, informative, and accurately reflect user preferences from reviews
The system enables 'scrutability': editing the text profile (e.g., adding 'I like action') directly changes the model's output to align with the new preference

📚 Prerequisite Knowledge

Prerequisites

Collaborative Filtering (CF)
Matrix Factorization
Large Language Models (LLMs) for recommendation
Prompt Engineering

Key Terms

warm-start: A recommendation setting where the system has access to significant historical interaction data for users and items

scrutability: The ability for a user to inspect and modify their internal representation within a system to correct errors or update preferences

set-based preferences: Preferences expressed over groups or categories of items (e.g., 'I like action movies') rather than individual item ratings

NeuMF: Neural Matrix Factorization—a model combining matrix factorization and multi-layer perceptrons for recommendation

PETER+: An item-level explainable recommendation model based on Transformers that predicts ratings and generates explanations

RMSE: Root Mean Squared Error—a standard metric for measuring the differences between predicted and observed values

MAE: Mean Absolute Error—a metric measuring the average magnitude of errors in a set of predictions

nDCG: Normalized Discounted Cumulative Gain—a measure of ranking quality that accounts for the position of relevant items

MAP: Mean Average Precision—a metric for evaluating ranked lists