Enhancing News Recommendation with Hierarchical LLM Prompting

📝 Paper Summary

News Recommendation LLM-based Data Augmentation Content-based Recommendation

PNR-LLM improves news recommendation by using Large Language Models to rewrite news titles and extract latent entities through a hierarchical prompting strategy, enriching sparse article representations.

Core Problem

News articles often contain sparse, ambiguous, or short titles that fail to capture deep semantic meaning, limiting the effectiveness of content-based recommendation systems.

Why it matters:

Traditional content-based methods rely on shallow text representations (titles/abstracts) that miss structured relationships between articles.
Graph-based methods depend on external knowledge graphs which often suffer from incompleteness and sparsity.
Existing LLM approaches focus on summarization, which can lose detail, rather than extrapolation to enrich content.

Concrete Example: A news title might be short and clickbaity, lacking key context. Standard encoders miss the topic. PNR-LLM uses an LLM to 'extrapolate' entities (e.g., inferring 'US Politics' from a vague headline) and rewrite the title to be more descriptive before feeding it to the recommender.

Key Novelty

Hierarchical Prompting for News Enrichment (PNR-LLM)

Uses a multi-step prompting strategy: first generates a direct title, then extracts relevant entities using the LLM's internal knowledge, and finally combines them to generate a semantically enriched title.
Treats the LLM as a knowledge base to 'extrapolate' hidden entities and context not present in the original short text, rather than just summarizing existing text.

Architecture

The PNR-LLM framework, illustrating the Hierarchical Prompting flow (Direct Prompting -> Exploration -> Hierarchical Prompting) feeding into the News Encoder and User Encoder.

Evaluation Highlights

Outperforms state-of-the-art baselines (GLoCIM, DIGAT) on MIND-SMALL and MIND-LARGE datasets across AUC, MRR, and nDCG metrics.
Achieves +0.8% AUC improvement on MIND-SMALL compared to the strongest baseline (GLoCIM).
Replacing original titles with hierarchically enriched titles consistently boosts performance of standard baselines like NAML and NRMS.

Breakthrough Assessment

7/10

Solid application of LLMs for data enrichment in recommender systems. The hierarchical prompting strategy is effective and model-agnostic, though the core architecture remains a standard news encoder.

⚙️ Technical Details

Problem Definition

Setting: Personalized News Recommendation: predicting the probability of a user clicking a candidate news article based on their click history.

Inputs: User click history sequence H_u and a candidate news article d_c (each containing title text and entities).

Outputs: Click probability score y_hat.

Pipeline Flow

News Enrichment (LLM-based Title Rewriting & Entity Extraction)
News Encoder (CNN + Attention over Text & Entities)
User Encoder (Attention over Browsed News)
Click Predictor (Dot product)

System Modules

News Enrichment Module

Generate enriched titles and extract entities from original articles using LLM.

Model or implementation: Gemini 1.0 Pro

News Encoder (Representation Learning)

Encode news articles into vector representations using enriched data.

Model or implementation: CNN + Attention (or DistilBERT in ablation)

User Encoder (Representation Learning)

Aggregate browsed news vectors into a user representation.

Model or implementation: Additive Attention Network

Novel Architectural Elements

Hierarchical Prompting Module: A specific 3-step pipeline (Direct -> Explore -> Refine) to generate inputs for the recommender, specifically designed to balance informativeness and conciseness.

Modeling

Base Model: Standard News Encoder architecture (CNN/Attention-based similar to NAML) augmented with LLM-generated data.

Training Method: Supervised learning with negative sampling

Objective Functions:

Purpose: Maximize likelihood of user clicking positive samples vs negative samples.

Formally: Cross-entropy loss with negative sampling.

Training Data:

MIND-LARGE (1M users) and MIND-SMALL (50k users) datasets.
Negative sampling ratio: 1:4.

Key Hyperparameters:

learning_rate: 1e-4
optimizer: Adam
negative_samples: 4

Compute: Not reported in the paper

Comparison to Prior Work

vs. Prompt4NR: PNR-LLM uses LLMs for *data enrichment* (preprocessing) rather than as the direct recommender/ranker.
vs. ONCE: PNR-LLM focuses on *extrapolation* (adding missing entities/context) rather than just summarization.
vs. Graph-based methods (DIGAT): PNR-LLM extracts entities via LLM knowledge rather than relying solely on static external Knowledge Graphs.

Limitations

Generated titles are longer than original ones, requiring tuning of token length (optimal found at 40).
Relying on LLMs (Gemini) for enrichment introduces latency and cost during the preprocessing phase.
Effectiveness depends on the LLM's internal knowledge of the specific news entities.

Reproducibility

Code: https://github.com/vinuni-research/PNR-LLM

Code is publicly available. Uses Gemini 1.0 Pro for data enrichment, which is a closed-source API. Entity verification uses Wikidata API.

📊 Experiments & Results

Evaluation Setup

News Recommendation on MIND dataset.

Benchmarks:

MIND-LARGE (News Recommendation)
MIND-SMALL (News Recommendation)

Metrics:

AUC
MRR
nDCG@5
nDCG@10
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Performance comparison on MIND-SMALL showing PNR-LLM outperforms all baselines.
MIND-SMALL	AUC	0.6865	0.6946	+0.0081
MIND-SMALL	nDCG@5	0.3524	0.3661	+0.0137
Performance comparison on MIND-LARGE showing PNR-LLM is competitive with state-of-the-art.
MIND-LARGE	AUC	0.6973	0.6966	-0.0007
Ablation showing improvement of baseline models when using PNR-LLM enriched titles.
MIND-SMALL	AUC	0.6653	0.6724	+0.0071
MIND-SMALL	AUC	0.6776	0.6817	+0.0041

Experiment Figures

Comparison of different prompting strategies (Topic, Abstract, Direct, Entity-based, Hierarchical) on AUC/nDCG.

Comparison of different LLMs (Gemini, LLaMA 3, GPT-4o mini) for the enrichment task.

Main Takeaways

Hierarchical prompting consistently outperforms direct prompting and entity-only prompting, suggesting the multi-step refinement is crucial.
Gemini 1.0 Pro outperformed LLaMA 3 and GPT-4o mini in the enrichment task.
Using a PLM (DistilBERT) as the text encoder with enriched titles yields better results than CNN-based encoders, benefiting from the semantically richer input.
Optimal token length for generated titles is around 40 tokens; too short loses info, too long adds noise.

📚 Prerequisite Knowledge

Prerequisites

Basics of News Recommendation (User/News Encoders)
Attention Mechanisms (Self-Attention, Additive Attention)
Large Language Models (Prompting)

Key Terms

Hierarchical Prompting: A multi-step prompting strategy that refines LLM outputs by first extracting entities and then using them to generate a richer text representation.

MIND: Microsoft News Dataset—a large-scale benchmark dataset for news recommendation containing user click logs.

PNR-LLM: Personalized News Recommendation via Large Language Models—the proposed framework.

nDCG: Normalized Discounted Cumulative Gain—a measure of ranking quality that accounts for position of relevant items.

AUC: Area Under the ROC Curve—a metric measuring the ability of the model to distinguish between positive (clicked) and negative samples.

MRR: Mean Reciprocal Rank—a statistical measure for evaluating any process that produces a list of possible responses to a sample of queries.

TransE: Translation-based Embedding—a method to embed entities and relationships from a knowledge graph into vector space.

GloVe: Global Vectors for Word Representation—an unsupervised learning algorithm for obtaining vector representations for words.

PLM: Pre-trained Language Model (e.g., BERT, DistilBERT).

LLM: Large Language Model (e.g., GPT-4, Llama, Gemini).