Enabling Explainable Recommendation in E-commerce with LLM-powered Product Knowledge Graph

📝 Paper Summary

E-commerce Recommendation Knowledge Graph Construction

LLM-PKG distills world knowledge from Large Language Models into a static product knowledge graph to provide explainable, real-time e-commerce recommendations without the latency of live LLM inference.

Core Problem

Deploying LLMs directly for real-time e-commerce recommendation is infeasible due to high latency, while traditional methods struggle to capture implicit user intent (like 'carnations for Mother's Day') from behavioral data alone.

Why it matters:

Modern e-commerce systems require extremely low latency (milliseconds), making real-time LLM calls unacceptable for high-traffic scenarios.
Traditional collaborative filtering misses the 'why' behind purchases, failing to leverage common sense or world knowledge (e.g., gift-giving occasions) that LLMs possess.
Building product knowledge graphs manually is expensive and labor-intensive, while mining them from user logs often fails to capture explicit intent.

Concrete Example: A user buys carnations. Traditional systems might recommend other flowers based on co-occurrence but miss the intent. An LLM knows carnations are for Mother's Day, but calling it live is too slow. LLM-PKG pre-computes this 'carnation -> Mother's Day' link offline.

Key Novelty

Offline Distillation & Semantic Mapping

Use LLMs offline to hallucinate a 'fantasy' knowledge graph of products and rationales, then rigorously prune it using the LLM itself as a critic.
Map the nodes of this synthetic graph to real enterprise inventory using vector search (BERT embeddings), effectively grounding LLM world knowledge into actual sellable items for low-latency lookup.

Architecture

The end-to-end framework of LLM-PKG, split into Offline Construction and Online Serving phases.

Evaluation Highlights

Significant increase in Item-Detail-Page (View Item) views in online A/B testing on a major e-commerce platform (eBay).
Significant increase in qualified transactions/purchases in the same online A/B test.
Successfully deployed in a high-traffic production environment by caching the graph as key-value pairs, bypassing the latency of graph database queries.

Breakthrough Assessment

7/10

A practical, engineering-focused solution to the 'LLM latency vs. utility' trade-off in production systems. While the methodology (distillation) is not theoretically novel, the specific pipeline for grounding hallucinated graphs to enterprise inventory is highly effective for industry application.

⚙️ Technical Details

Problem Definition

Setting: Constructing a Product Knowledge Graph (PKG) using LLMs and mapping it to enterprise inventory for item-based recommendation

Inputs: Seed products (titles) from enterprise data

Outputs: A refined Knowledge Graph where nodes are real enterprise products and edges are explainable relations (e.g., 'similar style', 'affordable alternative')

Pipeline Flow

Offline Construction: LLM Generation → Validation/Pruning → Internal Mapping
Online Serving: Key-Value Lookup → Candidate Generation

System Modules

Generator (Offline Construction)

Generate initial graph nodes (recommendations) and edges (rationales) from seed products

Model or implementation: Not explicitly named (generic LLM)

Validator (Offline Construction)

Score recommendations and verify rationales to prune low-quality edges

Model or implementation: Not explicitly named (generic LLM)

Mapper (Offline Construction)

Ground LLM-generated product titles to actual enterprise inventory

Model or implementation: Fine-tuned BERT + KNN Index

Serving Layer

Retrieve recommendations in real-time

Model or implementation: Key-Value Store (Cache)

Novel Architectural Elements

The 'Hallucinate-then-Ground' pipeline: intentionally letting the LLM generate potentially non-existent products based on world knowledge, then using vector search to snap them to the nearest real inventory item.

Modeling

Base Model: BERT (for embedding mapping); unspecified LLM for generation

Training Method: Fine-tuning (BERT only)

Adaptation: Fine-tuning of BERT model for product title embeddings

Trainable Parameters: BERT weights

Training Data:

Enterprise product titles for BERT fine-tuning

Compute: Not reported in the paper

Comparison to Prior Work

vs. Wang et al. (2023): LLM-PKG focuses on offline graph construction for low latency rather than real-time generation.
vs. Abu-Rasheed et al. (2024): LLM-PKG works in reverse—using the LLM to build the KG, rather than using the KG to aid the LLM.
vs. Direct Retrieval [not cited in paper]: LLM-PKG adds an explanatory layer (rationales) and leverages world knowledge absent in pure behavioral retrieval.

Limitations

Relies on the assumption that LLM world knowledge aligns with specific enterprise inventory.
Vector mapping (KNN) might map a hallucinated high-end item to a low-quality real item if semantic matching is imperfect.
Static graph construction means the system cannot react to new trends instantly without rebuilding/updating the graph.
User-centric recommendation capabilities mentioned but left for future work; experiments focus on item-centric.

Reproducibility

Prompt templates are provided in tables. Exact LLM version (e.g., GPT-4, Llama 2) is not specified. BERT fine-tuning details are minimal. Code is not provided.

📊 Experiments & Results

Evaluation Setup

A/B test on a live e-commerce website (eBay) using sneaker products.

Benchmarks:

Production Baseline (Item-to-Item Recommendation)

Metrics:

View Item (VI) count
Qualified Transactions
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Online A/B testing results demonstrate significant user engagement improvements.
Live Traffic	View Item (VI)	Not reported in the paper	Not reported in the paper	Significant increase
Live Traffic	Transactions	Not reported in the paper	Not reported in the paper	Significant increase

Experiment Figures

Comparison of LLM output formats with and without one-shot learning.

Main Takeaways

Offline distillation of LLM knowledge into a KG effectively bypasses real-time latency constraints while retaining explainability.
The 'Pruning' step (removing nodes with score < 6) and 'Mapping' step are critical for ensuring the graph contains high-quality, sellable items.
One-shot prompting significantly improved the correctness of the JSON output format compared to zero-shot.
Validation metrics (Average Edge Score, Relation Imprecise Rate) were improved iteratively by refining prompts.

📚 Prerequisite Knowledge

Prerequisites

Knowledge Graphs (nodes, edges, triples)
Large Language Models (prompting, hallucination)
Vector Search (embeddings, KNN)
E-commerce Recommendation Systems (candidate generation, ranking)

Key Terms

PKG: Product Knowledge Graph—a structured representation of products and their relationships

RDF: Resource Description Framework—a standard model for data interchange on the Web, using subject-predicate-object triples

KNN: K-Nearest Neighbors—an algorithm used here to find the real product most similar to the LLM's hallucinated product description

One-shot learning: Providing the model with a single example of the desired input-output pair within the prompt to guide its generation

Hallucination: When an LLM generates plausible-sounding but factually incorrect or non-existent information (e.g., inventing a product that doesn't exist)

BERT: Bidirectional Encoder Representations from Transformers—a language model used here to create vector embeddings of product titles for semantic search