Leverage Knowledge Graph and Large Language Model for Law Article Recommendation: A Case Study of Chinese Criminal Law

📝 Paper Summary

Legal AI Knowledge Graph Construction Retrieval-Augmented Generation (RAG)

A framework combining a Case-Enhanced Law Article Knowledge Graph (CLAKG) with LLMs to improve Chinese criminal law recommendation accuracy and mitigate hallucinations through grounded retrieval.

Core Problem

Grassroots courts face massive backlogs, and existing tools either lack semantic understanding (text classification) or suffer from hallucinations (LLMs), making them unreliable for high-stakes legal decisions.

Why it matters:

Judicial efficiency is critical for social stability, but current reliance on manual cognitive effort slows down decision-making
Direct use of LLMs in law is dangerous due to fabrication of legal citations (hallucinations)
Traditional classifiers (BERT, CNN) focus on fact-to-ID mapping, neglecting the semantic rationale and interpretability required in law

Concrete Example: Direct use of LLMs can produce plausible-sounding but non-existent law articles (hallucinations). The paper's method grounds the LLM in a verified knowledge graph (CLAKG) to ensure recommendations are based on actual statutes and historical precedents.

Key Novelty

Case-Enhanced Law Article Knowledge Graph (CLAKG)

Unifies abstract law articles and concrete historical cases into a single graph schema, enabling retrieval based on both statutory rules and similar past judgments
Uses a closed-loop human-machine collaboration where expert feedback on recommendations explicitly updates the Knowledge Graph to refine future performance

Architecture

The closed-loop law article recommendation pipeline

Evaluation Highlights

Boosts law article recommendation accuracy from 0.549 (LLM baseline) to 0.694 (Proposed LLM + CLAKG)
Outperforms strong baselines including BERT, DPCNN, Graph-RAG, and Light-RAG (specific baseline numbers not in snippet, but improvement is cited as significant)

Breakthrough Assessment

7/10

Significant accuracy jump (+14.5%) in a high-stakes domain by effectively integrating symbolic knowledge (KG) with probabilistic models (LLM). The closed-loop expert feedback is a practical addition for legal settings.

⚙️ Technical Details

Problem Definition

Setting: Predicting relevant law articles applicable to a case based on its factual description

Inputs: Factual description of a legal case

Outputs: List of applicable law article IDs and justifications

Pipeline Flow

Group 1: Construction: LLM extracts nodes/edges → Expert Review → CLAKG Construction
Group 2: Inference: User Case Input → Keyword Extraction → Candidate Retrieval (via RGCN) → LLM Recommendation

System Modules

CLAKG Constructor

Extracts entities (cases, articles) and relations from raw text to build the graph

Model or implementation: Large Language Model (specific architecture not named)

Graph Embedder (Inference)

Learns vector representations of nodes to enable similarity matching

Model or implementation: Relational Graph Convolutional Network (RGCN)

Candidate Retriever (Inference)

Selects potentially applicable law articles for the new case

Model or implementation: Keyword matching + Graph Embedding Similarity

Recommendation Generator (Inference)

Generates the final law article recommendation and reasoning

Model or implementation: Large Language Model (specific architecture not named)

Novel Architectural Elements

Closed-loop feedback mechanism where expert corrections on inference outputs are used to update the CLAKG structure
Hybrid retrieval combining keyword extraction with RGCN-based graph embedding similarity

Modeling

Base Model: Large Language Model (specific variant e.g., GPT-4/Llama not explicitly named in text snippet)

Training Method: Graph Embedding Training (RGCN)

Objective Functions:

Purpose: Distinguish true triples in the graph from fabricated ones.

Formally: Binary Cross-Entropy Loss L = - sum ( y log(p) + (1-y) log(1-p) ) over training triples T.
Purpose: Score the plausibility of triples for link prediction.

Formally: DistMult factorization f(s,r,o) = e_s^T diag(e_r) e_o.

Key Hyperparameters:

K (keywords matched): 8
q (candidate articles): 5
t (output articles): 1

Compute: Not reported in the paper

Comparison to Prior Work

vs. Text-CNN/DPCNN: Proposed method leverages semantic content of laws and historical cases via KG, rather than just mapping text to labels
vs. Standard RAG (TFIDF/BERT): Uses structured graph embeddings (RGCN) rather than just lexical or dense vector retrieval
vs. Graph-RAG: Incorporates a specific task-oriented schema (CLAKG) unifying laws and cases, plus a human-in-the-loop update mechanism

Limitations

Specific LLM architecture and size not detailed in the text snippet
Evaluation is limited to Chinese criminal law; generalization to other legal systems not tested
Reliance on legal experts for the feedback loop may be a bottleneck for scalability

Reproducibility

Source code and processed datasets are stated to be 'publicly available on GitHub (see Data Availability Statement)', but the URL is not contained in the provided text snippet. The paper uses China Judgments Online for data.

📊 Experiments & Results

Evaluation Setup

Law article recommendation on a constructed dataset of criminal judgments

Benchmarks:

Custom Chinese Criminal Law Dataset (Law article recommendation (predicting article ID from case facts)) [New]

Metrics:

Accuracy
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Chinese Criminal Law Dataset	Accuracy	0.549	0.694	+0.145

Experiment Figures

The graph embedding training process using RGCN

Main Takeaways

Integrating a Knowledge Graph (CLAKG) with LLMs significantly outperforms using LLMs alone for law article recommendation (+14.5% accuracy)
The method outperforms strong baselines including BERT, DPCNN, and other RAG variants (TFIDF-RAG, Graph-RAG, Light-RAG), though specific numeric scores for these baselines are not in the snippet
The closed-loop system allows the knowledge base to evolve via expert feedback, addressing the static nature of traditional training sets

📚 Prerequisite Knowledge

Prerequisites

Knowledge Graphs (entities, relations, schemas)
Retrieval-Augmented Generation (RAG)
Graph Neural Networks (specifically RGCN)

Key Terms

CLAKG: Case-Enhanced Law Article Knowledge Graph—a structured database linking law articles, key legal information, and historical adjudicated cases

RGCN: Relational Graph Convolutional Network—a type of neural network designed to learn embeddings for nodes in graphs with multiple types of relationships

RAG: Retrieval-Augmented Generation—technique where an LLM is provided with retrieved external data to ground its answers

LAKG: Law Article Knowledge Graph—subgraph containing law articles and their key extracted information

ACKG: Adjudicated Cases Knowledge Graph—subgraph containing case names, reasons, details, and court session times

Link Prediction: A task where the model predicts the likelihood of a connection between two nodes in a graph, used here to train graph embeddings