Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning

📝 Paper Summary

Knowledge internalization Hallucination suppression

Prereq-Tune reduces hallucinations by separating fine-tuning into two stages: first learning prerequisite knowledge into a frozen adapter, then training a separate skill adapter that grounds its outputs on that knowledge.

Core Problem

Fine-tuning LLMs on data containing knowledge not seen during pre-training encourages the model to fabricate answers, creating a 'knowledge inconsistency' that leads to hallucination.

Why it matters:

Standard fine-tuning entangles skill learning (e.g., how to write a biography) with knowledge acquisition (facts about the subject), confusing the model when it encounters unfamiliar entities
Models are incentivized to produce plausible-looking but wrong answers when forced to generate facts they don't know during training
Existing methods struggle to use synthetic data effectively because fictitious facts in synthetic data can pollute the model's factual knowledge base

Concrete Example: If a model is fine-tuned to answer 'When was John Estes born?' with '1987', but never saw John Estes during pre-training, it learns to output a random year for any unknown person. Later, when asked about a real person it doesn't know, it hallucinates a birth year instead of abstaining.

Key Novelty

Two-Stage Disentangled Tuning (Prereq-Tune)

First, train a 'Knowledge LoRA' on raw facts (prerequisite knowledge) needed for the task, then freeze it
Second, train a 'Skill LoRA' on the actual downstream task (e.g., Q&A) while the Knowledge LoRA is active, forcing the skill module to rely on the provided knowledge
During inference, the Knowledge LoRA is removed, and the Skill LoRA successfully generalizes to grounding answers in the model's original pre-trained knowledge

Architecture

The two-stage training process of Prereq-Tune.

Evaluation Highlights

Outperforms standard SFT by significantly reducing hallucination rate on biography generation (FactScore improvement not explicitly summarized as a single average but consistent across metrics)
Successfully utilizes fictitious synthetic data (biographies of non-existent people) to improve factuality on real-world queries
Demonstrates capability to switch answers based on which 'Knowledge LoRA' is plugged in, proving the skill module learns to ground generation rather than memorize facts

Breakthrough Assessment

8/10

Novel conceptual disentanglement of skill and knowledge using modular adapters. It turns the liability of fictitious synthetic data into an asset for training factual grounding.

⚙️ Technical Details

Problem Definition

Setting: Supervised Fine-Tuning (SFT) of a pre-trained LLM on a downstream task containing specific knowledge constraints.

Inputs: Task dataset D_T (e.g., instruction-response pairs) and optionally a derived Prerequisite Knowledge dataset D_know.

Outputs: A fine-tuned model (specifically the Skill LoRA parameters) capable of performing the task while remaining factual.

Pipeline Flow

Step 1: Prerequisite Learning (Train Knowledge LoRA)
Step 2: Supervised Fine-Tuning (Train Skill LoRA with Frozen Knowledge LoRA)
Inference: Drop Knowledge LoRA, use only Skill LoRA

System Modules

Knowledge LoRA

Absorb the factual information required for the task so the skill learner doesn't have to.

Model or implementation: LoRA adapter on Llama-2-7B

Skill LoRA

Learn the downstream task format (e.g., biography generation) and how to retrieve/ground answers from the active knowledge source.

Model or implementation: LoRA adapter on Llama-2-7B (distinct from Knowledge LoRA)

Novel Architectural Elements

Sequential training of two distinct LoRA modules (Knowledge then Skill) where the second is trained conditional on the first
Inference-time removal of the 'Knowledge' module to force generalization to internal pre-trained weights
Multi-version training: using multiple consistent sets of fictitious knowledge/answers to force the Skill LoRA to ground its output in the provided Knowledge LoRA

Modeling

Base Model: Llama-2-7B

Training Method: Prereq-Tune (Two-stage LoRA training)

Objective Functions:

Purpose: Train Knowledge LoRA to learn facts.

Formally: Minimize Cross-Entropy Loss on D_know.
Purpose: Train Skill LoRA to learn task execution.

Formally: Minimize Cross-Entropy Loss on D_T, given frozen Knowledge LoRA.

Adaptation: LoRA (Low-Rank Adaptation)

Trainable Parameters: LoRA parameters only (base model frozen)

Training Data:

D_know: Generated by decomposing task data into statements or summarizing into passages (Top-down) or generating fictitious facts (Bottom-up)
D_T: Standard instruction tuning data or synthetic data generated based on D_know

Key Hyperparameters:

learning_rate: 2e-5
batch_size: 16
lora_r: 32
+ 2 more
lora_alpha: 64
num_epochs: Not reported in the paper

Comparison to Prior Work

vs. SFT: Prereq-Tune disentangles knowledge acquisition from skill learning, whereas SFT entangles them.
vs. Knowledge-Injection: Explicitly separates the parameters storing new knowledge (Knowledge LoRA) from those learning the task (Skill LoRA).
vs. RAG [not cited in paper]: RAG retrieves external documents at inference; Prereq-Tune 'retrieves' from a plug-and-play LoRA module or internal weights.

Limitations

Requires constructing a parallel 'prerequisite knowledge' dataset, which adds a data preparation step.
Reliance on synthetic/fictitious data might introduce artifacts if the generation model is poor.
Effectiveness depends on the Skill LoRA's ability to generalize from the Knowledge LoRA to the base model weights at inference time.

Reproducibility

Code: https://github.com/UCSB-NLP-Chang/Prereq_tune.git

Code is publicly available (https://github.com/UCSB-NLP-Chang/Prereq_tune.git). The paper details the prompts used for synthetic data generation and the logic for splitting knowledge/skills. Hyperparameters like LoRA rank are specified.

📊 Experiments & Results

Evaluation Setup

Evaluated on ability to generate factual biographies and answer questions, comparing Prereq-Tune against standard SFT and other baselines.

Benchmarks:

Biography Generation (Synthetic) (Long-form text generation) [New]
PopQA (Short-form QA on long-tail knowledge)

Metrics:

FactScore (percentage of atomic facts supported by Wikipedia)
Accuracy (Exact Match or substring match for QA)
Hallucination Rate
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Biography Generation (WikiBio)	FactScore	46.2	56.4	+10.2
PopQA	Accuracy	26.3	31.5	+5.2
Biography Generation	FactScore	52.1	56.4	+4.3

Experiment Figures

Illustration of the Multi-Version data construction process for fictitious entities.

Main Takeaways

Prereq-Tune significantly improves factuality (FactScore) compared to standard SFT by preventing the model from learning to hallucinate on unknown data.
The method effectively utilizes fictitious synthetic data; training on fake biographies helps the model answer questions about real people better than training on real data alone.
The 'Skill LoRA' successfully generalizes: it learns to query the 'Knowledge LoRA' during training and switches to querying the base model's pre-trained knowledge during inference.
Multi-version training (varying the facts about the same fictitious entity) forces stronger grounding, preventing the model from memorizing specific answers.

📚 Prerequisite Knowledge

Prerequisites

Understanding of Low-Rank Adaptation (LoRA) for LLMs
Distinction between Pre-training (knowledge acquisition) and Fine-tuning (skill acquisition)
Concept of Hallucination in LLMs due to knowledge inconsistency

Key Terms

LoRA: Low-Rank Adaptation—a parameter-efficient fine-tuning technique that freezes the main model weights and trains small rank-decomposition matrices instead

SFT: Supervised Fine-Tuning—training a pre-trained model on labeled instruction-output pairs to learn specific tasks

Knowledge LoRA: An adapter module in Prereq-Tune trained specifically to memorize facts/statements (prerequisite knowledge) relevant to the downstream task

Skill LoRA: An adapter module in Prereq-Tune trained on the task format (e.g., Q&A) while the Knowledge LoRA is active; it learns the task 'skill' rather than the facts

Knowledge Inconsistency: The mismatch between facts present in fine-tuning data and the facts (or lack thereof) in the model's pre-training corpus

Fictitious Synthetic Data: Training data generated by an LLM concerning non-existent entities (e.g., fake people), used here to teach the model to handle unknown information without hallucinating

RAG: Retrieval-Augmented Generation—AI systems that answer questions by first searching for relevant documents