RLSR: Reinforcement Learning with Supervised Reward Outperforms SFT in Instruction Following

📝 Paper Summary

Instruction Tuning Reinforcement Learning

RLSR replaces token-level supervised fine-tuning with reinforcement learning, using semantic similarity to human-labeled responses as a reward to significantly improve a model's instruction-following capabilities.

Core Problem

Supervised Fine-Tuning (SFT) forces models to exactly replicate human responses token-by-token, limiting their ability to explore alternative phrasing and often confining them to suboptimal local optima.

Why it matters:

Token-level supervision severely restricts the exploratory capacity needed for robust generalization in large language models
Existing reinforcement learning alternatives (like RLHF) require complex, expensive reward models or are limited to domains with verifiable answers (like math)

Concrete Example: When fine-tuned with standard SFT, a model must predict the exact next token from the human dataset. If the model generates a semantically perfect but differently phrased response, SFT penalizes it, whereas RLSR correctly rewards the semantic match.

Key Novelty

Reinforcement Learning with Supervised Reward (RLSR)

The model generates multiple candidate responses for a given prompt to encourage exploration
A pre-trained embedding model evaluates each candidate by measuring its cosine similarity to the human-labeled target response
This semantic similarity score acts as the reward signal for reinforcement learning, bypassing the need for token-by-token matching

Evaluation Highlights

RLSR using SentenceBERT achieves a 26.34% AlpacaEval win rate on Qwen-7B, surpassing standard SFT's 21.01%
Combining SFT with RLSR yields the strongest performance, reaching a 30.73% AlpacaEval win rate on Qwen-7B

Breakthrough Assessment

7/10

Provides a highly practical, compute-efficient bridge between SFT and RLHF by repurposing existing datasets via semantic rewards, though heavily dependent on the chosen embedding model's quality.

⚙️ Technical Details

Problem Definition

Setting: Instruction tuning of base language models using prompt-response pairs within a reinforcement learning framework

Inputs: Natural language prompt x

Outputs: Generated response y

Pipeline Flow

Candidate Generation
Embedding Extraction
Reward Computation
Policy Update

System Modules

Policy Model

Generates multiple candidate responses for a given prompt to enable exploration

Model or implementation: Llama-3.1-8B, Qwen2.5-7B, or Qwen2.5-32B

Reward Evaluator

Computes semantic similarity between generated candidates and the human-labeled target

Model or implementation: SentenceBERT (SB) or Qwen3-Embedding-8B (Qwen-EM)

Novel Architectural Elements

Direct integration of a frozen semantic embedding model as the reward function within the RL pipeline, replacing both standard cross-entropy loss and learned preference models

Modeling

Base Model: Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-32B

Training Method: Reinforcement Learning with Supervised Reward (RLSR)

Objective Functions:

Purpose: Reward semantic alignment with the human-labeled response.

Formally: r(x, y, y^*) = cosine_similarity(E(x, y), E(x, y^*))
Purpose: Maximize expected reward while remaining close to the reference policy.

Formally: \max_{\pi_\theta} \mathbb{E}[r(x, y, y^*) - \beta D_{KL}(\pi_\theta || \pi_{ref})]
Purpose: Penalize repetitive generations.

Formally: Apply -1 penalty if LCS(y) > 100 and LCS(y)/|y^*| > 0.1

Training Data:

TULU dataset (100k sampled single-turn examples)
INFINITY dataset (100k sampled single-turn examples)

Key Hyperparameters:

learning_rate: 1e-6 (Llama), 3e-6 (Qwen)
batch_size: 1024
ppo_clip_epsilon: Not explicitly reported, but uses PPO/GRPO framework
+ 4 more
kl_coefficient: 0.001
ppo_mini_batch_size: 256
rollouts_per_prompt: 8
max_sequence_length: 4096

Compute: Not reported in the paper

Comparison to Prior Work

vs. SFT: Replaces token-level teacher-forcing with exploration and semantic similarity rewards
vs. RLHF: Uses a fixed embedding-based cosine similarity reward instead of training a separate human-preference reward model
vs. RLVR: Relies on semantic similarity to SFT data rather than sparse, verifiable correctness signals (e.g., math answers)
+ 2 more
vs. RFT: Fine-tunes the base model for general instruction following rather than adapting a specific reasoning model to new domains
vs. DPO: Optimizes directly against a target response embedding rather than requiring pairwise preference data [not cited in paper]

Limitations

Requires high-quality SFT data with ground-truth responses to compute rewards
Reward quality heavily depends on the representational capacity of the chosen embedding model (e.g., SentenceBERT struggled to guide the larger Qwen-32B model)
RL training consumes significantly more computational resources (FLOPs) than standard SFT
No statistical significance tests reported for the benchmark improvements

Reproducibility

No code repository, pre-trained weights, or exact data splits are provided. The methodology uses publicly available base models (Llama-3.1, Qwen2.5), embedding models (SentenceBERT, Qwen3-Embedding), and open-source frameworks (VERL for RL, OpenRLHF for SFT).

📊 Experiments & Results

Evaluation Setup

Instruction-following and open-ended generation evaluation across diverse benchmarks

Benchmarks:

AlpacaEval (Instruction-following and open-ended generation)
Arena-Hard (Instruction-following and open-ended generation)
LM Eval Harness (18 standard benchmarks) (Reasoning, mathematics, programming, bias, and factuality)

Metrics:

AlpacaEval Length-Controlled Win Rate (LC WR)
Arena-Hard Win Rate
Accuracy/Score on LM Eval Harness tasks
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Generative instruction-following performance on AlpacaEval demonstrates the superiority of RL-based methods over standard SFT.
AlpacaEval	Length-Controlled Win Rate (LC WR)	21.01	26.34	+5.33
AlpacaEval	Length-Controlled Win Rate (LC WR)	21.01	30.73	+9.72

Main Takeaways

RLSR consistently outperforms SFT across diverse model architectures (Llama, Qwen) and datasets (TULU, INFINITY), winning on the vast majority of the 18 LM Eval Harness benchmarks.
The SFT + RLSR pipeline is the strongest approach for generative instruction-following tasks (AlpacaEval, Arena-Hard), effectively preserving linguistic fluency while improving alignment.
RLSR alone excels slightly more in discriminative reasoning tasks compared to the hybrid SFT+RLSR approach.
Scale-dependent alignment effect: larger embedding models (Qwen-EM) provide better reward signals for larger LLMs (Qwen-32B), while smaller models (SentenceBERT) suffice for smaller LLMs.

📚 Prerequisite Knowledge

Prerequisites

Supervised Fine-Tuning (SFT)
Reinforcement Learning (RL) principles
Text embeddings and semantic similarity

Key Terms

SFT: Supervised Fine-Tuning—training a model to predict the exact next token of a human-provided response

RLHF: Reinforcement Learning from Human Feedback—using a separately trained reward model based on human preferences to guide policy updates

RLVR: Reinforcement Learning with Verifiable Rewards—using strict, rule-based correctness (like math answers) as rewards

RFT: Reinforcement Fine-Tuning—adapting reasoning models to specific domains using programmable graders

RLSR: Reinforcement Learning with Supervised Reward—the proposed method using semantic similarity to reference text as the reward

PPO: Proximal Policy Optimization—a stable reinforcement learning algorithm

GRPO: Group Relative Preference Optimization—a simplified RL alternative to PPO that eliminates the value network by using mean rewards

Teacher-forcing: A training paradigm where the model is strictly conditioned on the ground-truth sequence during optimization

Cosine similarity: A mathematical metric used to measure how semantically similar two text embeddings are

Embedding space: A vector representation where text with similar meanings are placed close together

LCS: Longest Common Substring—an algorithm used here to detect and penalize repetitive text generation

SB: SentenceBERT—a lightweight (33M parameter) text embedding model used for reward computation

Qwen-EM: Qwen3-Embedding-8B—a large (8B parameter) text embedding model used for reward computation

KL divergence: A penalty used in RL to prevent the model from drifting too far from its original behavior

AlpacaEval: A benchmark evaluating instruction-following capabilities using human-preference metrics

Arena-Hard: A benchmark designed for evaluating open-ended generation quality

FLOPs: Floating Point Operations—a measure of computational cost