Unifying Search and Recommendation in LLMs via Gradient Multi-Subspace Tuning

📝 Paper Summary

Unified Search and Recommendation (S&R) Parameter-Efficient Fine-Tuning (PEFT) Multi-Task Learning in LLMs

GEMS unifies search and recommendation in LLMs by projecting gradients into shared and task-specific subspaces to resolve conflicts, while using null-space projection to preserve the model's pre-trained reasoning capabilities.

Core Problem

Applying Parameter-Efficient Fine-Tuning (PEFT) to unified Search & Recommendation causes gradient conflicts due to divergent objectives and degrades the LLM's general reasoning abilities (catastrophic forgetting).

Why it matters:

Full fine-tuning of LLMs for S&R is computationally expensive and limits scalability.
Search (short-term intent) and Recommendation (long-term preference) generate conflicting gradients, destabilizing training when forced into shared low-rank parameters.
Overfitting to S&R data distorts the LLM's general-domain knowledge, causing it to fail at basic reasoning tasks it could previously handle.

Concrete Example: In a standard PEFT setup, the search task might push gradients to prioritize query relevance, while recommendation pushes for user history alignment. These vectors may oppose each other (Figure 1a), canceling out progress. simultaneously, the model might lose the ability to answer general questions (Figure 1b) due to representation shift.

Key Novelty

Gradient Multi-Subspace Tuning (GEMS)

Decomposes optimization into three low-rank subspaces: a 'shared' subspace for common signals, and two 'task-specific' subspaces for Search and Recommendation, preventing destructive interference.
Identifies the 'principal' directions of the pre-trained model's knowledge using SVD and projects all updates onto the 'null space' (orthogonal directions), ensuring new learning doesn't overwrite existing general knowledge.

Architecture

Schematic of the GEMS framework illustrating the Multi-Subspace Decomposition and Null-Space Projection processes during an optimization step.

Breakthrough Assessment

8/10

Proposes a mathematically grounded approach (subspace tuning) to solve the specific multi-task interference problem in LLMs without adding inference parameters, addressing both stability and forgetting.

⚙️ Technical Details

Problem Definition

Setting: Generative Unified Search and Recommendation (S&R) as conditional text generation

Inputs: Unified structured prompt x containing interaction history H_u (items and interaction types) and optional query q

Outputs: Target item identifier i (sequence of tokens)

Pipeline Flow

Input Processing: Construct prompt from history/query
Backbone LLM: Process input to hidden states
Subspace Optimization (Training only): Compute gradients -> Decompose -> Project -> Update
Inference: Beam search for Top-K items

System Modules

Backbone LLM

Process structured prompts and generate item identifiers

Model or implementation: LLM (architecture not specified in provided text, implied generic LLM)

Adaptive Gating Network

Dynamically weight the contribution of task-specific subspaces based on training statistics

Model or implementation: Two-layer MLP with ReLU

Novel Architectural Elements

No additional inference modules: unlike LoRA which adds adapter layers, GEMS updates the base weights directly via projected gradients, resulting in zero architectural change at inference time.

Modeling

Base Model: Large Language Models (specific variants not listed in provided text)

Training Method: Gradient Multi-Subspace Tuning (GEMS)

Objective Functions:

Purpose: Minimize negative log-likelihood of target item tokens.

Formally: L(Theta) = - sum log P(i_t | i_<t, x; Theta)
Purpose: Fuse gradients from shared and task-specific subspaces.

Formally: Delta_fused = Delta_shared + alpha_src * Delta_src + alpha_rec * Delta_rec

Adaptation: Subspace Tuning (updates restricted to low-rank basis derived from gradients)

Trainable Parameters: Full parameter space conceptually, but updates are rank-constrained (efficient optimizer states)

Key Hyperparameters:

T_svd: Interval for refreshing SVD basis (e.g., once every T_svd steps)
tau: Gate temperature factor for softmax in adaptive gating
rank: Subspace rank r (dimension of low-rank projection)

Compute: Requires maintaining optimizer states only for low-rank subspaces (shared + 2 task-specific), significantly lower memory than full fine-tuning.

Comparison to Prior Work

vs. BSR/GenSAR: GEMS avoids full fine-tuning, reducing computational cost while maintaining performance.
vs. LoRA: GEMS introduces no extra parameters (adapters) and explicitly handles gradient conflicts via subspace decomposition.
vs. LoRA-MoE: GEMS does not require routing mechanisms or multiple expert modules, relying instead on gradient projection.

Limitations

Requires frequent SVD computation (every T_svd steps) which can be computationally overhead if T_svd is too small.
Relies on the assumption that gradient conflict directions can be perfectly disentangled by subspace decomposition.
Null-space projection computation requires a representative general-domain corpus for covariance estimation.

Reproducibility

No specific code URL or dataset details were provided in the text. The method relies on computing SVD of gradients and hidden states, which is mathematically specified.

📊 Experiments & Results

Evaluation Setup

Unified Search and Recommendation (S&R) formulated as conditional generation

Benchmarks:

Not reported in the provided text (Search and Recommendation)

Metrics:

Not reported in the provided text
Statistical methodology: Not explicitly reported in the paper

Main Takeaways

GEMS consistently outperforms state-of-the-art baselines across search and recommendation tasks (qualitative claim from Abstract).
The framework maintains performance gains on LLMs with billions of parameters without introducing additional trainable weights.
Gradient conflicts are effectively mitigated by separating shared and task-specific signals into orthogonal subspaces.
General-domain knowledge is preserved via Null-Space Projection, preventing the semantic drift often seen in standard PEFT.

📚 Prerequisite Knowledge

Prerequisites

Linear Algebra (SVD, Null Space, Projections)
Parameter-Efficient Fine-Tuning (PEFT/LoRA)
Gradient Descent Optimization (Adam)
Generative Recommendation

Key Terms

GEMS: Gradient Multi-Subspace Tuning—the proposed framework that optimizes LLMs within specific low-rank subspaces to separate task signals.

PEFT: Parameter-Efficient Fine-Tuning—adapting pre-trained models by updating only a small subset of parameters (e.g., LoRA).

SVD: Singular Value Decomposition—a mathematical method used here to identify the principal directions (basis vectors) of gradients and pre-trained representations.

Null-space Projection: Restricting parameter updates to directions orthogonal to the pre-trained model's dominant features to prevent forgetting general knowledge.

S&R: Search and Recommendation—two core information retrieval tasks unified in this framework.

Gradient Conflict: When gradients from different tasks (e.g., Search vs. Rec) point in opposing directions, hindering joint optimization.

Subspace Tuning: An optimization technique where gradients are projected into a low-rank subspace spanned by principal directions, reducing memory and computation.

PLM: Pre-trained Language Model.