Continual Personalization for Diffusion Models

📝 Paper Summary

Generative personalization Memory internalization

CNS enables diffusion models to learn new personalized concepts sequentially without forgetting previous ones by identifying and updating only the sparse set of neurons specific to each new concept.

Core Problem

Existing personalization methods fail in continual learning settings: naively fine-tuning causes catastrophic forgetting of old concepts, while storing separate LoRA adapters for every concept is memory-inefficient and causes fusion conflicts.

Why it matters:

Real-world users accumulate new concepts (e.g., new pets, items) over time, requiring models to adapt continuously rather than being static
Current solutions like LoRA fusion suffer from 'concept vanishing' and 'concept confusion' when combining multiple adapters
Storing weights for every concept scales poorly; a fusion-free, single-model solution is needed for efficient deployment

Concrete Example: If a user first teaches a model their specific 'dog', and later teaches it a 'cat', standard fine-tuning will make the model forget the 'dog'. Alternatively, merging separate 'dog' and 'cat' LoRA adapters often results in a mixed-up image where the dog looks like the cat or vice versa.

Key Novelty

Concept Neuron Selection (CNS)

Identifies 'concept neurons' in the cross-attention layers that are highly responsive to the target concept but not to general image generation prompts
Selects these neurons by subtracting 'general neurons' (active for diverse prompts) from 'base neurons' (active for the specific concept images)
Updates only these specific neurons during training, using a regularization term to preserve weights associated with previously learned concepts

Architecture

The Concept Neuron Selection process. (a) Base neurons are selected by checking activation magnitude against a threshold for concept images. (b) General neurons are selected using calibration prompts. (c) Concept neurons are the set difference (Base - General).

Evaluation Highlights

Achieves state-of-the-art performance in both single and multi-concept personalization, outperforming LoRA fusion and Custom Diffusion methods
Maintains zero-shot generation capability better than baselines by freezing general neurons during updates
Reduces memory storage compared to LoRA-based approaches as it requires no additional adapter weights per concept (fusion-free)

Breakthrough Assessment

7/10

Offers a clever, efficient solution to the specific problem of continual personalization without expanding model size. While the scope is specific to diffusion models, the 'subtractive' neuron selection strategy is a strong methodological contribution.

⚙️ Technical Details

Problem Definition

Setting: Continual learning where a model learns a sequence of concepts 1...m, seeing only images for concept m at step m, while preserving knowledge of 1...m-1

Inputs: A sequence of small image sets (N_m images) for each new concept m

Outputs: A single diffusion model W_m capable of generating all concepts 1...m and retaining zero-shot capabilities

Pipeline Flow

Base Neuron Identification (finds neurons active for concept images)
General Neuron Identification (finds neurons active for general prompts)
Concept Neuron Selection (Computes Concept = Base AND NOT General)
Incremental Fine-tuning (Updates Concept neurons with regularization)

System Modules

Neuron Selector

Identify which weights in W_k and W_v should be updated

Model or implementation: Based on correlation between weights and text embeddings

Fine-tuner

Update the selected weights to learn the new concept while preserving old ones

Model or implementation: Standard Diffusion Loss + Regularization

Novel Architectural Elements

Neuron-level masking strategy for continual learning: unlike LoRA which adds modules, this selects sparse subsets of existing weights to update
The 'subtractive' selection logic (Concept = Base - General) to isolate concept-specific parameters

Modeling

Base Model: Latent Diffusion Models (LDMs)

Training Method: Sparse Fine-tuning with Regularization

Objective Functions:

Purpose: Minimize reconstruction error for the new concept.

Formally: Standard LDM noise prediction loss on concept images.
Purpose: Prevent forgetting of previous concepts and general knowledge.

Formally: L_reg = || M_reg * (W_m - W_{m-1}) ||^2 + || (1 - M_reg) * (W_m - W_0) ||^2, where M_reg is the union of current and past concept masks.

Adaptation: Updates specific neurons in cross-attention layers (Key and Value projections)

Trainable Parameters: Sparse subset of Cross-Attention weights (determined by M_concept)

Training Data:

Few-shot concept images (N_m images per concept)
Calibration prompts for general neuron detection (K=20 prompts)

Key Hyperparameters:

calibration_prompts_count: 20
thresholding: Pre-determined threshold for neuron selection (see supplementary)
general_neuron_overlap: ~53% (empirically observed)

Compute: Fusion-free inference (no extra test-time compute). Memory efficient (no extra weights stored).

Comparison to Prior Work

vs. DreamBooth/Custom Diffusion: CNS supports continual learning without catastrophic forgetting by regularizing non-concept neurons.
vs. C-LoRA: CNS does not require storing separate LoRA modules or merging them; it uses a single set of weights.
vs. LoRA Fusion methods: CNS is 'fusion-free'; avoids the concept confusion caused by merging conflicting adapter weights.

Limitations

Relies on the assumption that concept neurons are distinct from general neurons; overlap might still cause minor degradation.
The threshold for neuron selection is a hyperparameter that might need tuning for different domains.
Requires access to the base model weights W_0 and previous weights W_{m-1} for regularization.

Reproducibility

Code: https://github.com/YuChienLiao/CNS

Code is publicly available at https://github.com/YuChienLiao/CNS. The paper details the logic for neuron selection (Eq 3-6) and the specific loss function (Eq 7-8). The use of 20 calibration prompts is specified.

📊 Experiments & Results

Evaluation Setup

Personalization on single and multiple concepts, evaluating both concept fidelity and zero-shot preservation.

Benchmarks:

Custom Concept Datasets (Image Generation)

Metrics:

CLIP-T (Text alignment)
CLIP-I (Image alignment/Concept fidelity)
Catastrophic Forgetting (Retention of previous concepts)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Evaluation on real-world datasets demonstrates state-of-the-art performance.
Multi-concept generation	Visual Quality	Concept confusion / Overfitting	Clear separation of attributes	Improved

Experiment Figures

Jaccard similarity of selected neurons across different prompts, showing a large overlap (53%) as the number of prompts increases.

Main Takeaways

CNS successfully prevents catastrophic forgetting in continual learning settings compared to naive fine-tuning.
The method preserves zero-shot capabilities better than full fine-tuning or standard LoRA by explicitly freezing 'general' neurons.
Fusion-free operation significantly reduces memory usage and inference complexity compared to multi-LoRA approaches.
About 53% of neurons are identified as 'general neurons' (always selected across prompts), confirming the validity of masking them out for personalization.

📚 Prerequisite Knowledge

Prerequisites

Understanding of Latent Diffusion Models (LDMs) and cross-attention mechanisms
Familiarity with Low-Rank Adaptation (LoRA) and fine-tuning
Basic concepts of Continual Learning (catastrophic forgetting, regularization)

Key Terms

CNS: Concept Neuron Selection—the proposed method to identify and update only neurons specific to a new concept

LoRA: Low-Rank Adaptation—a parameter-efficient fine-tuning technique that freezes pre-trained weights and injects trainable rank decomposition matrices

Cross-attention: A mechanism in diffusion models where text conditions influence the image generation process; the weights W_k and W_v are the focus of this paper

Base neurons: Neurons in the cross-attention layers that activate strongly for the specific images of the target concept

General neurons: Neurons that activate strongly across a diverse set of general prompts, indicating they are responsible for general image synthesis rather than specific concepts

Concept neurons: The subset of neurons obtained by removing General neurons from Base neurons; these are the only ones updated for personalization

Catastrophic forgetting: The tendency of a neural network to completely lose previously learned information upon learning new information

Zero-shot generation: The ability of the model to generate images from text prompts it hasn't been explicitly fine-tuned on (e.g., standard objects)