ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

Abstract

Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images. Increasingly, these images come from a diffusion model rather than a camera. We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. To isolate the cause, we fine-tune two models from the same base model with the same DreamBooth recipe, one on real photos of a subject and one on synthetic images of that subject generated by the first. We trace the degradation to classifier-free guidance (CFG). For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions, and with it the norm of their difference, is much larger than for the model personalized on real photos. This inflation grows toward high frequencies and also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones. We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly. ReGain needs no real photos. On Stable Diffusion v1.5, ReGain closes 51–64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I. It also improves subject fidelity on SDXL and SD 3.5 and preserves text alignment on all three backbones.

Problem

Personalizing a diffusion model on synthetic images of a subject degrades its fidelity. Why?

To study this, we train diffusion models M_real and M_syn with the same recipe, the only difference being their training data: real photos and synthetic images respectively.

Real photos of the subject

DreamBooth from M_base

M_real

sample, same prompt and seed

Clean sample generated by the model trained on real photos

Generated: clean

M_real generates the five synthetic training images of the subject

Synthetic images of the subject

DreamBooth from M_base, same recipe

M_syn

sample, same prompt and seed

Oversaturated sample with excess texture generated by the model trained on synthetic images

Generated: oversaturated, excess texture

Prompt for both samples: “a photo of a [V] dog in front of the Acropolis”, where [V] is the identifier that DreamBooth binds to the subject.

Both models are fine-tuned with DreamBooth from M_base and sampled with the same prompt and seed.

Findings

Notation

Classifier-free guidance (CFG) with weight w:Δ = ε_c − ε_∅ε̂ = ε_∅ + w Δ

ε_c and ε_∅ are the noise predictions with and without the prompt.

The outputs of M_syn carry more power at every frequency.

Power spectrum of generated images in the latent space.

M_real M_syn

The guidance term Δ of M_syn is inflated throughout denoising.

M_real M_syn

The plots in Findings are for this subject on SD1.5. Step t = 0 is the noisiest. The two plots above show the mean ± s.e. over 10 seeds.

The Δ inflation, the excess in ‖Δ‖ of M_syn over M_real, has four properties.

1Angle-driven.

The norms of ε_c and ε_∅ do not grow. The angle θ between them is wider for M_syn at every step, so Δ inflates.

M_real trained on real photos

M_syn trained on synthetic images

Angles are drawn 6× wider so that they are visible. Mean over 10 seeds.

2Semantic.

The class name (“a dog”) shows nearly the full inflation and a related class (“a cat”) about half. Unrelated prompts show little to none.

Angle excess θ^M_syn(t) − θ^M_real(t) for the subject prompt and increasingly semantically distant prompts.

3Persists on the same input.

When M_real and M_syn denoise the same noisy copies of the training images of M_syn (as opposed to their own sampling trajectories in the plots above), the Δ of M_syn is still larger. The inflation is a property of the model.

4High-frequency and time-varying.

High-frequency bands inflate several-fold, and each band follows its own time profile. A lower guidance weight w acts uniformly across bands and steps, so it cannot undo this.

Energy of Δ in each radial frequency band k and step t, M_syn divided by M_real; red marks inflation.

low k (1-8) mid k (9-24) high k (25-45)

Excess of the angle θ of M_syn over M_real, in percent, within the low, mid and high bands.

Method

ReGain uses the base model M_base as the reference for how large Δ should be, because M_base is freely available and its Δ for the subject’s class is uninflated.

ReGain overview: (A) personalization on synthetic images, (B) the wider angle inflates the guidance term, (C) one-time calibration against the base model and per-step band-wise correction inside the subject mask — **Overview.**

AM_syn is DreamBooth fine-tuned on synthetic subject images generated by M_real.

BThe angle θ between the unconditional and conditional predictions is wider for M_syn, so its guidance term Δ = ε_c − ε_∅ is inflated.

C

ReGain.

**(i)**Once per model, M_base and M_syn are evaluated at forward-noised training images z_t; the square root of the ratio of their masked band energies e gives the gain ĝ(k, t), compressed into the schedule g_b(t) (darker blue attenuates more; the gain schedule below).

**(ii)**At each step, Δ is rescaled per band group by g_b(t) inside the subject mask m (green) and left unchanged outside it (grey), then scaled by w and added to ε_∅, the ReGain update below.

C (i)Gain schedule g_b(t)

\hat{g} (k, t) = \sqrt{\frac{e^{M_{base}} (k, t)}{e^{M_{syn}} (k, t)}}

compressed intog_b(t)

e(k, t)energy of Δ in frequency band k at step t, inside the subject mask

ĝ(k, t)gain for band k at step t

g_b(t)gain schedule, ĝ compressed into a few cells of constant gain, one row per band group b

For different subjects, measured using our setup.

C (ii)ReGain update:

ε̂ = ε_∅+w m ⊙ ∑_b g_b(t) B_b(Δ)inside the subject mask+w (1 − m) ⊙ Δoutside the mask

ε̂corrected noise prediction

mbinary mask of the subject

bband group, a contiguous range of frequency bands that share one gain

g_b(t)gain schedule, the factor that scales band group b at step t

B_b(Δ)component of Δ in band group b

⊙elementwise product

Results

Visual results

Each card is one subject, prompt and seed. Drag the slider to compare plain CFG with ReGain at the same guidance weight w.

Quantitative improvements in subject fidelity on the DreamBooth dataset

Each track runs from M_syn with plain CFG to M_real. The number is the share of that gap that ReGain recovers.

Show Table 1

	SD1.5, DreamBooth				SD1.5, DreamBooth-LoRA
Method	CLIP-I	DINO	DINOv2	CLIP-T	CLIP-I	DINO	DINOv2	CLIP-T
M_real + CFG	0.814	0.681	0.661	0.300	0.791	0.630	0.606	0.306
M_syn + CFG	0.788	0.611	0.611	0.289	0.750	0.540	0.510	0.300
M_syn + S-CFG	0.781	0.605	0.604	0.292	0.744	0.536	0.503	0.302
M_syn + CFG++	0.788	0.606	0.608	0.289	0.748	0.535	0.508	0.299
M_syn + FDG	0.792	0.608	0.614	0.284	0.755	0.538	0.517	0.297
M_syn + CFG + ReGain	0.802	0.647	0.643	0.290	0.775	0.594	0.568	0.304
↪ Gap recovered	54%	51%	64%	+0.001	61%	60%	60%	+0.004
	SDXL, DreamBooth-LoRA				SD3.5, DreamBooth-LoRA
M_real + CFG	0.757	0.558	0.535	0.289	0.811	0.682	0.650	0.311
M_syn + CFG	0.724	0.477	0.467	0.278	0.776	0.586	0.567	0.307
M_syn + CFG + ReGain	0.736	0.502	0.494	0.279	0.792	0.605	0.596	0.307
↪ Gap recovered	36%	31%	40%	+0.001	46%	20%	35%	0.000

Table 1: Subject and prompt fidelity on DreamBooth, averaged over 30 subjects, 25 prompts and 3 seeds (higher is better). The first row is the reference model M_real, and every other row samples from M_syn. Baselines, on SD1.5 only: S-CFG (Shen et al., 2024), CFG++ (Chung et al., 2025) and FDG (Sadat et al., 2025b). Bold marks the best M_syn result. Gap recovered is the part of the drop from M_real to M_syn, both with CFG, that ReGain wins back. For CLIP-T we report the change from M_syn + CFG instead.

Over-guidance artifacts

ReGain moves saturation, contrast and high-band power inside the subject region back toward M_real.

Show Table 2

Method	Saturation	Contrast	High band
M_real + CFG	0.301	0.211	0.061
M_syn + CFG	0.355	0.258	0.069
M_syn + CFG + ReGain	0.309	0.234	0.066
↪ Gap recovered	85%	51%	38%

Table 2: Over-guidance artifacts inside the subject region (SD1.5, DreamBooth). Bold marks the M_syn row closer to M_real.

BibTeX

If you find our work useful, please consider citing:

@misc{bhatnagar2026regainrestoringsubjectfidelity,
      title={ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images},
      author={Shubhang Bhatnagar and Ishan Bhatnagar and Viraj Shah and Narendra Ahuja},
      year={2026},
      eprint={2609.38680},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.38680},
}

Abstract

Problem

Findings

The outputs of Msyn carry more power at every frequency.

The guidance term Δ of Msyn is inflated throughout denoising.

1Angle-driven.

Mreal trained on real photos

Msyn trained on synthetic images

2Semantic.

3Persists on the same input.

4High-frequency and time-varying.

Method

Results

Visual results

Quantitative improvements in subject fidelity on the DreamBooth dataset

Over-guidance artifacts

BibTeX

The outputs of M_syn carry more power at every frequency.

The guidance term Δ of M_syn is inflated throughout denoising.

M_real trained on real photos

M_syn trained on synthetic images