Difference Inversion: Interpolate and Isolate the Difference with Token Consistency

How can we generate an image B' that satisfies A:A'::B:B', given the input images A,A' and B? Recent works have tackled this challenge through approaches like visual in-context learning or visual instruction. However, these methods are typically limited to specific models (e.g. InstructPix2Pix, inpainting models) rather than general diffusion models (e.g. Stable Diffusion, SDXL). This dependency may lead to inherited biases or lower editing capabilities. In this paper, we propose Difference Inversion, a method that isolates only the difference from A and A' and applies it to B to generate a plausible B'. To address model dependency, it is crucial to structure prompts in the form of a "Full Prompt" suitable for input to stable diffusion models, rather than using an "Instruction Prompt". To this end, we accurately extract the Difference between A and A' and combine it with the prompt of B, enabling a plug-and-play application of the difference. To extract a precise difference, we first identify it through 1) Delta Interpolation. Additionally, to ensure accurate training, we propose the 2) Token Consistency Loss and 3) Zero Initialization of Token Embeddings. Our extensive experiments demonstrate that Difference Inversion outperforms existing baselines both quantitatively and qualitatively, indicating its ability to generate more feasible B' in a model-agnostic manner.

• Delta Interpolation: Rather than relying solely on image‐space differences, we extract both an Image Delta (CLIP‐image embedding difference) and a Text Delta (CLIP‐text embedding difference from BLIP2 captions), then apply spherical linear interpolation (α=0.8) in CLIP’s joint hypersphere. This yields a more precise Delta that captures both visual and semantic changes between A and A′.
• Token Consistency Loss: To ensure the Difference Tokens encode only the intended modifications, we enforce bidirectional consistency: when \( \widetilde{D} \) is concatenated with prompt A, the model reconstructs A′; when \( \widetilde{D} \) is negated or removed with prompt A′, it reconstructs A. We implement this via a two‐term MSE loss on predicted noise (before/after) so that \( \widetilde{D} \) activates only the exact difference-information.
• Zero-Initialization of Token Embeddings: All Difference Tokens are initialized to zero, preventing any unintended artifacts. Starting from zero ensures that optimization focuses exclusively on learning the precise difference, stabilizing training and minimizing spurious artifacts.

• Top: CLIP and DINO directional similarity — measures how well the generated B′ aligns with the A:A′ transformation in embedding space, indicating semantic and visual fidelity.
• Bottom-left: Human evaluation — collects perceptual ratings from human annotators to assess the realism and adherence of B′ to the intended analogy, providing a subjective quality check.
• Bottom-right: Large-scale VLM evaluation — uses large vision–language models to automatically score how plausibly B′ satisfies the A:A′::B:B′ relationship, enabling scalable, objective assessment.