How to do outpainting correctly?

[darshats]

Hi,
Thanks for the great work!
My question is if this approach be used to do outpainting? Say I have an object without any other surrounding context, I'm having trouble getting it to fill up the context based on prompt.
Below on left is the object. The diffdiff map is the second image and outputs are the right two images. The trouble is it always adds another object of the same type in the spatial location that is masked. I used a binary diffdiff map as its ok to outpaint anything as per prompt.
Using a gradient mask based on distance also doesnt help as areas close to the object also need complete infilling.

[neph1]

I haven't yet tried out differential-diffusion, but I think this is a case of 'too much freedom, too little context'.
I think you would be better off generating a background you think would be suitable, separately, then merge the two images (crudely) and make a mask to let SD work on the area directly surrounding the bottle (inpainting rather than outpainting).

[Clement-Lelievre]

if you can afford that, maybe one thing you could try is leverage the change map by setting the pixels just surrounding the bottle (or your object more generally) to non-zero, to try and force the pipe to respect what was just around the bottle

[Hackerman25]

@darshats Can you share the settings and model you used for your image here so i can try the same, is it SDXL?