Pix-to-Pix as Viton?

[Bilal143260]

Is it a good idea to train pix-to-pix-instruct to act like a viton (virtual try-on)? The training data would include images of cloth and prompts as input and a model wearing that dress as ground truth.

[sayakpaul]

I think it's better to use IP Adapters. I responded the same thing in the issue you had created.

[Bilal143260]

Yeah. I am training it. Hoping to get some good results

[a-r-r-o-w]

Have you given existing VTON papers a try btw? I haven't played around much with training but I feel directly training Pix2Pix on cloth-model pairs may not work. A few current approaches involve learning separate embeddings for clothes and applying a warping module for cloth placement.

• ladi-vton
• DCI-vton
• TryOnDiffusion

I think it's better to use IP Adapters. I responded the same thing in the issue you had created.

They're mostly unable to handle cloth consistency from what I've seen, but it can do a decent job at times with the right prompting + img2img and correct parameters.

[Bilal143260]

Exisiting vitons lose the texture details. The best I tested is Stable viton