Changing the encoder

[Karllas]

Hello. Is it possible to swap the encoder with some different encoder like ReStyle (https://github.com/yuval-alaluf/restyle-encoder) or Hyperstyle (https://github.com/yuval-alaluf/hyperstyle) for more accurate reconstruction of the face?

[williamyang1991]

Yes.
you just need to change this line to your encoder to return W+ latent code of your face image, or directly load your W+ latent code to replace instyle:

DualStyleGAN/style_transfer.py

Lines 103 to 104 in 64285b1

	img_rec, instyle = encoder(F.adaptive_avg_pool2d(I, 256), randomize_noise=False, return_latents=True,
	z_plus_latent=True, return_z_plus_latent=True, resize=False)

And then here set input_is_latent=True and z_plus_latent=False to indicate your instyle is W+ latent code rather than Z+ latent code.

DualStyleGAN/style_transfer.py

Lines 126 to 127 in 64285b1

	img_gen, _ = generator([instyle], exstyle, input_is_latent=False, z_plus_latent=True,
	truncation=args.truncation, truncation_latent=0, use_res=True, interp_weights=args.weight)

[Karllas]

Thank you for the quick reply. As I understand when use_res is set to True the generator should return the reconstructed. I did this and the image returned from generator is very different from the encoded one. What might be the reason?

[williamyang1991]

You should use False to only use the intrinsic path to reconstruct the image

DualStyleGAN/model/dualstylegan.py

Line 97 in 64285b1

use_res=True, # whether to use the extrinsic style path

[Karllas]

This is the result of encoding:

This is the result after feeding the W+ latent code to generator with input_is_latent=True, z_plus_latent=False and use_res=False

[williamyang1991]

You need to set args.truncation=1 to use the original intrinsic style code

DualStyleGAN/style_transfer.py

Lines 126 to 127 in 64285b1

	img_gen, _ = generator([instyle], exstyle, input_is_latent=False, z_plus_latent=True,
	truncation=args.truncation, truncation_latent=0, use_res=True, interp_weights=args.weight)

[Karllas]

The generator's result is still different. How to keep as much identity of the face when applying some style?

[williamyang1991]

If you use

img_gen, _ = generator([instyle], None, input_is_latent=True, z_plus_latent=False,
                      truncation=1, truncation_latent=0, use_res=False)

DualStyleGAN should act the same as StyleGAN

img_gen, _ = generator.generator([instyle], input_is_latent=True, truncation=1, truncation_latent=0)

Have you tried directly using StyleGAN (generator.generator) with instyle as input?

Another possible reason is that your encoder might optimize both the latent input and the nosie input.
So you might have to set the noise the same as what you get from your encoder.

[Karllas]

It worked. Thank you!

[3maoyap]

It worked. Thank you!

What encoder did you change and did it work well?

[weiweiecho]

hi ,what did you do to make it work

[weiweiecho]

@Karllas hi ,what did you do to make it work?

[watertianyi]

如果你使用

img_gen, _ = generator([instyle], None, input_is_latent=True, z_plus_latent=False,
                      truncation=1, truncation_latent=0, use_res=False)

DualStyleGAN 应该与 StyleGAN 一样

img_gen, _ = generator.generator([instyle], input_is_latent=True, truncation=1, truncation_latent=0)

您是否尝试过直接使用 StyleGAN (generator.generator) 作为instyle输入？

另一个可能的原因是您的编码器可能会同时优化潜在输入和噪音输入。 因此，您可能必须将噪声设置为与从编码器获得的噪声相同。

Can you talk about how to modify the code, it is not easy to understand here?

[williamyang1991]

I have added wplus encoder.
You can specify --wplus to use the original pSp encoder to extract the W+ intrinsic style code, which may better preserve the face features of the content image.

If you use a new encoder, that returns an optimized style code instyle1 and an optimized noise noise1, you need to use

img_gen, _ = generator([instyle1], None, input_is_latent=True, z_plus_latent=False, noise=noise1,  randomize_noise=False,
                      truncation=1, truncation_latent=0, use_res=False)     

for precise reconstruction.

[watertianyi]

如果你使用一个新的编码器，返回一个优化的样式代码instyle1和一个优化的噪声noise1，你需要使用

img_gen, _ = generator([instyle1], None, input_is_latent=True, z_plus_latent=False, noise=noise1, randomize_noise=False,
truncation=1, truncation_latent=0, use_res=False)
新的编码器跟噪声这是哪里来的
@williamyang1991

[williamyang1991]

新的编码器跟噪声这是哪里来的

This issue's title is "Changing the encoder"

And in the first pose, you can find the new encoder link

Hello. Is it possible to swap the encoder with some different encoder like ReStyle (https://github.com/yuval-alaluf/restyle-encoder) or Hyperstyle (https://github.com/yuval-alaluf/hyperstyle) for more accurate reconstruction of the face?

[watertianyi]

@williamyang1991 你的意思是将psp编码器换成上述两个编码器中的一个吗

[williamyang1991]

Because you post questions under this issue Changing the encoder, and you quote me reply to Karllas in your first question,
I assume you want to know how to replace the encoder with a new one (ReStyle/HyperStyle) as the first poster Karllas.