Is there a way to extract embeddings for prosody, content and acoustic details?

[iv41879]

@lifeiteng Is there a way to extract embeddings for prosody, content and acoustic details? Thank you

[hmohebbi]

I tried this (authors may correct me(?)):

with torch.no_grad():
    enc_out = fa_encoder(test_wav)
    vq_post_emb, vq_id, _, quantized, spk_embs = fa_decoder(enc_out, eval_vq=False, vq=True)
    prosody_code = vq_id[:1]
    content_code = vq_id[1:3]
    residual_code = vq_id[3:]
    quantizer = fa_decoder.quantizer.eval()
    prosody_embedding = quantizer[0].vq2emb(prosody_code)
    content_embedding = quantizer[1].vq2emb(content_code)
    residual_embedding = quantizer[2].vq2emb(residual_code)