Ernie-M

[KnutJaegersberg]

Model description

Ernie-M looks pretty good in multilingual benchmarks, beating XLM-Roberta.
Paddlepaddle recently added ernie-m to the huggingface repo, we can use it with paddlenlp.transformers.
Would be nice to have the model supported in huggingface transformer as well.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

https://huggingface.co/PaddlePaddle/ernie-m-base
https://huggingface.co/PaddlePaddle/ernie-m-large

[shermansiu]

https://github.com/PaddlePaddle/ERNIE/blob/ernie-kit-open-v1.0/erniekit/modules/ernie.py

has more implementation details.

[susnato]

Hi, @shermansiu is there any pytorch/tf implementation of this model?

[shermansiu]

None that I'm aware of.

Anyways, the author of ERNIE-Pytorch ported over a few other Ernie models to Huggingface. I'm sure it could be adapted for this. And the PaddlePaddle syntax is quite similar to that of PyTorch, so I'm sure it should be relatively easy, though it'll probably take some time.

[susnato]

@shermansiu Thanks for the resources!
I am currently trying to port the model to huggingface(pytorch), (done till Embedding Layer with acceptable tolerance of 1e-3)

[susnato]

Hi @KnutJaegersberg, Ernie-M is implemented!