LED

[patrickvonplaten]

What does this PR do?

Adds LongformerEncoderDecoder (LED) from @ibeltagy - see: https://github.com/allenai/longformer#longformer

Todo:

• Important: position embeddings have to be cut to correctly convert original Bart-ilke checkpoints to LED. The reason is that Bart uses a position embedding hack because of which the embedding idx 0 and 1 are never used resulting in an embedding matrix that has a length of 1026 instead of 1024, see:

  transformers/src/transformers/models/bart/modeling_bart.py

  Line 131 in 88ef889

  # Bart is set up so that if padding_idx is specified then offset the embedding ids by 2

  . All LED checkpoints are hence cut to remove this hack in LED:

model = LEDForConditionalGeneration.from_pretrained("./led-base-16384")
model.model.encoder.embed_positions.weight = torch.nn.Parameter(model.model.encoder.embed_positions.weight[2:, :])
model.model.decoder.embed_positions.weight = torch.nn.Parameter(model.model.decoder.embed_positions.weight[2:, :])
model.save_pretrained("./led-base-16384")

• Make Pytorch integration tests pass. See LEDIntegrationTests in tests/test_modeling_led.py.
• Add gradient_checkpointing
• Make common tests work
• Add convenient padding function so that input can be of whatever size and add global_attn logic to mask
• Automatically create attention_mask in encoder if not provided
• Finish PT version
• Make TF version work
• Add tips in docs for LED
• Eval notebook: https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing
• Nice to have: Fine-tune notebook: https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing
• Make nice model cards

TODO after PR is merged:

• Correctly add # Copied from .... statements from Bart and Longformer (this probably requires the Bart refactor to be merged before)
• Open issue regarding problems with TF save_model test
• Correct templates: delete unnecessary test for tf bart; add gradient checkpointing by default in PT

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

[patrickvonplaten]

@ibeltagy this line ensures that the whole class has to be exactly the same as the corresponding class in LongformerSelfAttention or else the tests would throw an error. This is our safety check to make sure that the original class (which in other libs would just be imported and re-used here) cannot change without this class being changed accordingly as well.

[jplu]

Awesome work!!! Just left few tiny comments.

[LysandreJik]

Great job implementing this! Happy to hear the model templates made your life easier, too.

Left a few nits, but LGTM!

PS: That 16384 embedding size is incredible!

[sgugger]

Looks great to me! Thanks for adding this new model!

[jplu]

@patrickvonplaten when you have time, can you fix the conflicts and apply the same updates merged in Longformer to LED. Thanks!