Validation Curves for SEA and Performer on OpenWebText Dataset (Figure A.5 in Appendix A.9)

[JinA0218]

This PR implements a script that generates validation curves comparing SEA and Performer models when trained on the OpenWebText dataset (Figure A.5 in Appendix A.9). The script evaluates and visualizes the convergence performance of both models, showcasing SEA's significantly faster convergence compared to Performer.

Key Features

• Validation Curve Visualization:

  • Plots Perplexity (PPL) scores as a function of optimizer steps for SEA and Performer models.
  • Includes a comparison of validation curves highlighting SEA's faster convergence.

• Dataset: OpenWebText dataset