Transformers, while powerful, struggle with longer sequences due to their quadratic complexity. This talk will address this challenge by introducing recent novel methods for length extrapolation using positional encoding. Specifically, we'll explore two low-cost strategies based on rotary position encoding. Additionally, a brief overview of other recent extrapolation techniques and their limitations will be presented. The session will conclude with insights into current challenges and potential directions for future work. This seminar aims to provide attendees with practical solutions for extending Transformer's reach without extensive training overhead.

Recording: https://drive.google.com/file/d/1Uw3JI8W229FuVfOZWlYJiNuys_MwZIbX/view?usp=drive_link