Length Extrapolation by Manipulating Transformer’s Positional Encoding

Suyuchen Wang (suyuchen (point) wang <at> umontreal (point) ca)

RALI, DIRO

Le 11 octobre 2023 à 11 h 30

Salle 3195, Pavillon André-Aisenstadt — Diffusion simultanée sur Zoom


Transformers, while powerful, struggle with longer sequences due to their quadratic complexity. This talk will address this challenge by introducing recent novel methods for length extrapolation using positional encoding. Specifically, we'll explore two low-cost strategies based on rotary position encoding. Additionally, a brief overview of other recent extrapolation techniques and their limitations will be presented. The session will conclude with insights into current challenges and potential directions for future work. This seminar aims to provide attendees with practical solutions for extending Transformer's reach without extensive training overhead.

Recording: https://drive.google.com/file/d/1Uw3JI8W229FuVfOZWlYJiNuys_MwZIbX/view?usp=drive_link


Pour recevoir les annonces hebdomadaires par courriel, visitez http://rali.iro.umontreal.ca/rali/?q=fr/node/1631

Liste des autres séminaires