Séminaires RALI-OLST

Length Extrapolation by Manipulating Transformer’s Positional Encoding

Suyuchen Wang (suyuchen (point) wang <at> umontreal (point) ca)

RALI, DIRO

Wednesday 11 October 2023 at 11:30 AM

Salle 3195, Pavillon André-Aisenstadt — Diffusion simultanée sur Zoom


Transformers, while powerful, struggle with longer sequences due to their quadratic complexity. This talk will address this challenge by introducing recent novel methods for length extrapolation using positional encoding. Specifically, we'll explore two low-cost strategies based on rotary position encoding. Additionally, a brief overview of other recent extrapolation techniques and their limitations will be presented. The session will conclude with insights into current challenges and potential directions for future work. This seminar aims to provide attendees with practical solutions for extending Transformer's reach without extensive training overhead.

Recording: https://drive.google.com/file/d/1Uw3JI8W229FuVfOZWlYJiNuys_MwZIbX/view?usp=drive_link



Join with Zoom at this url.
Meeting ID: 916 9097 5818, 𝑷𝒂𝒔𝒔𝒄𝒐𝒅𝒆: 343273.
Phone numbers: https://umontreal.zoom.us/u/abitNZzLg.
One tap mobile: +14388097799,,91690975818#,,,,,,0#,,343273#


Follow this link to subscribe to future RALI-OLST announcements.
http://rali.iro.umontreal.ca/rali/?q=fr/node/1631

See all the weekly talks for the year:

1991 1992 1993 1994 1995 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024