Séminaires RALI-OLST

Vector-Space Proximity-Based Document Retrieval For Document Embeddings Built By Transformers

Pavel Khloponin (pioneerappx <at> gmail (point) com)

ClaC, Concordia University

Le mercredi 15 juin 2022 à 11 h 30

Réunion Zoom, ci-dessous

In this presentation, I will explain my approach to the TREC News Track shared task run by NIST. This task is organized in collaboration with the Washington Post which helped to build a test collection of 670K news articles. Given 50 query news articles from the same collection, participants have to select and order 100 most relevant articles (backlinks). The results were then pulled and evaluated by experts, which leads to a rank for each assessed backlink. Each submission is evaluated (with nDCG@5) based on how far it is from the ideal possible ranking. It means participants not only need to find relevant news, but also order them from most to less relevant.

In this work, Okapi BM25 was used as our baseline model. A variety of transformer-based embedding models (19 models, from 5 families) were also used to build embeddings for news articles and a plethora of proximity measures (85 different ones) to retrieve the backlinks. Additional exploration of other hyperparameters led to the evaluation of 47,332 unique configurations of our system. We also explored performance of a combined model where Okapi BM25 and proximity-based models are working together.

Our baseline model got the highest score at the TREC News Track 2020. The performance of the proximity-based approach alone was below the median, but the combined approach showed improvement on the topics that BM25 was struggling with.

Recording: https://drive.google.com/file/d/1uG2fK8zGQvf6_2e42XhMwm04zDmmc8oG/view?usp=sharing

Joignez-vous à nous avec Zoom à l'aide de cette URL.
ID de réunion : 916 9097 5818, Code : 343273.
Numéros de téléphone : https://umontreal.zoom.us/u/abitNZzLg.
One tap mobile : +14388097799,,91690975818#,,,,,,0#,,343273#

Pour recevoir les annonces hebdomadaires par courriel, envoyez un message à l'adresse majordomo@iro.umontreal.ca. Il suffit d'envoyer un message ne contenant que la ligne 'subscribe ralli' (sans les apostrophes, avec un double 'l' dans 'ralli').

Liste de tous les séminaires pour l'année :

1991 1992 1993 1994 1995 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022