Vector-Space Proximity-Based Document Retrieval For Document Embeddings Built By Transformers
Pavel Khloponin (pioneerappx <at> gmail (point) com)
ClaC, Concordia University
Wednesday 15 June 2022 at 11:30 AM
Réunion Zoom, below
In this presentation, I will explain my approach to the TREC News Track shared task run by NIST. This task is organized in collaboration with the Washington Post which helped to build a test collection of 670K news articles. Given 50 query news articles from the same collection, participants have to select and order 100 most relevant articles (backlinks). The results were then pulled and evaluated by experts, which leads to a rank for each assessed backlink. Each submission is evaluated (with nDCG@5) based on how far it is from the ideal possible ranking. It means participants not only need to find relevant news, but also order them from most to less relevant.
In this work, Okapi BM25 was used as our baseline model. A variety of transformer-based embedding models (19 models, from 5 families) were also used to build embeddings for news articles and a plethora of proximity measures (85 different ones) to retrieve the backlinks. Additional exploration of other hyperparameters led to the evaluation of 47,332 unique configurations of our system. We also explored performance of a combined model where Okapi BM25 and proximity-based models are working together.
Our baseline model got the highest score at the TREC News Track 2020. The performance of the proximity-based approach alone was below the median, but the combined approach showed improvement on the topics that BM25 was struggling with.
To receive weekly talk announcements, please send an e-mail to firstname.lastname@example.org. Simply write a message containing the single line 'subscribe ralli' (without the quotes, with a double 'l' in 'ralli').
See all the weekly talks for the year:1991 1992 1993 1994 1995 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022