We will present mycoSORT, a supervised machine learning based system to support in an automatic manner the first task of the biomedical literature manual curation process, called triage. A representative dataset of the literature triage is commonly large and highly imbalanced, since when curators are screening for relevant research papers, usually the majority of documents is rejected and only a few documents are kept for further analysis.

To tackle the large dataset size and imbalance class distribution issues, we studied the performance of 108 classification models. The results show that the best approach relies on a classification model composed by domain annotations, a balanced dataset (similar class distributions), and the use of a Logistic Model Trees classifier. The system is publicly released as an open source toolkit.