Relation classification aims to determine if a sentence containing two entities expresses a relation between them. It is often difficult to have a large amount of labeled data for training. Distant supervision provides a means to create a large number of weakly labeled data at low cost by leveraging triplets in knowledge bases. This talk will explore how to make full use of such noisy labeled data for relation classification.

Many existing approaches try to select reliable instances for training from noisy data by heuristic rules or by making strong assumptions about the data. However, the selected instances may be wrong. In this talk, we consider the scenario where a small set of clean labeled data is available along with a large set of noisy labeled data. The clean data is leveraged to guide the selection of noisy data. We propose a meta-learning based reweighting approach to assign weights to noisy instances depending on how the model built on them can correctly classify the clean data. Experimental results on a public dataset on relation classification show that our approach achieves significant improvements over strong baselines.