Previous approaches (Riedel et al., 2010; Hoffmann et al., 2011;<cite> Surdeanu et al., 2012)</cite> bypassed this problem by heavily under-sampling the "negative" class. We instead deal with a learning scenario where we only have entity-pair level labels that are either positive or unlabeled.