We extend the learning-to-rank techniques (Liu, 2011) to distant supervision setting to create a robust passage retrieval system. While relation extraction systems exploit rich and complex features that are necessary to extract the exact relation (Mintz et al., 2009; Riedel et al., 2010; <cite>Hoffmann et al., 2011</cite>) , passage retrieval components use coarse features in order to provide different and complementary feedback to information extraction models.