Recently, <cite>Yang et al. (2019)</cite> showed that combining a BERT-based reader with passage retrieval using the Anserini IR toolkit yields a large improvement in question answering directly from a Wikipedia corpus, measured in terms of exact match on a standard benchmark (Chen et al., 2017) . Interestingly, the approach of <cite>Yang et al. (2019)</cite> represents a simple method to combining BERT with off-the-shelf IR. In this paper, we build on these initial successes to explore how much further we can push this simple architecture by data augmentation, taking advantage of distant supervision techniques to gather more and higher-quality * equal contribution training data to fine tune BERT.