More similar to our work, Mallen, Alex Troy and Asai, Akari and Zhong, Victor and Das, Rajarshi and Khashabi, Daniel and Hajishirzi, Hannaneh [22] propose a dataset and method for measuring when non-parametric information needs to be retrieved. They present the PopQA dataset that contains 14K questions about a set of entities with varying popularity. The popularity of an entity is measured by the page views of its Wikipedia page. In order to solve this QA task, they use a popularity score threshold calculated on the PopQA dataset. If the popularity score of an individual entity is below the threshold they perform a retrieval step. On the contrary, if the score is greater than the threshold they directly answer the question. This method yields better results than vanilla retrieval but it requires the calculation of a popularity score that is not available in realistic QA scenarios.