Most large language models (LLMs)  are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality ofLLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduceFreshQA, a novel dynamicQAbenchmark encompassing a diverse range of question and answer types, including questions that requirefast-changingworld knowledge as well as questions withfalse premisesthat need to be debunked. We benchmark a diverse array of both closed and open-sourceLLMsunder a two-mode evaluation procedure that allows us to measure both correctness and hallucination.
Through human evaluations involving more than50Kjudgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size)  struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we presentFreshPrompt, a simple few-shot prompting method that substantially boosts the performance of anLLMonFreshQAby incorporating relevant and up-to-date information retrieved from a search engine into the prompt.
Our experiments show thatFreshPromptoutperforms both competing search engine-augmented prompting methods such asSelf-Askas well as commercial systems such asPerplexity.AI.111https://www.perplexity.aiFurther analysis ofFreshPromptreveals that both the number of retrieved evidences and their order play a key role in influencing the correctness ofLLM-generated answers. Additionally, instructing theLLMto generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we releaseFreshQAatgithub.com/freshllms/freshqaand commit to updating it at regular intervals.