MMLU BenchmarkTo properly evaluate the capabilities of LLMs on knowledge-intensive tasks, we selected four distinct tasks from the Massively Multilingual Language Understanding Evaluation (MMLU) benchmark (Hendrycks et al., 2021) in the topics of anatomy, astronomy, college biology, college chemistry and prehistory. The chosen tasks were selected based on their emphasis on factual knowledge and the minimal reliance on reasoning. As a heuristic, we opted for tasks where the questions are short and involve no context. In practice we selected four STEM subjects as well as one humanities subject, to ensure the evaluation is not limited to certain fields. Note that prehistory involves questions spanning all non-modern history. This approach aims to enable us to test LLM proficiency in comprehending and manipulating information in isolation from its reasoning processes.