Knowledge is a, if not the, pivotal component of human intelligence, accumulated over our extensive history. Large language models like GPT-4 are celebrated not just for their sophisticated logic but also for their superior knowledge base. Despite rumors of GPT-4 having over 1T parameters, _is it necessary to store all human knowledge?_ Could a 10B model, if trained sufficiently with high-quality data, match GPT-4's knowledge capacity? Our paper seeks to address these questions.