ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Abstract
In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge the gap between open-access LLMs and leading proprietary models (e.g., GPT-4-Turbo) in long-context understanding and retrieval-augmented generation (RAG) capabilities. These two capabilities are essential for LLMs to process large volumes of information that cannot fit into a single prompt and are complementary to each other, depending on the downstream tasks and computational budgets. We present a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model achieves accuracy comparable to GPT-4-Turbo-2024-0409 on many long-context understanding tasks and surpasses it on the RAG benchmark. Interestingly, we find that the state-of-the-art long-context retriever can alleviate the top-k context fragmentation issue in RAG, further improving RAG-based results for long-context understanding tasks. We also provide extensive comparisons between RAG and long-context solutions using state-of-the-art long-context LLMs.
Community
Interesting, Where is the model?
Great!
Did they shared the model & training data?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (2024)
- LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models (2024)
- LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs (2024)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend