Update README.md

README.md

@@ -108,6 +108,33 @@ You can use the following script to reproduce our results and evaluate any other
    bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}
    ```
 
+## FAQ on Continual Pre-Training from LLama3
+**Q1: Do you use the official Llama3 instruction prompt for pre-training?**
+
+No, the provided Llama3 instruction prompt is designed for the [instruction-tuned model](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), but our continual pre-training is conducted on the [pre-trained base model](https://huggingface.co/meta-llama/Meta-Llama-3-8B) where only BOS (`<|begin_of_text|>`) and EOS (`<|end_of_text|>`) tokens are required.
+
+**Q2: For the general instructions from OpenOrca, do you concatenate each instruction with its output using '\n'?**
+
+No, as mentioned in the pre-training suggestions, we use a simple whitespace to concatenate each question with its response for the general instruction data from OpenOrca. This is because OpenOrca's data is already templated with diverse natural languge templates (such as those with `\n`), so a whitespace is sufficient to formulate the data.
+
+Note that when using our templated instruction-augmented texts, you don't need to add any concatenations.
+
+**Q3: What about those system prompts in OpenOrca?**
+
+We simply discard the system prompts.
+
+**To put it all together, the text before tokenization looks like this:**
+
+`general_instruction_response_text = "<|begin_of_text|>{question} {response}<|end_of_text|>"`
+
+or
+
+`instruction_augmented_text = "<|begin_of_text|>{instruction augmented text}<|end_of_text|>"`
+
+Then, for tokenization, you don't need to add BOS and EOS token ids. The tokenization code looks like this:
+
+`text_ids = tokenizer(text, add_special_tokens=False, **kwargs).input_ids`
+
 
 ## Citation
 If you find our work helpful, please cite us: