region:us
---
dataset_info:
  features:
  - name: prompt
    dtype: string
  - name: completion
    dtype: string
  - name: score
    dtype: int64
  splits:
  - name: before_cleaning
    num_bytes: 58053871
    num_examples: 71968
  - name: round_0
    num_bytes: 58053871
    num_examples: 71968
  - name: round_1
    num_bytes: 58053871
    num_examples: 71968
  - name: round_2
    num_bytes: 58053871
    num_examples: 71968
  - name: round_3
    num_bytes: 58053871
    num_examples: 71968
  download_size: 144722712
  dataset_size: 290269355
---
# Test instruction backtranslation

This is the dataset I obtained by applying [instruction backtranslation](https://github.com/bigcode-project/bigcode-finetuning/tree/main/instruction_backtranslation) (just the `self-curation part`, no `self-augmentation`).
The model used for curation is starcoder, fine-tuned on OpenAssistant-guanaco. Here is the command :

```
python3 -u -m torch.distributed.run main.py
  --model_name_or_path=bigcode/starcoder
  --dataset_name_or_path=ArmelR/oasst1_guanaco
  --shuffle_buffer 100
  --seq_length 2048
  --max_steps 160
  --batch_size 1
  --dialogue_template_name standard
  --input_column_name=prompt
  --output_column_name=completion
  --num_workers 1
  --gradient_accumulation_steps 4
  --learning_rate 5e-5
  --lr_scheduler_type=cosine
  --log_freq 5
  --eval_freq 10
  --num_warmup_steps 10
  --save_freq 20
  --weight_decay 0.05
  --save_total_limit 3
  --output_dir=./checkpoints
  --synthetic_data_path=./data/nuprl_cleaned.jsonl
  --dataset_text_field=content
  --request_batch_size 16
  --max_new_tokens 512
  --temperature 0.7
  --top_p 0.9
  --num_beams 1
  --repetition_penalty 1.2
  --number_of_rounds 5
  --max_samples 157767
  --curation_model_name_or_path bigcode/starcoder
  --do_self_curation
```