|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- grammarly/coedit |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
tags: |
|
- torchtune |
|
- grammar-correction |
|
--- |
|
|
|
|
|
### Llama3 CoEdit |
|
|
|
This is a Llama3 8B based model trained using [torchtune](https://pytorch.org/torchtune) on the `grammarly/coedit` dataset. |
|
|
|
### Training details |
|
|
|
The exact training script ([lora_finetune_distributed](./lora_finetune_distributed.py)) and config ([8B_lora.yaml](./8B_lora.yaml)) are both included in this repository. |
|
|
|
**Training command**: ``tune run --nproc_per_node 8 lora_finetune_distributed --config 8B_lora.yaml`` |
|
|
|
> Yes I used 8 GPUs :) |
|
|
|
In order to add the dataset, I added the following lines to the config: |
|
|
|
``` |
|
dataset: |
|
_component_: torchtune.datasets.instruct_dataset |
|
source: grammarly/coedit |
|
template: GrammarErrorCorrectionTemplate |
|
column_map: {"sentence": "src", "output": "tgt"} |
|
train_on_input: False |
|
split: train |
|
``` |
|
|
|
**Loss curve** |
|
|
|
 |
|
|
|
|
|
### Evaluation results |
