# Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.



## Prompting

For instructions that **do not** require extraneous inputs, the recommended prompt is:

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here --- >

### Response:
```

For instructions that **do** require extraneous inputs, the recommended prompt is:

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here -- >

### Input:

< -- extraneous input goes here -- >

### Response:
```

Since the model performs [causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling), the model's response to the prompt is the text completing the sequence beginning with the prompt.


## Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

- **Epochs**: 3
- **Batch size**: 128
- **Cutoff length**: 2048
- **Learning rate**: 3.2e-5, cosine decay
- **LoRA _r_**: 8
- **LoRA _alpha_**: 16
- **LoRA _dropout_**: 0.0
- **LoRA target modules**: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- **License**: The instruction-tuning data is subject to the [Creative Commons 4.0](https://creativecommons.org/licenses/by/4.0/) license.

## Base Model

This model was instruction-tuned from a 12B variant from the Pythia family. 

- **Repository:** [EleutherAI/pythia-12b](https://huggingface.co/EleutherAI/pythia-12b)
- **Paper:** [arxiv:2304.01373](https://arxiv.org/abs/2304.01373)
- **License**: The base model is subject to the Apache 2.0 license.
- **Model type**: Transformer-based Language Model

## Licensing Information

We release this adapter under the [Creative Commons NonCommercial (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license.
{"license": "cc-by-4.0"}
transformers gpt_neox text-generation arxiv:2304.01373 license:cc-by-4.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #gpt_neox #text-generation #arxiv-2304.01373 #license-cc-by-4.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

# Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model.



## Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:



For instructions that do require extraneous inputs, the recommended prompt is:



Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.


## Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

- Epochs: 3
- Batch size: 128
- Cutoff length: 2048
- Learning rate: 3.2e-5, cosine decay
- LoRA _r_: 8
- LoRA _alpha_: 16
- LoRA _dropout_: 0.0
- LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- License: The instruction-tuning data is subject to the Creative Commons 4.0 license.

## Base Model

This model was instruction-tuned from a 12B variant from the Pythia family. 

- Repository: EleutherAI/pythia-12b
- Paper: arxiv:2304.01373
- License: The base model is subject to the Apache 2.0 license.
- Model type: Transformer-based Language Model

## Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.
# Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model. ## Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:



For instructions that do require extraneous inputs, the recommended prompt is:



Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt. ## Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

- Epochs: 3
- Batch size: 128
- Cutoff length: 2048
- Learning rate: 3.2e-5, cosine decay
- LoRA _r_: 8
- LoRA _alpha_: 16
- LoRA _dropout_: 0.0
- LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- License: The instruction-tuning data is subject to the Creative Commons 4.0 license. ## Base Model

This model was instruction-tuned from a 12B variant from the Pythia family. 

- Repository: EleutherAI/pythia-12b
- Paper: arxiv:2304.01373
- License: The base model is subject to the Apache 2.0 license.
- Model type: Transformer-based Language Model ## Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.
TAGS
#transformers #gpt_neox #text-generation #arxiv-2304.01373 #license-cc-by-4.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 # Model Card

We provide a low-rank adapter for an instruction-tuned 12B-parameter GPT3-style language model. ## Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:



For instructions that do require extraneous inputs, the recommended prompt is:



Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt. ## Instruction-Tuning

This model was instruction-tuned on an internally-curated Alpaca-style dataset.

- Epochs: 3
- Batch size: 128
- Cutoff length: 2048
- Learning rate: 3.2e-5, cosine decay
- LoRA _r_: 8
- LoRA _alpha_: 16
- LoRA _dropout_: 0.0
- LoRA target modules: 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'
- License: The instruction-tuning data is subject to the Creative Commons 4.0 license. ## Base Model

This model was instruction-tuned from a 12B variant from the Pythia family. 

- Repository: EleutherAI/pythia-12b
- Paper: arxiv:2304.01373
- License: The base model is subject to the Apache 2.0 license.
- Model type: Transformer-based Language Model ## Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.