|
--- |
|
language: |
|
- en |
|
license: mit |
|
library_name: transformers |
|
tags: |
|
- batch prompting |
|
- batch |
|
- BatchPrompt |
|
- BatchPrompting |
|
- GLUE |
|
- Llama |
|
- fine-tuned |
|
- Llama3 |
|
- Llama-3-8B-Instruct |
|
datasets: |
|
- achandlr/BatchPrompting |
|
metrics: |
|
- accuracy |
|
pipeline_tag: question-answering |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
This model is a fine-tuned version of Llama-3-8B-Instruct on the BatchPrompting dataset, which spans 13 diverse NLP tasks. The model has been fine-tuned to effectively perform batch prompting - answering multiple questions concatenated into a single prompt in one inference pass. |
|
|
|
## Model Details |
|
|
|
This model is a fine-tuned version of Llama-3-8B-Instruct on the BatchPrompting dataset, which spans 13 diverse NLP tasks. The model has been fine-tuned to effectively perform batch prompting - answering multiple questions concatenated into a single prompt in one inference pass. |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. TODO--> |
|
|
|
|
|
|
|
- **Developed by:** Alex Chandler, Sebastian Joseph |
|
- **Model type:** Large Language Model (Llama-3 variant |
|
- **Language(s) (NLP):** English |
|
- **License:** MIT |
|
- **Finetuned from model [optional]:** Llama-3-8B-Instruct |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** Forthcoming |
|
- **Paper:** Forthcoming |
|
- **Demo:** Forthcoming |
|
|
|
## Uses |
|
|
|
### How to Use |
|
Use with transformers |
|
See the snippet below for usage with Transformers: |
|
```python |
|
import transformers |
|
import torch |
|
|
|
model_id = "achandlr/Llama-3-8B-Instruct-BatchPromptQA" |
|
|
|
# Load the model pipeline |
|
pipeline = transformers.pipeline("text-generation", model=model_id) |
|
|
|
# Generate text using the pipeline |
|
generated_text = pipeline("Hey how are you doing today?") |
|
print(generated_text) |
|
``` |
|
|
|
|
|
### Direct Use |
|
|
|
The model can be used for efficient question-answering on a variety of NLP tasks by concatenating multiple questions into a single prompt. It demonstrates strong generalization to unseen tasks and maintains performance with larger batch sizes compared to the non-fine-tuned model. |
|
|
|
### Out-of-Scope Use |
|
|
|
The model should not be used for tasks that may cause harm or for generating factually incorrect or biased content. Caution should be exercised if using the model for high-stakes decision making. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
The model may exhibit biases present in its pretraining data or the BatchPrompting dataset. It has not been extensively tested for fairness or potential misuse. Performance may degrade on out-of-distribution examples or tasks very dissimilar to the training data. |
|
|
|
### Recommendations |
|
|
|
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
|
|
Users should be made aware of the model's potential limitations and biases. The model's outputs should be carefully monitored, especially when used for sensitive applications. More testing is needed to fully characterize its capabilities and shortcomings. |
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
[More Information Needed] |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The model was fine-tuned on our BatchPrompting dataset consisting of 13 NLP tasks: |
|
- **GLUE Benchmark Tasks**: A collection of datasets used for evaluating the performance of models on a variety of natural language understanding tasks. |
|
- **Mathematical Reasoning Datasets**: |
|
- **GSM8K**: Focuses on numerical and logical reasoning challenges. |
|
- **GSM8K-Hard**: Contains more complex problems from the GSM8K dataset. |
|
- **CommonsenseQA**: Tests the model's commonsense reasoning ability through multiple-choice question answering. |
|
- **RACE Reading Comprehension Dataset**: Consists of passages and questions designed to assess reading comprehension, derived from English exams. |
|
|
|
|
|
### Training Procedure |
|
|
|
The model was fine-tuned using the LoRA method. |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** Forthcoming <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
|
|
|
#### Speeds, Sizes, Times [optional] |
|
|
|
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
|
Forthcoming |
|
|
|
[More Information Needed] |
|
|
|
## Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
Testing Data, Factors & Metrics |
|
Evaluation was performed on tasks that were excluded from the training run. Key metrics included accuracy and BatchPrompt error rate (failure to answer a question or conform to the specified format). |
|
A table of our results is forthcoming. |
|
|
|
### Testing Data, Factors & Metrics |
|
Forthcoming |
|
|
|
#### Testing Data |
|
Forthcoming |
|
<!-- This should link to a Dataset Card if possible. --> |
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Metrics |
|
Forthcoming |
|
<!-- These are the evaluation metrics being used, ideally with a description of why. --> |
|
|
|
[More Information Needed] |
|
|
|
### Results |
|
Forthcoming |
|
[More Information Needed] |
|
|
|
#### Summary |
|
Forthcoming |
|
|
|
|
|
## Model Examination [optional] |
|
Forthcoming |
|
<!-- Relevant interpretability work for the model goes here --> |
|
|
|
[More Information Needed] |
|
|
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
## Environmental Impact |
|
- **Hardware Type:** [More Information Needed] |
|
- **Hours used:** [More Information Needed] |
|
- **Cloud Provider:** [More Information Needed] |
|
- **Compute Region:** [More Information Needed] |
|
- **Carbon Emitted:** [More Information Needed] |
|
|
|
--> |
|
|
|
|
|
### Model Architecture and Objective |
|
|
|
[More Information Needed] |
|
|
|
### Compute Infrastructure |
|
|
|
[More Information Needed] |
|
|
|
#### Hardware |
|
|
|
[More Information Needed] |
|
|
|
#### Software |
|
|
|
[More Information Needed] |
|
|
|
## Citation |
|
|
|
Forthcoming |
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
Forthcoming |
|
|
|
**APA:** |
|
|
|
[More Information Needed] |
|
|
|
## Glossary [optional] |
|
|
|
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. --> |
|
|
|
[More Information Needed] |
|
|
|
## More Information [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Authors [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Contact |
|
|
|
[More Information Needed] |
