
# GGML converted versions of [BigScience](https://huggingface.co/bigscience)'s BloomZ models

## Description

> We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find the resulting models capable of crosslingual generalization to unseen tasks & languages.

- **Repository:** [bigscience-workshop/xmtf](https://github.com/bigscience-workshop/xmtf)
- **Paper:** [Crosslingual Generalization through Multitask Finetuning](https://arxiv.org/abs/2211.01786)
- **Point of Contact:** [Niklas Muennighoff](mailto:niklas@hf.co)
- **Languages:** Refer to [bloom](https://huggingface.co/bigscience/bloom) for pretraining & [xP3](https://huggingface.co/datasets/bigscience/xP3) for finetuning language proportions. It understands both pretraining & finetuning languages.

### Intended use

We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper: 
- 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
- Suggest at least five related search terms to "Mạng neural nhân tạo".
- Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
- Explain in a sentence in Telugu what is backpropagation in neural networks.

## Converted Models
| Name                                                                                                            | Based on                                                                | Type   | Container   | GGML Version   |
|:----------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:-------|:------------|:---------------|
| [bloomz-1b1-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-f16.bin)               | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | F16    | GGML        | V3             |
| [bloomz-1b1-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0.bin)             | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGML        | V3             |
| [bloomz-1b1-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q4_0-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q4_0   | GGJT        | V3             |
| [bloomz-1b1-q5_1.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q5_1.bin)             | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q5_1   | GGML        | V3             |
| [bloomz-1b1-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b1-q5_1-ggjt.bin)   | [bigscience/bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)   | Q5_1   | GGJT        | V3             |
| [bloomz-1b7-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-f16.bin)               | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | F16    | GGML        | V3             |
| [bloomz-1b7-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0.bin)             | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGML        | V3             |
| [bloomz-1b7-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q4_0-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q4_0   | GGJT        | V3             |
| [bloomz-1b7-q5_1.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q5_1.bin)             | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q5_1   | GGML        | V3             |
| [bloomz-1b7-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-1b7-q5_1-ggjt.bin)   | [bigscience/bloomz-1b7](https://huggingface.co/bigscience/bloomz-1b7)   | Q5_1   | GGJT        | V3             |
| [bloomz-3b-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-f16.bin)                 | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | F16    | GGML        | V3             |
| [bloomz-3b-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0.bin)               | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGML        | V3             |
| [bloomz-3b-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q4_0-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q4_0   | GGJT        | V3             |
| [bloomz-3b-q5_1.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q5_1.bin)               | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q5_1   | GGML        | V3             |
| [bloomz-3b-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-3b-q5_1-ggjt.bin)     | [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b)     | Q5_1   | GGJT        | V3             |
| [bloomz-560m-f16.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-f16.bin)             | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | F16    | GGML        | V3             |
| [bloomz-560m-q4_0.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0.bin)           | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGML        | V3             |
| [bloomz-560m-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q4_0-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q4_0   | GGJT        | V3             |
| [bloomz-560m-q5_1.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q5_1.bin)           | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q5_1   | GGML        | V3             |
| [bloomz-560m-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloomz-ggml/blob/main/bloomz-560m-q5_1-ggjt.bin) | [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) | Q5_1   | GGJT        | V3             |

## Usage

### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python):

#### Installation
Via pip: `pip install llm-rs`

#### Run inference
```python
from llm_rs import AutoModel

#Load the model, define any model you like from the list above as the `model_file`
model = AutoModel.from_pretrained("rustformers/bloomz-ggml",model_file="bloomz-3b-q4_0-ggjt.bin")

#Generate
print(model.generate("The meaning of life is"))
```

### Rust via [Rustformers/llm](https://github.com/rustformers/llm): 

#### Installation
```
git clone --recurse-submodules https://github.com/rustformers/llm.git
cd llm
cargo build --release
```

#### Run inference
```
cargo run --release -- bloom infer -m path/to/model.bin  -p "Tell me how cool the Rust programming language is:"
```
{"language": ["ak", "ar", "as", "bm", "bn", "ca", "code", "en", "es", "eu", "fon", "fr", "gu", "hi", "id", "ig", "ki", "kn", "lg", "ln", "ml", "mr", "ne", "nso", "ny", "or", "pa", "pt", "rn", "rw", "sn", "st", "sw", "ta", "te", "tn", "ts", "tum", "tw", "ur", "vi", "wo", "xh", "yo", "zh", "zu"], "license": "bigscience-bloom-rail-1.0", "tags": ["llm-rs", "ggml"], "datasets": ["bigscience/xP3"], "programming_language": ["C", "C++", "C#", "Go", "Java", "JavaScript", "Lua", "PHP", "Python", "Ruby", "Rust", "Scala", "TypeScript"], "pipeline_tag": "text-generation"}
transformers llm-rs ggml text-generation ak ar as bm bn ca code en es eu fon fr gu hi id ig ki kn lg ln ml mr ne nso ny or pa pt rn rw sn st sw ta te tn ts tum tw ur vi wo xh yo zh zu dataset:bigscience/xP3 arxiv:2211.01786 license:bigscience-bloom-rail-1.0 endpoints_compatible region:us
TAGS
#transformers #llm-rs #ggml #text-generation #ak #ar #as #bm #bn #ca #code #en #es #eu #fon #fr #gu #hi #id #ig #ki #kn #lg #ln #ml #mr #ne #nso #ny #or #pa #pt #rn #rw #sn #st #sw #ta #te #tn #ts #tum #tw #ur #vi #wo #xh #yo #zh #zu #dataset-bigscience/xP3 #arxiv-2211.01786 #license-bigscience-bloom-rail-1.0 #endpoints_compatible #region-us 

GGML converted versions of BigScience's BloomZ models
=====================================================


Description
-----------



> 
> We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find the resulting models capable of crosslingual generalization to unseen tasks & languages.
> 
> 
> 


* Repository: bigscience-workshop/xmtf
* Paper: Crosslingual Generalization through Multitask Finetuning
* Point of Contact: Niklas Muennighoff
* Languages: Refer to bloom for pretraining & xP3 for finetuning language proportions. It understands both pretraining & finetuning languages.


### Intended use


We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper:


* 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
* Suggest at least five related search terms to "Mạng neural nhân tạo".
* Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
* Explain in a sentence in Telugu what is backpropagation in neural networks.


Converted Models
----------------



Usage
-----


### Python via llm-rs:


#### Installation


Via pip: 'pip install llm-rs'


#### Run inference


### Rust via Rustformers/llm:


#### Installation


#### Run inference



### Intended use


We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper:


* 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
* Suggest at least five related search terms to "Mạng neural nhân tạo".
* Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
* Explain in a sentence in Telugu what is backpropagation in neural networks.


Converted Models
----------------



Usage
----- ### Python via llm-rs: #### Installation


Via pip: 'pip install llm-rs' #### Run inference ### Rust via Rustformers/llm: #### Installation #### Run inference
TAGS
#transformers #llm-rs #ggml #text-generation #ak #ar #as #bm #bn #ca #code #en #es #eu #fon #fr #gu #hi #id #ig #ki #kn #lg #ln #ml #mr #ne #nso #ny #or #pa #pt #rn #rw #sn #st #sw #ta #te #tn #ts #tum #tw #ur #vi #wo #xh #yo #zh #zu #dataset-bigscience/xP3 #arxiv-2211.01786 #license-bigscience-bloom-rail-1.0 #endpoints_compatible #region-us 
 ### Intended use


We recommend using the model to perform tasks expressed in natural language. For example, given the prompt "*Translate to English: Je t’aime.*", the model will most likely answer "*I love you.*". Some prompt ideas from our paper:


* 一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。你认为这句话的立场是赞扬、中立还是批评?
* Suggest at least five related search terms to "Mạng neural nhân tạo".
* Write a fairy tale about a troll saving a princess from a dangerous dragon. The fairy tale is a masterpiece that has achieved praise worldwide and its moral is "Heroes Come in All Shapes and Sizes". Story (in Spanish):
* Explain in a sentence in Telugu what is backpropagation in neural networks.


Converted Models
----------------



Usage
----- ### Python via llm-rs: #### Installation


Via pip: 'pip install llm-rs' #### Run inference ### Rust via Rustformers/llm: #### Installation #### Run inference