
# GGML converted versions of [BigScience](https://huggingface.co/bigscience)'s Bloom models

## Description

BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks.


## Converted Models
| Name                                                                                                         | Based on                                                              | Type   | Container   | GGML Version   |
|:-------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------|:-------|:------------|:---------------|
| [bloom-1b1-f16.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b1-f16.bin)               | [bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)   | F16    | GGML        | V3             |
| [bloom-1b1-q4_0.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b1-q4_0.bin)             | [bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)   | Q4_0   | GGML        | V3             |
| [bloom-1b1-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b1-q4_0-ggjt.bin)   | [bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)   | Q4_0   | GGJT        | V3             |
| [bloom-1b1-q5_1.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b1-q5_1.bin)             | [bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)   | Q5_1   | GGML        | V3             |
| [bloom-1b1-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b1-q5_1-ggjt.bin)   | [bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)   | Q5_1   | GGJT        | V3             |
| [bloom-1b7-f16.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b7-f16.bin)               | [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)   | F16    | GGML        | V3             |
| [bloom-1b7-q4_0.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b7-q4_0.bin)             | [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)   | Q4_0   | GGML        | V3             |
| [bloom-1b7-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b7-q4_0-ggjt.bin)   | [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)   | Q4_0   | GGJT        | V3             |
| [bloom-1b7-q5_1.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b7-q5_1.bin)             | [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)   | Q5_1   | GGML        | V3             |
| [bloom-1b7-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-1b7-q5_1-ggjt.bin)   | [bigscience/bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)   | Q5_1   | GGJT        | V3             |
| [bloom-3b-f16.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-3b-f16.bin)                 | [bigscience/bloom-3b](https://huggingface.co/bigscience/bloom-3b)     | F16    | GGML        | V3             |
| [bloom-3b-q4_0.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-3b-q4_0.bin)               | [bigscience/bloom-3b](https://huggingface.co/bigscience/bloom-3b)     | Q4_0   | GGML        | V3             |
| [bloom-3b-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-3b-q4_0-ggjt.bin)     | [bigscience/bloom-3b](https://huggingface.co/bigscience/bloom-3b)     | Q4_0   | GGJT        | V3             |
| [bloom-3b-q5_1.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-3b-q5_1.bin)               | [bigscience/bloom-3b](https://huggingface.co/bigscience/bloom-3b)     | Q5_1   | GGML        | V3             |
| [bloom-3b-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-3b-q5_1-ggjt.bin)     | [bigscience/bloom-3b](https://huggingface.co/bigscience/bloom-3b)     | Q5_1   | GGJT        | V3             |
| [bloom-560m-f16.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-560m-f16.bin)             | [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | F16    | GGML        | V3             |
| [bloom-560m-q4_0.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-560m-q4_0.bin)           | [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | Q4_0   | GGML        | V3             |
| [bloom-560m-q4_0-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-560m-q4_0-ggjt.bin) | [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | Q4_0   | GGJT        | V3             |
| [bloom-560m-q5_1.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-560m-q5_1.bin)           | [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | Q5_1   | GGML        | V3             |
| [bloom-560m-q5_1-ggjt.bin](https://huggingface.co/rustformers/bloom-ggml/blob/main/bloom-560m-q5_1-ggjt.bin) | [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | Q5_1   | GGJT        | V3             |

## Usage

### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python):

#### Installation
Via pip: `pip install llm-rs`

#### Run inference
```python
from llm_rs import AutoModel

#Load the model, define any model you like from the list above as the `model_file`
model = AutoModel.from_pretrained("rustformers/bloom-ggml",model_file="bloom-3b-q4_0-ggjt.bin")

#Generate
print(model.generate("The meaning of life is"))
```

### Rust via [Rustformers/llm](https://github.com/rustformers/llm): 

#### Installation
```
git clone --recurse-submodules https://github.com/rustformers/llm.git
cd llm
cargo build --release
```

#### Run inference
```
cargo run --release -- bloom infer -m path/to/model.bin  -p "Tell me how cool the Rust programming language is:"
```
{"language": ["ak", "ar", "as", "bm", "bn", "ca", "code", "en", "es", "eu", "fon", "fr", "gu", "hi", "id", "ig", "ki", "kn", "lg", "ln", "ml", "mr", "ne", "nso", "ny", "or", "pa", "pt", "rn", "rw", "sn", "st", "sw", "ta", "te", "tn", "ts", "tum", "tw", "ur", "vi", "wo", "xh", "yo", "zh", "zu"], "license": "bigscience-bloom-rail-1.0", "tags": ["llm-rs", "ggml"], "programming_language": ["C", "C++", "C#", "Go", "Java", "JavaScript", "Lua", "PHP", "Python", "Ruby", "Rust", "Scala", "TypeScript"], "pipeline_tag": "text-generation"}
transformers llm-rs ggml text-generation ak ar as bm bn ca code en es eu fon fr gu hi id ig ki kn lg ln ml mr ne nso ny or pa pt rn rw sn st sw ta te tn ts tum tw ur vi wo xh yo zh zu license:bigscience-bloom-rail-1.0 endpoints_compatible region:us
TAGS
#transformers #llm-rs #ggml #text-generation #ak #ar #as #bm #bn #ca #code #en #es #eu #fon #fr #gu #hi #id #ig #ki #kn #lg #ln #ml #mr #ne #nso #ny #or #pa #pt #rn #rw #sn #st #sw #ta #te #tn #ts #tum #tw #ur #vi #wo #xh #yo #zh #zu #license-bigscience-bloom-rail-1.0 #endpoints_compatible #region-us 

GGML converted versions of BigScience's Bloom models
====================================================


Description
-----------


BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks.


Converted Models
----------------



Usage
-----


### Python via llm-rs:


#### Installation


Via pip: 'pip install llm-rs'


#### Run inference


### Rust via Rustformers/llm:


#### Installation


#### Run inference



### Python via llm-rs: #### Installation


Via pip: 'pip install llm-rs' #### Run inference ### Rust via Rustformers/llm: #### Installation #### Run inference
TAGS
#transformers #llm-rs #ggml #text-generation #ak #ar #as #bm #bn #ca #code #en #es #eu #fon #fr #gu #hi #id #ig #ki #kn #lg #ln #ml #mr #ne #nso #ny #or #pa #pt #rn #rw #sn #st #sw #ta #te #tn #ts #tum #tw #ur #vi #wo #xh #yo #zh #zu #license-bigscience-bloom-rail-1.0 #endpoints_compatible #region-us 
 ### Python via llm-rs: #### Installation


Via pip: 'pip install llm-rs' #### Run inference ### Rust via Rustformers/llm: #### Installation #### Run inference