metadata

tags:
  - ctranslate2
  - int8
  - float16
license: cc
datasets:
  - VMware/open-instruct-v1-oasst-dolly-hhrlhf
language:
  - en
library_name: transformers
pipeline_tag: text-generation

# Fast-Inference with Ctranslate2

Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.

quantized version of VMware/open-llama-7b-open-instruct

pip install hf-hub-ctranslate2>=2.10.0 ctranslate2>=3.16.0

# from transformers import AutoTokenizer
model_name = "michaelfeil/ct2fast-open-llama-7b-open-instruct"

from hf_hub_ctranslate2 import GeneratorCT2fromHfHub
model = GeneratorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name,
        device="cuda",
        compute_type="int8_float16",
        # tokenizer=AutoTokenizer.from_pretrained("{ORG}/{NAME}")
)
outputs = model.generate(
    text=["def fibonnaci(", "User: How are you doing? Bot:"],
    max_length=64,
    include_prompt_in_result=False
)
print(outputs)

Checkpoint compatible to ctranslate2>=3.16.0 and hf-hub-ctranslate2>=2.10.0

compute_type=int8_float16 for device="cuda"
compute_type=int8 for device="cpu"

Converted on 2023-06-16 using

ct2-transformers-converter --model VMware/open-llama-7b-open-instruct --output_dir ~/tmp-ct2fast-open-llama-7b-open-instruct --force --copy_files README.md tokenizer_config.json generation_config.json special_tokens_map.json .gitattributes --quantization int8_float16 --trust_remote_code

Licence and other remarks:

This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

Original description

VMware/open-llama-7B-open-instruct

Instruction-tuned version of the fully trained Open LLama 7B model. The model is open for COMMERCIAL USE.

NOTE : The model was trained using the Alpaca prompt template NOTE : Fast tokenizer results in incorrect encoding, set the use_fast = False parameter, when instantiating the tokenizer

License

Commercially Viable
Instruction dataset, VMware/open-instruct-v1-oasst-dolly-hhrlhf is under cc-by-sa-3.0
Language Model, (openlm-research/open_llama_7b) is under apache-2.0

Nomenclature

Model : Open-llama
Model Size: 7B parameters
Dataset: Open-instruct-v1 (oasst,dolly, hhrlhf)

Use in Transformers

import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'VMware/open-llama-7b-open-instruct'


tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map='sequential')

prompt_template = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:"

prompt = 'Explain in simple terms how the attention mechanism of a transformer model works'


inputt = prompt_template.format(instruction= prompt)
input_ids = tokenizer(inputt, return_tensors="pt").input_ids.to("cuda")

output1 = model.generate(input_ids, max_length=512)
input_length = input_ids.shape[1]
output1 = output1[:, input_length:]
output = tokenizer.decode(output1[0])

print(output)

'''
 Attention is a mechanism used in deep learning models, such as transformer models, to capture global dependencies between different parts of the input. In a transformer model, the attention mechanism works by computing a weighted sum of the input vectors and then applying a non-linear activation function to the result.

The attention mechanism in a transformer model works in two steps:

1. Query-Key Mapping: First, the input sequence is divided into two parts: the query vector and the key vector. The query vector represents the input at the current position, and the key vector represents the input at a previous position.

2. Attention Weight Calculation: Second, the attention weights are calculated using the dot product between the query vector and each key vector. The attention weights represent the importance of the input at the previous position to the current position.

The attention weights are then used to compute the attention score for each input element. The attention score represents the relevance of the input element to the current position.

The attention mechanism in a transformer model is designed to capture global dependencies between different parts of the input. By attending to input elements from different positions, the model can learn to understand the relationships between different parts of the input. This allows the model to perform more complex tasks, such as understanding the relationships between words in a sentence or pixels in an image.</s>

'''

Finetuning details

The finetuning scripts will be available in our RAIL Github Repository

Evaluation

TODO