Deeokay/mistral-7b-v0.3-custom-tokens-GGUF

Uploaded model

• Developed by: Deeokay
• License: apache-2.0
• Finetuned from model : unsloth/mistral-7b-v0.3-bnb-4bit

This mistral model was trained 2x faster with Unsloth and Huggingface's TRL library.

README

This is a test model on a the following

• a private dataset
• slight customization on alpaca chat template
• Works with Ollama create but requires customization to Modelfile
• One reason for this was wanted to try doing Q2_K and see if it was actually good(?) -> Exceeds Expectation!!
• My examples will be based on unslot.Q2_K.GGUF file, however other quantization should work as well

HOW TO USE

The whole point of conversion for me was I wanted to be able to to use it through Ollama or (other local options) For Ollama, it required to be a GGUF file. Once you have this it is pretty straight forward

If you want to try it first, the Q2_K version of this is available in Ollama => deeokay/minimistral

ollama pull deeokay/minimistral

Quick Start:

• You must already have Ollama running in your setting
• Download the unsloth.Q2_K.gguf model from Files
• In the same directory create a file call "Modelfile"
• Inside the "Modelfile" type

FROM ./mistrial_unsloth.Q2_K.gguf

PARAMETER stop <|STOP|>
PARAMETER stop "<|STOP|>"
PARAMETER stop <|END_RESPONSE|>
PARAMETER stop "<|END_RESPONSE|>"
PARAMETER temperature 0.4

TEMPLATE """<|BEGIN_QUERY|>
{{.Prompt}}
<|END_QUERY|>
<|BEGIN_RESPONSE|>
"""

SYSTEM """You are an AI assistant. Respond to the user's query between the BEGIN_QUERY and END_QUERY tokens. Use the appropriate BEGIN_ and END_ tokens for different types of content in your response.""""""

• Save a go back to the folder (folder where model + Modelfile exisit)
• Now in terminal make sure you are in the same location of the folder and type in the following command

ollama create mycustomai  # "mycustomai" <- you can name it anything u want

After than you should be able to use this model to chat! This GGUF is based on unsloth/mistral-7b-instruct-v0.3-bnb-4bit by Unslot,

NOTE: DISCLAIMER

Please note this is not for the purpose of production, but result of Fine Tuning through self learning This is my Fine Tuning pass through with personalized customized dataset. Please feel free to customize the Modelfile, and if you do get a better response than mine, please share!!

If would like to know how I started creating my dataset, you can check this link Crafting GPT2 for Personalized AI-Preparing Data the Long Way (Part1)

The training data has the following Template:

special_tokens_dict = {
    'eos_token': '<|STOP|>',
    'bos_token': '<|STOP|>',
    'pad_token': '<|PAD|>',
    'additional_special_tokens': ['<|BEGIN_QUERY|>', '<|BEGIN_QUERY|>',
                                  '<|BEGIN_ANALYSIS|>', '<|END_ANALYSIS|>',
                                  '<|BEGIN_RESPONSE|>', '<|END_RESPONSE|>',
                                  '<|BEGIN_SENTIMENT|>', '<|END_SENTIMENT|>',
                                  '<|BEGIN_CLASSIFICATION|>', '<|END_CLASSIFICATION|>',]
}

tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))

tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids('<|STOP|>')
tokenizer.bos_token_id = tokenizer.convert_tokens_to_ids('<|STOP|>')
tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids('<|PAD|>')

The data is in the following format:

def combine_text(user_prompt, analysis, sentiment, new_response, classification):    
    user_q = f"<|STOP|><|BEGIN_QUERY|>{user_prompt}<|END_QUERY|>"
    analysis = f"<|BEGIN_ANALYSIS|>{analysis}<|END_ANALYSIS|>"
    new_response = f"<|BEGIN_RESPONSE|>{new_response}<|END_RESPONSE|>"
    classification = f"<|BEGIN_CLASSIFICATION|>{classification}<|END_CLASSIFICATION|>"
    sentiment = f"<|BEGIN_SENTIMENT|>Sentiment: {sentiment}<|END_SENTIMENT|><|STOP|>"
    return user_q + analysis + new_response + classification + sentiment