QuantFactory/gemma-2-9b-it-function-calling-GGUF-GGUF

This is quantized version of DiTy/gemma-2-9b-it-function-calling-GGUF created using llama.cpp

Original Model Card

DiTy/gemma-2-9b-it-function-calling-GGUF

This model is a fine-tuned version of google/gemma-2-9b-it for the Function Calling task on non-synthetic data, fully annotated by humans only, on the English version of the DiTy/function-calling dataset.

NB: This model has a fairly high quality, but you might want to try a big guy DiTy/gemma-2-27b-it-function-calling-GGUF.

In addition to safetensors, the model is available in GGUF formats (in this case, you need to download only a single file (how to inference GGUF model)):

Filename	Quant type	File Size	Description
gemma-2-9B-it-function-calling-F16.gguf	F16	18.5GB	Base model with float16
gemma-2-9B-it-function-calling-Q8_0.gguf	Q8_0	9.83GB	Extremely high quality, generally unneeded but max available quant.
gemma-2-9B-it-function-calling-Q6_K.gguf	Q6_K	7.59GB	Very high quality, near perfect, recommended.
gemma-2-9B-it-function-calling-Q5_K_M.gguf	Q5_K_M	6.65GB	High quality, very usable.
gemma-2-9B-it-function-calling-Q5_K_S.gguf	Q5_K_S	6.48GB	High quality, very usable.

Model card tree

• How prepare your functions (tools) for Function Calling
• Just use chat template for generation
• Prompt structure and expected content
• Evaluation of function calling models

Usage (HuggingFace Transformers)

Below we share some code snippets on how to get quickly started with running the model. First, install the Transformers library with:

pip install -U transformers

Prepare your functions for Function Calling

You should write the functions (tools) used by the model in Python code and make sure to add Python docstrings as in the example below:

def get_weather(city: str):
    """
    A function that returns the weather in a given city.
    
    Args:
        city: The city to get the weather for.
    """
    import random
    
    return "sunny" if random.random() > 0.5 else "rainy"


def get_sunrise_sunset_times(city: str):
    """
    A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].
    
    Args:
        city: The city to get the sunrise and sunset times for.
    """
    
    return ["6:00 AM", "6:00 PM"]

Just use chat template

Next, you need to download the model and tokenizer:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "DiTy/gemma-2-9b-it-function-calling-GGUF",
    device_map="auto",
    torch_dtype=torch.bfloat16,  # use float16 or float32 if bfloat16 is not available to you.
    cache_dir=PATH_TO_MODEL_DIR,  # optional
)
tokenizer = AutoTokenizer.from_pretrained(
    "DiTy/gemma-2-9b-it-function-calling-GGUF",
    cache_dir=PATH_TO_MODEL_DIR,  # optional
)

To get the result of generation, just use apply_chat_template. In order to take into account our written functions (tools), we need to pass them as a list through the tools attribute and also use add_prompt_generation=True.

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
]

inputs = tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,  # adding prompt for generation
    tools=[get_weather, get_sunrise_sunset_times],  # our functions (tools)
)

print(inputs)

Then our inputs will look like this:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model

Now we can generate a model's response. Be careful because, after apply_chat_template, there is no need to add special tokens during tokenization. So, use add_special_tokens=False:

terminator_ids = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<end_of_turn>"),
]

prompt_ids =  tokenizer.encode(inputs, add_special_tokens=False, return_tensors='pt').to(model.device)
generated_ids = model.generate(
    prompt_ids,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
    bos_token_id=tokenizer.bos_token_id,
)
generated_response = tokenizer.decode(generated_ids[0][prompt_ids.shape[-1]:], skip_special_tokens=False)  # `skip_special_tokens=False` for debug

print(generated_response)

We get the generation as a function call:

Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>

Great, now we can pick up and process the results with our called function, and then provide the model with the function's response:

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},  # a hypothetical response from our function
]

inputs = tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,  # adding prompt for generation
    tools=[get_weather, get_sunrise_sunset_times],  # our functions (tools)
)

print(inputs)

Let's make sure the inputs are correct:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model
Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>
<start_of_turn>user
Function response: {"times_list": ["6:00 AM", "6:00 PM"]}<end_of_turn>
<start_of_turn>model

Similarly, we generate a response from the model:

prompt_ids =  tokenizer.encode(inputs, add_special_tokens=False, return_tensors='pt').to(model.device)
generated_ids = model.generate(
    prompt_ids,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
    bos_token_id=tokenizer.bos_token_id,
)
generated_response = tokenizer.decode(generated_ids[0][prompt_ids.shape[-1]:], skip_special_tokens=False)  # `skip_special_tokens=False` for debug

print(generated_response)

As a result, we get the model's response:

The sunrise time in Los Angeles is 6:00 AM.<end_of_turn>

Usage via transformers pipeline

Generation via pipeline

from transformers import pipeline


generation_pipeline = pipeline(
    "text-generation",
    model="DiTy/gemma-2-9b-it-function-calling-GGUF",
    model_kwargs={
        "torch_dtype": torch.bfloat16,  # use float16 or float32 if bfloat16 is not supported for you. 
        "cache_dir": PATH_TO_MODEL_DIR,  # OPTIONAL
    },
    device_map="auto",
)

history_messages = [
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},
]

inputs = generation_pipeline.tokenizer.apply_chat_template(
    history_messages,
    tokenize=False,
    add_generation_prompt=True,
    tools=[get_weather, get_sunrise_sunset_times],
)

terminator_ids = [
    generation_pipeline.tokenizer.eos_token_id,
    generation_pipeline.tokenizer.convert_tokens_to_ids("<end_of_turn>")
]

outputs = generation_pipeline(
    inputs,
    max_new_tokens=512,
    eos_token_id=terminator_ids,
)

print(outputs[0]["generated_text"][len(inputs):])

Prompt structure and expected content

For the most correct operation of the model, it is assumed that apply_chat_template will be used. It is necessary to transmit the message history in a certain format.

history_messages = [
    {"role": "...", "content": "..."},
    ...
]

The following roles are available for use:

• system - an optional role, its content is always placed at the very beginning and before listing the functions available to the model (tools). You can always use the standard option that was used during the training: "You are a helpful assistant with access to the following functions. Use them if required - "
• user - the user's request is transmitted through this role.
• function-call - The body of the function call is passed through this role. Although the model is trained to generate a function call in the form of "Function call: {...}<end_of_turn>", you should still pass only the body "{...}" to the "content" field, since using apply_chat_template, the postscript in the instructions is added automatically.
• function-response - in this role, we must pass the response of our function in the "content" field as a dictionary '{"name_returnable_value": value}'.
• model - the content under this role is considered to be the generated text of the model.

Chat history with Function Calling

[
    {"role": "system", "content": "You are a helpful assistant with access to the following functions. Use them if required - "},
    {"role": "user", "content": "Hi, can you tell me the time of sunrise in Los Angeles?"},
    {"role": "function-call", "content": '{"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}'},
    {"role": "function-response", "content": '{"times_list": ["6:00 AM", "6:00 PM"]}'},
]

It looks like:

<bos><start_of_turn>user
You are a helpful assistant with access to the following functions. Use them if required - {
    "name": "get_weather",
    "description": "A function that returns the weather in a given city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the weather for."
            }
        },
        "required": [
            "city"
        ]
    }
},
{
    "name": "get_sunrise_sunset_times",
    "description": "A function that returns the time of sunrise and sunset at the present moment, for a given city, in the form of a list: [sunrise_time, sunset_time].",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get the sunrise and sunset times for."
            }
        },
        "required": [
            "city"
        ]
    }
}

Hi, can you tell me the time of sunrise in Los Angeles?<end_of_turn>
<start_of_turn>model
Function call: {"name": "get_sunrise_sunset_times", "arguments": {"city": "Los Angeles"}}<end_of_turn>
<start_of_turn>user
Function response: {"times_list": ["6:00 AM", "6:00 PM"]}<end_of_turn>

Chat history with a standard user-model template

[
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Tell me about California"},
]

It looks like:

<bos><start_of_turn>user
You are a helpful assistant

Tell me about California<end_of_turn>

Evaluation

During the learning process, the validation error was approximated to the following values:

Model	Generation Language	Approximately Validation Loss
DiTy/gemma-2-27b-it-function-calling-GGUF	EN	0.47
DiTy/gemma-2-9b-it-russian-function-calling-GGUF	RU	0.57
DiTy/gemma-2-9b-it-function-calling-GGUF	EN	0.5
DiTy/gemma-2-2b-it-function-calling	EN	0.66

Citation

@article{gemma_2024,
    title={Gemma},
    url={https://www.kaggle.com/m/3301},
    DOI={10.34740/KAGGLE/M/3301},
    publisher={Kaggle},
    author={Gemma Team},
    year={2024}
}