metadata

license: apache-2.0
datasets:
  - JetBrains/KExercises
base_model: JetBrains/CodeLlama-7B-Kexer
results:
  - task:
      type: text-generation
    dataset:
      name: MultiPL-HumanEval (Kotlin)
      type: openai_humaneval
    metrics:
      - name: pass@1
        type: pass@1
        value: 42.24
tags:
  - code

CodeLlama-7B-Kexer-GGUF

This is quantized version of JetBrains/CodeLlama-7B-Kexer created using llama.cpp

Model Description

Kexer models are a collection of open-source generative text models fine-tuned on the Kotlin Exercices dataset. This is a repository for the fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.

Training setup

The model was trained on one A100 GPU with the following hyperparameters:

Hyperparameter	Value
`warmup`	10%
`max_lr`	1e-4
`scheduler`	linear
`total_batch_size`	256 (~130K tokens per step)
`num_epochs`	4

More details about fine-tuning can be found in the technical report (coming soon!).

Fine-tuning data

For tuning this model, we used 15K exmaples from the synthetically generated Kotlin Exercices dataset. Every example follows the HumanEval format. In total, the dataset contains about 3.5M tokens.

Evaluation

For evaluation, we used the Kotlin HumanEval dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the datasets's page.

Here are the results of our evaluation:

Model name	Kotlin HumanEval Pass Rate
`CodeLlama-7B`	26.89
`CodeLlama-7B-Kexer`	42.24

Ethical considerations and limitations

CodeLlama-7B-Kexer is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-Kexer's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-Kexer, developers should perform safety testing and tuning tailored to their specific applications of the model.