Text Generation
GGUF
code
Inference Endpoints
munish0838 commited on
Commit
cec7e22
1 Parent(s): b5a508a

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +92 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - JetBrains/KStack-clean
5
+ base_model: JetBrains/CodeLlama-7B-KStack-clean
6
+ results:
7
+ - task:
8
+ type: text-generation
9
+ dataset:
10
+ name: MultiPL-HumanEval (Kotlin)
11
+ type: openai_humaneval
12
+ metrics:
13
+ - name: pass@1
14
+ type: pass@1
15
+ value: 37.89
16
+ tags:
17
+ - code
18
+ ---
19
+ # CodeLlama-7B-KStack-clean-GGUF
20
+ This is quantized version of [JetBrains/CodeLlama-7B-KStack-clean](https://huggingface.co/JetBrains/CodeLlama-7B-KStack-clean) created using llama.cpp
21
+
22
+ # Model description
23
+
24
+ This is a repository for the **CodeLlama-7b** model fine-tuned on the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset with rule-based filtering, in the *Hugging Face Transformers* format. KStack-clean is a small subset of [KStack](https://huggingface.co/datasets/JetBrains/KStack), the largest collection of permissively licensed Kotlin code, automatically filtered to include files that have the highest "educational value for learning algorithms in Kotlin".
25
+
26
+ # How to use
27
+
28
+ ```python
29
+ from transformers import AutoModelForCausalLM, AutoTokenizer
30
+
31
+ # Load pre-trained model and tokenizer
32
+ model_name = 'JetBrains/CodeLlama-7B-KStack-clean'
33
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
34
+ model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
35
+
36
+ # Create and encode input
37
+ input_text = """\
38
+ This function takes an integer n and returns factorial of a number:
39
+ fun factorial(n: Int): Int {\
40
+ """
41
+ input_ids = tokenizer.encode(
42
+ input_text, return_tensors='pt'
43
+ ).to('cuda')
44
+
45
+ # Generate
46
+ output = model.generate(
47
+ input_ids, max_length=60, num_return_sequences=1,
48
+ pad_token_id=tokenizer.eos_token_id
49
+ )
50
+
51
+ # Decode output
52
+ generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
53
+ print(generated_text)
54
+ ```
55
+
56
+ As with the base model, we can use FIM. To do this, the following format must be used:
57
+ ```
58
+ '<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
59
+ ```
60
+
61
+ # Training setup
62
+
63
+ The model was trained on one A100 GPU with following hyperparameters:
64
+
65
+ | **Hyperparameter** | **Value** |
66
+ |:---------------------------:|:----------------------------------------:|
67
+ | `warmup` | 100 steps |
68
+ | `max_lr` | 5e-5 |
69
+ | `scheduler` | linear |
70
+ | `total_batch_size` | 32 (~30K tokens per step) |
71
+ | `num_epochs` | 2 |
72
+
73
+ More details about fine-tuning can be found in the technical report (coming soon!).
74
+
75
+ # Fine-tuning data
76
+
77
+ For tuning the model, we used 25K exmaples from the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset, selected from the larger [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset according to educational value for learning algorithms. In total, the dataset contains about 23M tokens.
78
+
79
+ # Evaluation
80
+
81
+ For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).
82
+
83
+ Here are the results of our evaluation:
84
+
85
+ | **Model name** | **Kotlin HumanEval Pass Rate** |
86
+ |:---------------------------:|:----------------------------------------:|
87
+ | `CodeLlama-7B` | 26.89 |
88
+ | `CodeLlama-7B-KStack-clean` | **37.89** |
89
+
90
+ # Ethical Considerations and Limitations
91
+
92
+ CodeLlama-7B-KStack-clean is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.