---
library_name: transformers
license: bigcode-openrail-m
tags:
- code
---

# Model Card for StarCoderBase1B-Racket-SelfInstruct

Each commit to this repository has a checkpoint (one per epoch) for a fine-tuned 
StarCoderBase-1B. The dataset for fine-tuning is a Racket self-instruction
dataset. As shown in Evaluation below, self-instruction was not effective, and
this model is barely any better than StarCoderBase-1B.

## Finetuning Dataset and Hyperparameters

- Dataset: https://huggingface.co/datasets/nuprl/MultiPL-T-racket-selfinstruct
- Base Model: https://huggingface.co/bigcode/starcoderbase-1b
- Batch size: 8
- Epochs: 7
- Learning rate: 3e-05
- Warmup steps: 100
- Sequence length: 2048
  
## Evaluation

The results on MultiPL-HumanEval-Racket are as follows:

```
Dataset,Pass@k,Estimate,NumProblems,MinCompletions,MaxCompletions
humaneval-rkt-checkpoint_1494-0.2-reworded,1,7.70,161,50,50
humaneval-rkt-checkpoint_1992-0.2-reworded,1,6.86,161,50,50
humaneval-rkt-checkpoint_2490-0.2-reworded,1,6.82,161,50,50
humaneval-rkt-checkpoint_2988-0.2-reworded,1,6.91,161,50,50
humaneval-rkt-checkpoint_498-0.2-reworded,1,6.19,161,50,50
humaneval-rkt-checkpoint_6973-0.2-reworded,1,6.53,161,50,50
humaneval-rkt-checkpoint_996-0.2-reworded,1,7.08,161,50,50
```