
# TinyStarCoderPy

This is a 164M parameters model with the same architecture as [StarCoder](https://huggingface.co/bigcode/starcoder) (8k context length, MQA & FIM). It was trained on the Python data from [StarCoderData](https://huggingface.co/datasets/bigcode/starcoderdata)
for ~6 epochs which amounts to 100B tokens.


## Use

### Intended use

The model was trained on GitHub code, to assist with some tasks like [Assisted Generation](https://huggingface.co/blog/assisted-generation). For pure code completion, we advise using our 15B models [StarCoder]() or [StarCoderBase]().


### Generation
```python
# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/tiny_starcoder_py"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```

### Fill-in-the-middle
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:

```python
input_text = "<fim_prefix>def print_one_two_three():\n    print('one')\n    <fim_suffix>\n    print('three')<fim_middle>"
inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```

# Training

## Model

- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective
- **Pretraining steps:** 50k
- **Pretraining tokens:** 100 billion
- **Precision:** bfloat16

## Hardware

- **GPUs:** 32 Tesla A100
- **Training time:** 18 hours

## Software

- **Orchestration:** [Megatron-LM](https://github.com/bigcode-project/Megatron-LM)
- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)
- **BP16 if applicable:** [apex](https://github.com/NVIDIA/apex)

# License
The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement [here](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).

{"license": "bigcode-openrail-m", "library_name": "transformers", "tags": ["code"], "datasets": ["bigcode/the-stack-dedup"], "metrics": ["code_eval"], "pipeline_tag": "text-generation", "inference": true, "widget": [{"text": "def print_hello_world():", "example_title": "Hello world", "group": "Python"}], "model-index": [{"name": "Tiny-StarCoder-Py", "results": [{"task": {"type": "text-generation"}, "dataset": {"name": "HumanEval", "type": "openai_humaneval"}, "metrics": [{"type": "pass@1", "value": "7.84%", "name": "pass@1", "verified": false}]}]}]}
transformers pytorch safetensors gpt_bigcode text-generation code dataset:bigcode/the-stack-dedup license:bigcode-openrail-m model-index autotrain_compatible endpoints_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #safetensors #gpt_bigcode #text-generation #code #dataset-bigcode/the-stack-dedup #license-bigcode-openrail-m #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 


# TinyStarCoderPy

This is a 164M parameters model with the same architecture as StarCoder (8k context length, MQA & FIM). It was trained on the Python data from StarCoderData
for ~6 epochs which amounts to 100B tokens.


## Use

### Intended use

The model was trained on GitHub code, to assist with some tasks like Assisted Generation. For pure code completion, we advise using our 15B models [StarCoder]() or [StarCoderBase]().


### Generation


### Fill-in-the-middle
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:



# Training

## Model

- Architecture: GPT-2 model with multi-query attention and Fill-in-the-Middle objective
- Pretraining steps: 50k
- Pretraining tokens: 100 billion
- Precision: bfloat16

## Hardware

- GPUs: 32 Tesla A100
- Training time: 18 hours

## Software

- Orchestration: Megatron-LM
- Neural networks: PyTorch
- BP16 if applicable: apex

# License
The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement here.

# TinyStarCoderPy

This is a 164M parameters model with the same architecture as StarCoder (8k context length, MQA & FIM). It was trained on the Python data from StarCoderData
for ~6 epochs which amounts to 100B tokens. ## Use ### Intended use

The model was trained on GitHub code, to assist with some tasks like Assisted Generation. For pure code completion, we advise using our 15B models [StarCoder]() or [StarCoderBase](). ### Generation ### Fill-in-the-middle
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output: # Training ## Model

- Architecture: GPT-2 model with multi-query attention and Fill-in-the-Middle objective
- Pretraining steps: 50k
- Pretraining tokens: 100 billion
- Precision: bfloat16 ## Hardware

- GPUs: 32 Tesla A100
- Training time: 18 hours ## Software

- Orchestration: Megatron-LM
- Neural networks: PyTorch
- BP16 if applicable: apex # License
The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement here.
TAGS
#transformers #pytorch #safetensors #gpt_bigcode #text-generation #code #dataset-bigcode/the-stack-dedup #license-bigcode-openrail-m #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
 # TinyStarCoderPy

This is a 164M parameters model with the same architecture as StarCoder (8k context length, MQA & FIM). It was trained on the Python data from StarCoderData
for ~6 epochs which amounts to 100B tokens. ## Use ### Intended use

The model was trained on GitHub code, to assist with some tasks like Assisted Generation. For pure code completion, we advise using our 15B models [StarCoder]() or [StarCoderBase](). ### Generation ### Fill-in-the-middle
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output: # Training ## Model

- Architecture: GPT-2 model with multi-query attention and Fill-in-the-Middle objective
- Pretraining steps: 50k
- Pretraining tokens: 100 billion
- Precision: bfloat16 ## Hardware

- GPUs: 32 Tesla A100
- Training time: 18 hours ## Software

- Orchestration: Megatron-LM
- Neural networks: PyTorch
- BP16 if applicable: apex # License
The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement here.