# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

GPT-6B_Tuned_small_pile is a GPT-j-6B model trained on 0.1 million example of pile dataset.

n_embd: 4096, n_layer: 28, n_positions: 2048

Tuning Parameters:
  
    val_split_percent: 20,
  
    momentum: 0.9 
  
    train_batch_size (eff) : 32
  
    train_micro_batch: 16
  
    gradient_accumulation_steps: 2
    
    gradient_clipping: 0.5
    
    learning_rate: 0.00001
    
    weight_decay: 0.01
    
    lr_schedular: cosine
    
    lr_warmup_steps: 1000
    
    lr_decay: 0.1
    
    lr_decay_step: 2000
    
    mixed_precision: bf16



![image.png](https://s3.amazonaws.com/moonup/production/uploads/642bb1915df44ff245471fca/Ke-ShGT0sBVGEjrShxped.png)## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model:** EleutherAI/gpt-j-6b 



{"language": ["en"], "tags": ["casual_lm"], "datasets": ["ola13/small-the_pile"], "metrics": ["accuracy", "code_eval"], "pipeline_tag": "text-generation"}
transformers pytorch gptj text-generation casual_lm en dataset:ola13/small-the_pile autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #gptj #text-generation #casual_lm #en #dataset-ola13/small-the_pile #autotrain_compatible #endpoints_compatible #region-us 

# Model Card for Model ID



GPT-6B_Tuned_small_pile is a GPT-j-6B model trained on 0.1 million example of pile dataset.

n_embd: 4096, n_layer: 28, n_positions: 2048

Tuning Parameters:
  
    val_split_percent: 20,
  
    momentum: 0.9 
  
    train_batch_size (eff) : 32
  
    train_micro_batch: 16
  
    gradient_accumulation_steps: 2
    
    gradient_clipping: 0.5
    
    learning_rate: 0.00001
    
    weight_decay: 0.01
    
    lr_schedular: cosine
    
    lr_warmup_steps: 1000
    
    lr_decay: 0.1
    
    lr_decay_step: 2000
    
    mixed_precision: bf16



!URL## Model Details

### Model Description





- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model: EleutherAI/gpt-j-6b 



# Model Card for Model ID



GPT-6B_Tuned_small_pile is a GPT-j-6B model trained on 0.1 million example of pile dataset.

n_embd: 4096, n_layer: 28, n_positions: 2048

Tuning Parameters:
  
    val_split_percent: 20,
  
    momentum: 0.9 
  
    train_batch_size (eff) : 32
  
    train_micro_batch: 16
  
    gradient_accumulation_steps: 2
    
    gradient_clipping: 0.5
    
    learning_rate: 0.00001
    
    weight_decay: 0.01
    
    lr_schedular: cosine
    
    lr_warmup_steps: 1000
    
    lr_decay: 0.1
    
    lr_decay_step: 2000
    
    mixed_precision: bf16



!URL## Model Details ### Model Description





- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model: EleutherAI/gpt-j-6b
TAGS
#transformers #pytorch #gptj #text-generation #casual_lm #en #dataset-ola13/small-the_pile #autotrain_compatible #endpoints_compatible #region-us 
 # Model Card for Model ID



GPT-6B_Tuned_small_pile is a GPT-j-6B model trained on 0.1 million example of pile dataset.

n_embd: 4096, n_layer: 28, n_positions: 2048

Tuning Parameters:
  
    val_split_percent: 20,
  
    momentum: 0.9 
  
    train_batch_size (eff) : 32
  
    train_micro_batch: 16
  
    gradient_accumulation_steps: 2
    
    gradient_clipping: 0.5
    
    learning_rate: 0.00001
    
    weight_decay: 0.01
    
    lr_schedular: cosine
    
    lr_warmup_steps: 1000
    
    lr_decay: 0.1
    
    lr_decay_step: 2000
    
    mixed_precision: bf16



!URL## Model Details ### Model Description





- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model: EleutherAI/gpt-j-6b