Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

Indic Language Bloom Model Training

This repository contains the code and resources for fine-tuning the Huggingface Bloom model on the Indic language dataset using Low-Rank Adaptation (LoRA). The goal is to create a high-performance language model specifically tailored to Indic languages.

Dataset

The dataset used for training is provided by AI4Bharat. I have uploaded it to huggingface hub at:

Progress

Completed

  • Low-Rank Adaptation fine-tuning of the Bloom model on streaming data
  • Single checkpoint available (training logs at Weights & Biases)

To Do

  • Benchmark current multilingual LLMs on IndicGLUE using lm-evaluation-harness
  • Integrate DeepSpeed for better resource utilization
  • Convert current instruction dataset to Indic languages and train (dolly v2 dataset, distilled from GPT, etc.)
  • Model doesn't stop producing text - how to fix?
  • Deploy RLHF community app using Cheese

Using the Model

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "aashay96/indic-BloomLM"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)



batch = tokenizer("आप कैसे हैं", return_tensors='pt')

with torch.cuda.amp.autocast():
  output_tokens = model.generate(**batch, max_new_tokens=10)

print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .