README.md

---
language:
- en
- ja
- de
- fr
- multilingual
tags:
- t5
- adapter
- flan-t5
- peft
- lora
datasets:
- yahma/alpaca-cleaned
- databricks/databricks-dolly-15k
- samsum
pipeline_tag: text2text-generation
base_model: google/flan-t5-xl
---

# Usage

Find below some example scripts on how to use the model in `transformers`:

## Using the Pytorch model
```python
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# Load peft config for pre-trained checkpoint etc.
peft_model_id = "rsonavane/flan-t5-xl-alpaca-dolly-lora-peft"
config = PeftConfig.from_pretrained(peft_model_id)

# load base LLM model and tokenizer
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path,  load_in_8bit=True,  device_map={"":0})
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id, device_map={"":0})
```

## Prompt generation
```python
def generate_prompt(instruction: str, input_ctxt: str = "") -> str:
    if input_ctxt:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_ctxt}

### Response:"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:"""
```
## Inference
```python

input_ctxt = ""
instruction = ""

input_text = generate_prompt(instruction, input_ctxt)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
```

## Training Details
Intended for conversation analysis, closed qna and summarization.
Trained on instructions from doll-15k, alpaca-52k and samsum dataset.