
<p align="center" width="100%">
<img src="https://i.postimg.cc/MKmyP9wH/new-banner.png"  width="80%" height="80%">
</p>


<div>
<div align="center">
    <a href='https://brianboli.com/' target='_blank'>Bo Li*<sup>1</sup></a>&emsp;
    <a href='https://zhangyuanhan-ai.github.io/' target='_blank'>Yuanhan Zhang*<sup>,1</sup></a>&emsp;
    <a href='https://cliangyu.com/' target='_blank'>Liangyu Chen*<sup>,1</sup></a>&emsp;
    <a href='https://king159.github.io/' target='_blank'>Jinghao Wang*<sup>,1</sup></a>&emsp;
    <a href='https://pufanyi.github.io/' target='_blank'>Fanyi Pu*<sup>,1</sup></a>&emsp;
    </br>
    <a href='https://jingkang50.github.io/' target='_blank'>Jingkang Yang<sup>1</sup></a>&emsp;
    <a href='https://chunyuan.li/' target='_blank'>Chunyuan Li<sup>2</sup></a>&emsp;
    <a href='https://liuziwei7.github.io/' target='_blank'>Ziwei Liu<sup>1</sup></a>
</div>
<div>
<div align="center">
    <sup>1</sup>S-Lab, Nanyang Technological University&emsp;
    <sup>2</sup>Microsoft Research, Redmond
</div>

This weight is for **initilizing training for Otter**. It's directly converted from Openflamingo.

You can load and try this model using
```python
model = OtterForConditionalGeneration.from_pretrained("luodian/OTTER-LLaMA7B-Init", device_map="sequential")
model.text_tokenizer.padding_side = "left"
tokenizer = model.text_tokenizer
image_processor = transformers.CLIPImageProcessor()
model.eval()
```

You can also start training Otter via the commands
```python
python -m accelerate.commands.launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml \
pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=luodian/OTTER-LLaMA7B-Init \
--mimicit_path=/data/azure_storage/otter/mimicit/xx/xx_instructions.json \
--images_path=/data/azure_storage/otter/mimicit/xx/xx.json \
--batch_size=4 --num_epochs=1 --report_to_wandb \
--wandb_entity=ntu-slab \
--external_save_dir=/data/bli/checkpoints \
--save_hf_model \
--run_name=OTTER-MPT1B \
--wandb_project=OTTER-MPT1B \
--workers=4 \
--lr_scheduler=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01
```

If you wish to init a video instruction tuning, you should add 
```json
"max_num_frames": 128
```
to `config.json` inside the folder.
{"license": "mit"}
transformers pytorch flamingo text2text-generation license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #flamingo #text2text-generation #license-mit #autotrain_compatible #endpoints_compatible #region-us 


<p align="center" width="100%">
<img src="https://i.URL  width="80%" height="80%">
</p>


<div>
<div align="center">
    <a href='URL target='_blank'>Bo Li*<sup>1</sup></a>&emsp;
    <a href='URL target='_blank'>Yuanhan Zhang*<sup>,1</sup></a>&emsp;
    <a href='URL target='_blank'>Liangyu Chen*<sup>,1</sup></a>&emsp;
    <a href='URL target='_blank'>Jinghao Wang*<sup>,1</sup></a>&emsp;
    <a href='URL target='_blank'>Fanyi Pu*<sup>,1</sup></a>&emsp;
    </br>
    <a href='URL target='_blank'>Jingkang Yang<sup>1</sup></a>&emsp;
    <a href='URL target='_blank'>Chunyuan Li<sup>2</sup></a>&emsp;
    <a href='URL target='_blank'>Ziwei Liu<sup>1</sup></a>
</div>
<div>
<div align="center">
    <sup>1</sup>S-Lab, Nanyang Technological University&emsp;
    <sup>2</sup>Microsoft Research, Redmond
</div>

This weight is for initilizing training for Otter. It's directly converted from Openflamingo.

You can load and try this model using


You can also start training Otter via the commands


If you wish to init a video instruction tuning, you should add 

to 'URL' inside the folder.
TAGS
#transformers #pytorch #flamingo #text2text-generation #license-mit #autotrain_compatible #endpoints_compatible #region-us 
