|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
|
|
## Paper |
|
|
|
arxiv.org/abs/2405.20222 |
|
|
|
|
|
## Introduction |
|
|
|
This repo provides the inference Gradio demo for **Hybrid (Trajectory + Landmark)** Control of [MOFA-Video](https://myniuuu.github.io/MOFA_Video/). |
|
|
|
## Environment Setup |
|
|
|
``` |
|
cd MOFA-Hybrid |
|
conda create -n mofa python==3.10 |
|
conda activate mofa |
|
pip install -r requirements.txt |
|
pip install opencv-python-headless |
|
pip install "git+https://github.com/facebookresearch/pytorch3d.git" |
|
``` |
|
|
|
**IMPORTANT:** Gradio Version of **4.5.0** should be used since other versions may cause errors. |
|
|
|
|
|
## Checkpoints Download |
|
1. Download the checkpoint of CMP from [here](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid/blob/main/models/cmp/experiments/semiauto_annot/resnet50_vip%2Bmpii_liteflow/checkpoints/ckpt_iter_42000.pth.tar) and put it into `./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints`. |
|
|
|
2. Downloading the necessary pretrained checkpoints from [huggingface](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid). It is recommended to directly using git lfs to clone the [huggingface repo](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid). The checkpoints should be orgnized as `./ckpt_tree.md` (they will be automatically organized if you use git lfs to clone the [huggingface repo](https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid)). |
|
|
|
|
|
## Run Gradio Demo |
|
|
|
### Using audio to animate the facial part |
|
|
|
`python run_gradio_audio_driven.py` |
|
|
|
### Using refernce video to animate the facial part |
|
|
|
`python run_gradio_video_driven.py` |
|
|
|
**IMPORTANT:** Please refer to the instructions on the gradio interface during the inference process. |
