license: apache-2.0
Paper
arxiv.org/abs/2405.20222
Introduction
This repo provides the inference Gradio demo for Hybrid (Trajectory + Landmark) Control of MOFA-Video.
Environment Setup
cd MOFA-Hybrid
conda create -n mofa python==3.10
conda activate mofa
pip install -r requirements.txt
pip install opencv-python-headless
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
IMPORTANT: Gradio Version of 4.5.0 should be used since other versions may cause errors.
Checkpoints Download
Download the checkpoint of CMP from here and put it into
./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints.Downloading the necessary pretrained checkpoints from huggingface. It is recommended to directly using git lfs to clone the huggingface repo. The checkpoints should be orgnized as
./ckpt_tree.md(they will be automatically organized if you use git lfs to clone the huggingface repo).
Run Gradio Demo
Using audio to animate the facial part
python run_gradio_audio_driven.py
Using refernce video to animate the facial part
python run_gradio_video_driven.py
IMPORTANT: Please refer to the instructions on the gradio interface during the inference process.