Upload folder using huggingface_hub

README.md

@@ -1,141 +1,141 @@
----
-title: Cinemo
-app_file: demo.py
-sdk: gradio
-sdk_version: 4.39.0
-tags:
-- Image-2-Video
-- LLM
-- Large Language Model
-short_description: Multimodal Image-to-Video
-emoji: 🎥
-colorFrom: green
-colorTo: indigo
----
-## Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models<br><sub>Official PyTorch Implementation</sub>
-
-
-[![Arxiv](https://img.shields.io/badge/Arxiv-b31b1b.svg)](https://arxiv.org/abs/2407.15642) 
-[![Project Page](https://img.shields.io/badge/Project-Website-blue)](https://maxin-cn.github.io/cinemo_project/)
-
-
-This repo contains pre-trained weights, and sampling code for our paper exploring image animation with motion diffusion models (Cinemo). You can find more visualizations on our [project page](https://maxin-cn.github.io/cinemo_project/).
-
-In this project, we propose a novel method called Cinemo, which can perform motion-controllable image animation with strong consistency and smoothness. To improve motion smoothness, Cinemo learns the distribution of motion residuals, rather than directly generating subsequent frames. Additionally, a structural similarity index-based method is proposed to control the motion intensity. Furthermore, we propose a noise refinement technique based on discrete cosine transformation to ensure temporal consistency. These three methods help Cinemo generate highly consistent, smooth, and motion-controlled image animation results. Compared to previous methods, Cinemo offers simpler and more precise user control and better generative performance.
- 
-<div align="center">
-    <img src="visuals/pipeline.svg">
-</div>
-
-## News
-
-- (🔥 New) Jul. 23, 2024. 💥 Our paper is released on [arxiv](https://arxiv.org/abs/2407.15642).
-
-- (🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main).
-
-
-## Setup
-
-First, download and set up the repo:
-
-```bash
-git clone https://github.com/maxin-cn/Cinemo
-cd Cinemo
-```
-
-We provide an [`environment.yml`](environment.yml) file that can be used to create a Conda environment. If you only want 
-to run pre-trained models locally on CPU, you can remove the `cudatoolkit` and `pytorch-cuda` requirements from the file.
-
-```bash
-conda env create -f environment.yml
-conda activate cinemo
-```
-
-
-## Animation 
-
-You can sample from our **pre-trained Cinemo models** with [`animation.py`](pipelines/animation.py). Weights for our pre-trained Cinemo model can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main).  The script has various arguments for adjusting sampling steps, changing the classifier-free guidance scale, etc:
-
-```bash
-bash pipelines/animation.sh
-```
-
-All related checkpoints will download automatically and then you will get the following results,
-
-<table style="width:100%; text-align:center;">
-<tr>
-  <td align="center">Input image</td>
-  <td align="center">Output video</td>
-  <td align="center">Input image</td>
-  <td align="center">Output video</td>
-</tr>
-<tr>
-  <td align="center"><img src="visuals/animations/people_walking/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/animations/people_walking/people_walking.gif" width="100%"></td>
-  <td align="center"><img src="visuals/animations/sea_swell/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/animations/sea_swell/sea_swell.gif" width="100%"></td>
-</tr>
-<tr>
-  <td align="center" colspan="2">"People Walking"</td>
-  <td align="center" colspan="2">"Sea Swell"</td>
-</tr>
-<tr>
-  <td align="center"><img src="visuals/animations/girl_dancing_under_the_stars/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/animations/girl_dancing_under_the_stars/girl_dancing_under_the_stars.gif" width="100%"></td>
-  <td align="center"><img src="visuals/animations/dragon_glowing_eyes/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/animations/dragon_glowing_eyes/dragon_glowing_eyes.gif" width="100%"></td>
-</tr>
-<tr>
-  <td align="center" colspan="2">"Girl Dancing under the Stars"</td>
-  <td align="center" colspan="2">"Dragon Glowing Eyes"</td>
-</tr>
-
-</table>
-
-
-## Other Applications
-
-You can also utilize Cinemo for other applications, such as motion transfer and video editing:
-
-```bash
-bash pipelines/video_editing.sh
-```
-
-All related checkpoints will download automatically and you will get the following results,
-
-<table style="width:100%; text-align:center;">
-<tr>
-  <td align="center">Input video</td>
-  <td align="center">First frame</td>
-  <td align="center">Edited first frame</td>
-  <td align="center">Output video</td>
-</tr>
-<tr>
-  <td align="center"><img src="visuals/video_editing/origin/a_corgi_walking_in_the_park_at_sunrise_oil_painting_style.gif" width="100%"></td>
-  <td align="center"><img src="visuals/video_editing/origin/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/video_editing/edit/0.jpg" width="100%"></td>
-  <td align="center"><img src="visuals/video_editing/edit/editing_a_corgi_walking_in_the_park_at_sunrise_oil_painting_style.gif" width="100%"></td>
-</tr>
-
-</table>
-
-
-
-## Citation
-If you find this work useful for your research, please consider citing it.
-```bibtex
-@article{ma2024cinemo,
-  title={Cinemo: Latent Diffusion Transformer for Video Generation},
-  author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
-  journal={arXiv preprint arXiv:2407.15642},
-  year={2024}
-}
-```
-
-
-## Acknowledgments
-Cinemo has been greatly inspired by the following amazing works and teams: [LaVie](https://github.com/Vchitect/LaVie) and [SEINE](https://github.com/Vchitect/SEINE), we thank all the contributors for open-sourcing.
-
-
-## License
+---
+title: Cinemo
+app_file: demo.py
+sdk: gradio
+sdk_version: 4.37.2
+tags:
+- Image-2-Video
+- LLM
+- Large Language Model
+short_description: Multimodal Image-to-Video
+emoji: 🎥
+colorFrom: green
+colorTo: indigo
+---
+## Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models<br><sub>Official PyTorch Implementation</sub>
+
+
+[![Arxiv](https://img.shields.io/badge/Arxiv-b31b1b.svg)](https://arxiv.org/abs/2407.15642) 
+[![Project Page](https://img.shields.io/badge/Project-Website-blue)](https://maxin-cn.github.io/cinemo_project/)
+
+
+This repo contains pre-trained weights, and sampling code for our paper exploring image animation with motion diffusion models (Cinemo). You can find more visualizations on our [project page](https://maxin-cn.github.io/cinemo_project/).
+
+In this project, we propose a novel method called Cinemo, which can perform motion-controllable image animation with strong consistency and smoothness. To improve motion smoothness, Cinemo learns the distribution of motion residuals, rather than directly generating subsequent frames. Additionally, a structural similarity index-based method is proposed to control the motion intensity. Furthermore, we propose a noise refinement technique based on discrete cosine transformation to ensure temporal consistency. These three methods help Cinemo generate highly consistent, smooth, and motion-controlled image animation results. Compared to previous methods, Cinemo offers simpler and more precise user control and better generative performance.
+ 
+<div align="center">
+    <img src="visuals/pipeline.svg">
+</div>
+
+## News
+
+- (🔥 New) Jul. 23, 2024. 💥 Our paper is released on [arxiv](https://arxiv.org/abs/2407.15642).
+
+- (🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main).
+
+
+## Setup
+
+First, download and set up the repo:
+
+```bash
+git clone https://github.com/maxin-cn/Cinemo
+cd Cinemo
+```
+
+We provide an [`environment.yml`](environment.yml) file that can be used to create a Conda environment. If you only want 
+to run pre-trained models locally on CPU, you can remove the `cudatoolkit` and `pytorch-cuda` requirements from the file.
+
+```bash
+conda env create -f environment.yml
+conda activate cinemo
+```
+
+
+## Animation 
+
+You can sample from our **pre-trained Cinemo models** with [`animation.py`](pipelines/animation.py). Weights for our pre-trained Cinemo model can be found [here](https://huggingface.co/maxin-cn/Cinemo/tree/main).  The script has various arguments for adjusting sampling steps, changing the classifier-free guidance scale, etc:
+
+```bash
+bash pipelines/animation.sh
+```
+
+All related checkpoints will download automatically and then you will get the following results,
+
+<table style="width:100%; text-align:center;">
+<tr>
+  <td align="center">Input image</td>
+  <td align="center">Output video</td>
+  <td align="center">Input image</td>
+  <td align="center">Output video</td>
+</tr>
+<tr>
+  <td align="center"><img src="visuals/animations/people_walking/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/animations/people_walking/people_walking.gif" width="100%"></td>
+  <td align="center"><img src="visuals/animations/sea_swell/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/animations/sea_swell/sea_swell.gif" width="100%"></td>
+</tr>
+<tr>
+  <td align="center" colspan="2">"People Walking"</td>
+  <td align="center" colspan="2">"Sea Swell"</td>
+</tr>
+<tr>
+  <td align="center"><img src="visuals/animations/girl_dancing_under_the_stars/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/animations/girl_dancing_under_the_stars/girl_dancing_under_the_stars.gif" width="100%"></td>
+  <td align="center"><img src="visuals/animations/dragon_glowing_eyes/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/animations/dragon_glowing_eyes/dragon_glowing_eyes.gif" width="100%"></td>
+</tr>
+<tr>
+  <td align="center" colspan="2">"Girl Dancing under the Stars"</td>
+  <td align="center" colspan="2">"Dragon Glowing Eyes"</td>
+</tr>
+
+</table>
+
+
+## Other Applications
+
+You can also utilize Cinemo for other applications, such as motion transfer and video editing:
+
+```bash
+bash pipelines/video_editing.sh
+```
+
+All related checkpoints will download automatically and you will get the following results,
+
+<table style="width:100%; text-align:center;">
+<tr>
+  <td align="center">Input video</td>
+  <td align="center">First frame</td>
+  <td align="center">Edited first frame</td>
+  <td align="center">Output video</td>
+</tr>
+<tr>
+  <td align="center"><img src="visuals/video_editing/origin/a_corgi_walking_in_the_park_at_sunrise_oil_painting_style.gif" width="100%"></td>
+  <td align="center"><img src="visuals/video_editing/origin/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/video_editing/edit/0.jpg" width="100%"></td>
+  <td align="center"><img src="visuals/video_editing/edit/editing_a_corgi_walking_in_the_park_at_sunrise_oil_painting_style.gif" width="100%"></td>
+</tr>
+
+</table>
+
+
+
+## Citation
+If you find this work useful for your research, please consider citing it.
+```bibtex
+@article{ma2024cinemo,
+  title={Cinemo: Latent Diffusion Transformer for Video Generation},
+  author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
+  journal={arXiv preprint arXiv:2407.15642},
+  year={2024}
+}
+```
+
+
+## Acknowledgments
+Cinemo has been greatly inspired by the following amazing works and teams: [LaVie](https://github.com/Vchitect/LaVie) and [SEINE](https://github.com/Vchitect/SEINE), we thank all the contributors for open-sourcing.
+
+
+## License
 The code and model weights are licensed under [LICENSE](LICENSE).

demo.py

@@ -269,13 +269,28 @@ with gr.Blocks() as demo:
         preview_button.click(fn=update_and_resize_image, inputs=[input_image_path, height, width], outputs=[input_image])
         input_image_path.submit(fn=update_and_resize_image, inputs=[input_image_path, height, width], outputs=[input_image])
 
+#         EXAMPLES = [
+#             ["./example/aircrafts_flying/0.jpg", "aircrafts flying"                   , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/fireworks/0.jpg", "fireworks"                                 , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/flowers_swaying/0.jpg", "flowers swaying"                     , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/girl_walking_on_the_beach/0.jpg", "girl walking on the beach" , "", 50, 320, 512, 7.5, True, 0.23, 985, 10, 200],
+#             ["./example/house_rotating/0.jpg", "house rotating"                       , "", 50, 320, 512, 7.5, True, 0.23, 985, 10, 100],
+#             ["./example/people_runing/0.jpg", "people runing"                         , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/shark_swimming/0.jpg", "shark swimming"                       , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/car_moving/0.jpg", "car moving"                               , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+#             ["./example/windmill_turning/0.jpg", "windmill turning"                   , "", 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+# ]
+        
         EXAMPLES = [
-            ["./example/aircrafts_flying/0.jpg", "aircrafts flying"                   , 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
-            ["./example/fireworks/0.jpg", "fireworks"                                 , 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
-            ["./example/flowers_swaying/0.jpg", "flowers swaying"                     , 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
-            ["./example/girl_walking_on_the_beach/0.jpg", "girl walking on the beach" , 50, 320, 512, 7.5, True, 0.23, 985, 10, 200],
-            ["./example/house_rotating/0.jpg", "house rotating"                       , 50, 320, 512, 7.5, True, 0.23, 985, 10, 100],
-            ["./example/people_runing/0.jpg", "people runing"                         , 50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/aircrafts_flying/0.jpg", "aircrafts flying"                   ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/fireworks/0.jpg", "fireworks"                                 ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/flowers_swaying/0.jpg", "flowers swaying"                     ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/girl_walking_on_the_beach/0.jpg", "girl walking on the beach" ,  50, 320, 512, 7.5, True, 0.23, 985, 10, 200],
+            ["./example/house_rotating/0.jpg", "house rotating"                       ,  50, 320, 512, 7.5, True, 0.23, 985, 10, 100],
+            ["./example/people_runing/0.jpg", "people runing"                         ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/shark_swimming/0.jpg", "shark swimming"                       ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/car_moving/0.jpg", "car moving"                               ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
+            ["./example/windmill_turning/0.jpg", "windmill turning"                   ,  50, 320, 512, 7.5, True, 0.23, 975, 10, 100],
 ]
 
         examples = gr.Examples(
@@ -283,8 +298,8 @@ with gr.Blocks() as demo:
             fn = gen_video,
             inputs=[input_image, prompt_textbox, sample_step_slider, height, width, txt_cfg_scale, use_dctinit, dct_coefficients, noise_level, motion_bucket_id, seed_textbox],
             outputs=[result_video],
-            # cache_examples=True,
-            cache_examples="lazy",
+            cache_examples=True,
+            # cache_examples="lazy",
         )
 
         generate_button.click(

gradio_cached_examples/40/log.csv

@@ -0,0 +1,10 @@
+Generated Animation,flag,username,timestamp
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/a3f7225a444fbd51e1df/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/6bf88e8cee9a88a69c12b5abe5f9570920bc64ab/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:20:16.156080
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/0bd2c892d93cea6575fb/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/deb9620f616c3681cb074388781099f78a25dc8f/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:20:27.176634
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/216805b5d5d2f5ab6f3b/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/d9e392300169a439b3f5721579849e3e5ce6abf9/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:20:38.238643
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/430581314c8707974074/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/a115eecb56cea7444252bf5d5c5d6bf946cb90fb/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:20:49.321367
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/3527a82ecd4f8d855fde/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/0d370bf6a6284e9832de2b865a19a8166fd7b2da/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:21:00.445240
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/9f4316aca9aae42dac66/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/7e3c6838256fd5a89b20ad62ce42288212e62097/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:21:11.558142
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/76b89e66ff946d2c03e3/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/81b0981cf82c96460f1cd5bd8de8e0cf5855b5f9/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:21:22.638538
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/9dc7d55d2941d8284460/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/cd0c3875e3725ff5c892c4f728a9e62a5cdcd6a3/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:21:33.763127
+"{""video"": {""path"": ""gradio_cached_examples/40/Generated Animation/68d9ae906d26a709880e/temp.mp4"", ""url"": ""/file=/data/pe1/000scratch/slurm_tmpdir/20240731_job_53275693.XzFi/gradio/a74b0b67f332bb3e1cd0b40cde137223c5c65a1a/temp.mp4"", ""size"": null, ""orig_name"": ""temp.mp4"", ""mime_type"": null, ""is_stream"": false, ""meta"": {""_type"": ""gradio.FileData""}}, ""subtitles"": null}",,,2024-07-31 13:21:44.911136