Added code to DL models

MOFA-Video-Traj/README.md

@@ -1,42 +0,0 @@
-## Environment Setup
-
-`pip install -r requirements.txt`
-
-## Download checkpoints
-
-1. Download the pretrained checkpoints of [SVD_xt](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1) from huggingface to `./ckpts`.
-
-2. Download the checkpoint of [MOFA-Adapter](https://huggingface.co/MyNiuuu/MOFA-Video-Traj) from huggingface to `./ckpts`.
-
-3. Download the checkpoint of CMP from [here](https://huggingface.co/MyNiuuu/MOFA-Video-Traj/blob/main/models/cmp/experiments/semiauto_annot/resnet50_vip%2Bmpii_liteflow/checkpoints/ckpt_iter_42000.pth.tar) and put it into `./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints`.
-
-The final structure of checkpoints should be:
-
-
-```text
-./ckpts/
-|-- controlnet
-|   |-- config.json
-|   `-- diffusion_pytorch_model.safetensors
-|-- stable-video-diffusion-img2vid-xt-1-1
-|   |-- feature_extractor
-|       |-- ...
-|   |-- image_encoder
-|       |-- ...
-|   |-- scheduler
-|       |-- ...
-|   |-- unet
-|       |-- ...
-|   |-- unet_ch9
-|       |-- ...
-|   |-- vae
-|       |-- ...
-|   |-- svd_xt_1_1.safetensors
-|   `-- model_index.json
-```
-
-## Run Gradio Demo
-
-`python run_gradio.py`
-
-Please refer to the instructions on the gradio interface during the inference process.

README.md

@@ -1,85 +1,42 @@
+## Environment Setup
 
+`pip install -r requirements.txt`
 
+## Download checkpoints
 
+1. Download the pretrained checkpoints of [SVD_xt](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1) from huggingface to `./ckpts`.
 
-<div align="center">
-  <h1>
-    🦄️ MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model
-  </h1>
-<a href='https://arxiv.org/abs/2405.20222'><img src='https://img.shields.io/badge/ArXiv-PDF-red'></a> &nbsp; <a href='https://myniuuu.github.io/MOFA_Video'><img src='https://img.shields.io/badge/Project-Page-Green'></a> &nbsp; <a href='https://huggingface.co/MyNiuuu/MOFA-Video-Traj'><img src='https://img.shields.io/badge/🤗 huggingface-MOFA_Traj-blue'></a>
-<div>
-    <a href='https://myniuuu.github.io/' target='_blank'>Muyao Niu</a> <sup>1,2</sup> &nbsp;
-    <a href='https://vinthony.github.io/academic/' target='_blank'>Xiaodong Cun</a><sup>2,*</sup> &nbsp;
-    <a href='https://xinntao.github.io/' target='_blank'>Xintao Wang</a><sup>2</sup> &nbsp;
-    <a href='https://yzhang2016.github.io/' target='_blank'>Yong Zhang</a><sup>2</sup> &nbsp; <br>
-    <a href='https://scholar.google.com/citations?user=4oXBp9UAAAAJ&hl=en' target='_blank'>Ying Shan</a><sup>2</sup> &nbsp;
-    <a href='https://scholar.google.com/citations?user=JD-5DKcAAAAJ&hl=en' target='_blank'>Yinqiang Zheng</a><sup>1,*</sup> &nbsp;
-</div>
-<div>
-    <sup>1</sup> The University of Tokyo &nbsp; <sup>2</sup> Tencent AI Lab &nbsp; <sup>*</sup> Corresponding Author &nbsp; 
-</div>
-</div>
+2. Download the checkpoint of [MOFA-Adapter](https://huggingface.co/MyNiuuu/MOFA-Video-Traj) from huggingface to `./ckpts`.
 
----
+3. Download the checkpoint of CMP from [here](https://huggingface.co/MyNiuuu/MOFA-Video-Traj/blob/main/models/cmp/experiments/semiauto_annot/resnet50_vip%2Bmpii_liteflow/checkpoints/ckpt_iter_42000.pth.tar) and put it into `./models/cmp/experiments/semiauto_annot/resnet50_vip+mpii_liteflow/checkpoints`.
 
-<div align="center">
-Check the gallery of our <a href='https://myniuuu.github.io/MOFA_Video' target='_blank'>project page</a> for many visual results!
-</div>
+The final structure of checkpoints should be:
 
 
-
-
-## New Features/Updates 🔥🔥🔥
-
-We have released the Gradio inference code and the checkpoints for trajectory-based image animation! Please refer to `./MOFA-Video-Traj/README.md` for instructions.
-
-
-## 📰 CODE RELEASE
-- [x] (2024.05.31) Gradio demo and checkpoints for trajectory-based image animation
-- [ ] Training scripts for trajectory-based image animation
-- [ ] Inference scripts and checkpoints for keypoint-based facial image animation
-- [ ] Training scripts for keypoint-based facial image animation
-- [ ] Inference Gradio demo for hybrid image animation
-
-
-## Introduction
-
-<div align="center">
-  <h3>
-    TL;DR: Image 🏞️ + Hybrid Controls 🕹️ = Videos 🎬🍿
-  </h3>
-</div>
-
-<div align="center">
-  <img src="assets/images/project-mofa.png">
-</div>
-
-We introduce MOFA-Video, a method designed to adapt motions from different domains to the frozen Video Diffusion Model. By employing <u>sparse-to-dense (S2D) motion generation</u> and <u>flow-based motion adaptation</u>, MOFA-Video can effectively animate a single image using various types of control signals, including trajectories, keypoint sequences, AND their combinations.
-
-<p align="center">
-  <img src="assets/images/pipeline.png">
-</p>
-
-During the training stage, we generate sparse control signals through sparse motion sampling and then train different MOFA-Adapters to generate video via pre-trained SVD. During the inference stage, different MOFA-Adapters can be combined to jointly control the frozen SVD.
-
-
-## 💫 Trajectory-based Image Animation 
-
-### Inference
-
-Our inference demo is based on Gradio. Please refer to `./MOFA-Video-Traj/README.md` for instructions.
-
-
-## Citation
-```
-@article{niu2024mofa,
-  title={MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model},
-  author={Niu, Muyao and Cun, Xiaodong and Wang, Xintao and Zhang, Yong and Shan, Ying and Zheng, Yinqiang},
-  journal={arXiv preprint arXiv:2405.20222},
-  year={2024}
-}
+```text
+./ckpts/
+|-- controlnet
+|   |-- config.json
+|   `-- diffusion_pytorch_model.safetensors
+|-- stable-video-diffusion-img2vid-xt-1-1
+|   |-- feature_extractor
+|       |-- ...
+|   |-- image_encoder
+|       |-- ...
+|   |-- scheduler
+|       |-- ...
+|   |-- unet
+|       |-- ...
+|   |-- unet_ch9
+|       |-- ...
+|   |-- vae
+|       |-- ...
+|   |-- svd_xt_1_1.safetensors
+|   `-- model_index.json
 ```
 
-## Acknowledgements
-We sincerely appreciate the code release of the following projects: [DragNUWA](https://arxiv.org/abs/2308.08089), [SadTalker](https://github.com/OpenTalker/SadTalker), [AniPortrait](https://github.com/Zejun-Yang/AniPortrait), [Diffusers](https://github.com/huggingface/diffusers), [SVD_Xtend](https://github.com/pixeli99/SVD_Xtend), [Conditional-Motion-Propagation](https://github.com/XiaohangZhan/conditional-motion-propagation), and [Unimatch](https://github.com/autonomousvision/unimatch).
+## Run Gradio Demo
+
+`python run_gradio.py`
 
+Please refer to the instructions on the gradio interface during the inference process.

MOFA-Video-Traj/run_gradio.py → app.py

@@ -28,6 +28,7 @@ from diffusers.utils.import_utils import is_xformers_available
 
 from utils.flow_viz import flow_to_image
 from utils.utils import split_filename, image2arr, image2pil, ensure_dirname
+from huggingface_hub import login, hf_hub_download, snapshot_download
 
 
 output_dir_video = "./outputs/videos"
@@ -85,7 +86,12 @@ def get_sparseflow_and_mask_forward(
 
     return s_flow, mask
 
-
+def download_models(ckpts_path):
+    try:
+        snapshot_download(repo_id="vdo/stable-video-diffusion-img2vid-xt-1-1", local_dir=ckpts_path, cache_dir=ckpts_path)
+        snapshot_download(repo_id="MyNiuuu/MOFA-Video-Traj", local_dir=ckpts_path, cache_dir=ckpts_path, allow_patterns=["ckpts/controlnet/*"])
+    except (Exception, BaseException) as error:
+        print(error)
 
 def init_models(pretrained_model_name_or_path, resume_from_checkpoint, weight_dtype, device='cuda', enable_xformers_memory_efficient_attention=False, allow_tf32=False):
 
@@ -216,11 +222,14 @@ class Drag:
     def __init__(self, device, height, width, model_length):
         self.device = device
 
+        ckpts_dir = "ckpts/"
         svd_ckpt = "ckpts/stable-video-diffusion-img2vid-xt-1-1"
         mofa_ckpt = "ckpts/controlnet"
 
         self.device = 'cuda'
         self.weight_dtype = torch.float16
+        
+        download_models(ckpts_dir)
 
         self.pipeline, self.cmp = init_models(
             svd_ckpt, 
@@ -631,6 +640,10 @@ class Drag:
         return hint_path, outputs_path, flows_path, outputs_mp4_path, flows_mp4_path
 
 
+# Download checkpoints to the right place
+
+
+
 with gr.Blocks() as demo:
     gr.Markdown("""<h1 align="center">MOFA-Video</h1><br>""")
 
@@ -828,4 +841,5 @@ with gr.Blocks() as demo:
 
     run_button.click(DragNUWA_net.run, [first_frame_path, tracking_points, inference_batch_size, motion_brush_mask, motion_brush_viz, ctrl_scale], [hint_image, output_video, output_flow, output_video_mp4, output_flow_mp4])
 
-    demo.launch(server_name="0.0.0.0", debug=True, server_port=80)
+    demo.launch()
+    # demo.launch(server_name="0.0.0.0", debug=True, server_port=80)

assets/images/README.md

@@ -1 +0,0 @@
-README