thanks to alimama-creative ❤

README.md

@@ -0,0 +1,131 @@
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
+language:
+- en
+base_model:
+- black-forest-labs/FLUX.1-dev
+pipeline_tag: image-to-image
+tags:
+- ComfyUI
+- Inpainting
+library_name: diffusers
+---
+
+<div style="display: flex; justify-content: center; align-items: center;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama.png" alt="alimama" style="width: 20%; height: auto;">
+</div>
+
+# FLUX.1-dev ControlNet Inpainting - Beta
+
+This repository hosts an improved Inpainting ControlNet checkpoint for the [alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha) model, developed by the AlimamaCreative Team.
+
+## Key Enhancements
+
+Our latest inpainting model brings significant improvements compared to the previous version:
+
+1. **1024 Resolution Support**: Capable of directly processing and generating 1024x1024 resolution images without additional upscaling steps, providing higher quality and more detailed output results.
+2. **Enhanced Detail Generation**: Fine-tuned to capture and reproduce finer details in inpainted areas.
+3. **Improved Prompt Control**: Offers more precise control over generated content through enhanced prompt interpretation.
+
+## Showcase
+
+The following images were generated using a ComfyUI workflow ([click here to download](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json)) with these settings:
+`control-strength` = 1.0, `control-end-percent` = 1.0, `true_cfg` = 1.0
+
+| Image & Prompt Input                                | Alpha Version                                      | Beta Version                                   |
+|-----------------------------------------------------|----------------------------------------------------|----------------------------------------------------|
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>'Write a few lines of words "alimama creative" on the wooden board'</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"a girl with big beautiful white wing"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"red hair"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>" "</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"Albert Einstein"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"Ravello Outdoor Sectional Sofa Set with Coffee Table"</i>
+</div>
+
+### ComfyUI Usage Guidelines:
+
+Download example ComfyUI workflow [here](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json).
+
+- Using `t5xxl-FP16` and `flux1-dev-fp8` models for 30-step inference @1024px & H20 GPU:
+  - GPU memory usage: 27GB
+  - Inference time: 48 seconds (true_cfg=3.5), 26 seconds (true_cfg=1)
+
+- Different results can be achieved by adjusting the following parameters:
+
+| Parameter | Recommended Range | Effect |
+|-----------|------------------|--------|
+| control-strength | 0.6 - 1.0 | Controls how much influence the ControlNet has on the generation. Higher values result in stronger adherence to the control image. |
+| controlend-percent | 0.35 - 1.0 | Determines at which step in the denoising process the ControlNet influence ends. Lower values allow for more creative freedom in later steps. |
+| true-cfg (Classifier-Free Guidance Scale) | 1.0 or 3.5 | Influences how closely the generation follows the prompt. Higher values increase prompt adherence but may reduce image quality. |
+
+- More comprehensive full-image prompts can lead to better overall results. For example, in addition to describing the area to be repaired, you can also describe the background, atmosphere, and style of the entire image. This approach can make the generated results more harmonious and natural.
+
+## Diffusers Integration
+
+1. Install the required diffusers version:
+```shell
+pip install diffusers==0.30.2
+```
+
+2. Clone this repository:
+````shell
+git clone https://github.com/alimama-creative/FLUX-Controlnet-Inpainting.git
+````
+
+3. Configure `image_path`, `mask_path`, and `prompt` in `main.py`, then execute:
+````shell
+python main.py
+````
+
+## Model Specifications
+
+- Training dataset: 15M images from LAION2B and proprietary sources
+- Optimal inference resolution: 1024x1024
+
+## License
+
+Our model weights are released under the [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) Non-Commercial License.

config.json

@@ -0,0 +1,20 @@
+{
+  "_class_name": "FluxControlNetModel",
+  "_diffusers_version": "0.30.2",
+  "_name_or_path": "/data/oss_bucket_0/linjinpeng.ljp/exp_flux/r768_bs96_adamw_lr5e-6_bf16_cfg3.5_sin0_dou6_s11/checkpoint-50000",
+  "attention_head_dim": 128,
+  "axes_dims_rope": [
+    16,
+    56,
+    56
+  ],
+  "extra_condition_channels": 4,
+  "guidance_embeds": true,
+  "in_channels": 64,
+  "joint_attention_dim": 4096,
+  "num_attention_heads": 24,
+  "num_layers": 6,
+  "num_single_layers": 0,
+  "patch_size": 1,
+  "pooled_projection_dim": 768
+}

diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca46c5f7b5de02caee7c069f2aedbf628af8def8578319ceae3be1588d448448
+size 4281803800