| --- |
| license: apache-2.0 |
| --- |
| # Qwen-Image Image Structure Control Model - Depth ControlNet |
|
|
|  |
|
|
| ## Model Introduction |
|
|
| This model is an image structure control model based on [Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image), with a ControlNet architecture that enables structural control of generated images using depth maps. The training framework is built upon [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), and the dataset used for training is [BLIP3o](https://modelscope.cn/datasets/BLIP3o/BLIP3o-60k). |
|
|
| ## Result Demonstration |
|
|
| |Depth Map|Generated Image 1|Generated Image 2| |
| |-|-|-| |
| |||| |
| |||| |
| |||| |
|
|
| ## Inference Code |
| ``` |
| git clone https://github.com/modelscope/DiffSynth-Studio.git |
| cd DiffSynth-Studio |
| pip install -e . |
| ``` |
|
|
| ```python |
| from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig, ControlNetInput |
| from PIL import Image |
| import torch |
| from modelscope import dataset_snapshot_download |
| |
| |
| pipe = QwenImagePipeline.from_pretrained( |
| torch_dtype=torch.bfloat16, |
| device="cuda", |
| model_configs=[ |
| ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"), |
| ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"), |
| ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), |
| ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth", origin_file_pattern="model.safetensors"), |
| ], |
| tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"), |
| ) |
| |
| dataset_snapshot_download( |
| dataset_id="DiffSynth-Studio/example_image_dataset", |
| local_dir="./data/example_image_dataset", |
| allow_file_pattern="depth/image_1.jpg" |
| ) |
| |
| controlnet_image = Image.open("data/example_image_dataset/depth/image_1.jpg").resize((1328, 1328)) |
| ``` |
|
|
| prompt = "Exquisite portrait, underwater girl, flowing blue dress, gently floating hair, translucent lighting, surrounded by bubbles, serene expression, intricate details, dreamy and ethereal." |
| image = pipe( |
| prompt, seed=0, |
| blockwise_controlnet_inputs=[ControlNetInput(image=controlnet_image)] |
| ) |
| image.save("image.jpg") |
