Instructions to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with Diffusers:
pip install -U diffusers transformers accelerate
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline controlnet = ControlNetModel.from_pretrained("akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0") pipe = StableDiffusionControlNetPipeline.from_pretrained( "MiniMaxAI/MiniMax-H3-Image", controlnet=controlnet ) - VideoX Fun
How to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with VideoX Fun:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
MiniMax-H3 + Fun ControlNet-Union 2.0 β video-model control, validated
TL;DR: this repo is the ControlNet-Union-2.0 branch working with the MiniMax-H3 video model β all 9 official demos reproduced, every condition validated, a ComfyUI workflow, and a repeated-frame recipe that turns the video stack into a still-image controlnet (canny/gray/HED).
What we tried that did NOT work: driving the MiniMax-H3-Image
model (the fast 6-step PDD-distilled image DiT) with this branch. The runtime wiring is verified
mathematically exact (scale-0 output is bit-identical to base), but the video-trained control
blocks collapse the spatial signal on the image model's distilled trajectory β it perturbs,
never steers (full evidence: assets/diag*, assets/image-dit-official/).
Future work: fine-tune the control branch on the H3-Image DiT (frozen DiT, zero-gated start β
all machinery in merge/).
Contents: MiniMaxAI/MiniMax-H3-Image packaged with the alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 control branch (8 union conditions + inpainting), plus verified, reproducible scripts and samples.
Showcase: extreme poses β man in a suit
Each row, left to right: a text-to-video source pose (woman in athletic wear), its canny
map driving the ControlNet-Union-2.0 video stack (22-frame repeated control, 40 steps,
guidance_scale=1.0, seed 904231), the no-control baseline at the same seed, and the
controlled output β the same extreme pose as a man in a tailored business suit in a different
setting. Structure comes from the control; identity, clothing and setting from the prompt.
Honest caveat (row 4): when the control carries a strong body silhouette, identity can follow the
source instead of the prompt. Full-res frames: assets/showcase2/.
Official card demos, reproduced
All 9 official demos from the upstream card, reproduced end-to-end: the official control
videos (alibaba-pai/.../asset/)
at exact card settings β 40 steps, guidance_scale=1.0, control_context_scale=1.0, seed 43,
704Γ1280 budget, 24 fps β one H200, one model load
(merge/card_repro_job.py). The upstream card publishes no prompts, so
ours are scene-matched. Mid-frames shown; compare with the official result videos on the
upstream card.
First frames, extracted control maps and every sample: assets/card-repro/.
More samples (same-seed on/off proof, static-frame coverage, diagnostics): assets/.
Reproduce / use
- All 9 demos: run
merge/card_repro_job.pyon an H200 (~1 h incl. downloads). - Your own control:
merge/expC_video_pipeline.pyβ repeat a control image into a 22-frame video and run the VideoX-Fun control pipeline; any output frame is a controlled image (~8 min on an H200). - All 8 conditions from one photo:
merge/coverage_job.py.
ComfyUI
ComfyUI β₯ 0.35 ships the core Apply MiniMax H3 Fun ControlNet node β no custom nodes
needed. Drag comfyui/workflows/minimax-h3-fun-controlnet-union-2.0.json
onto the canvas (the official template, adapted to load the 2.0 union patch).
| File | Goes in |
|---|---|
minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) |
models/diffusion_models/ |
minimax_h3_fun_controlnet_union_2.0_pruned_int8_convrot.safetensors (4.5 GB) β the 2.0 patch (bf16 also available) |
models/model_patches/ |
qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) |
models/text_encoders/ |
minimax_h3_video_vae_int8_convrot.safetensors + minimax_h3_audio_vae_fp32.safetensors |
models/vae/ |
Usage notes:
- Weights caveat: ComfyUI's pruned bases need the curve-form patches from Comfy-Org (table
above). The checkpoint in this repo's
controlnet/is the original full-width AdaLN release β use it with the VideoX-Fun pipeline (merge/scripts), not with ComfyUI's pruned models. strength=control_context_scale(1.0 strongest, 0 off); keep guidance at 1.0. Control is windowable viastart_percent/end_percent.- Frames snap to the 17n+5 grid at 24 fps. For single-image control, repeat the control image into a 22-frame clip β static frames steer for canny/gray/HED (see coverage below); the other conditions want real motion.
- Preprocessors are built in: Canny (Detect Edges), Depth (Depth Anything 3 / MoGe), Pose
(SDPose, wired in the template). Inpainting: connect
mask(+ optionalsource_video).
What's in this repo
controlnet/ ControlNet-Union-2.0 branch (13.6 GB) + config + license
transformer_diffusers/ H3-Image DiT (PDD, 6-step), unmodified
vae_diffusers/, tokenizer/, processor/, scheduler/, modular_model_index.json
merge/ wiring + reproduction + coverage scripts
comfyui/workflows/ ComfyUI template adapted for the 2.0 union patch
assets/ samples: card-repro, validation, diagnostics
Official controls on the H3-Image DiT (negative result)
The same 8 official control frames through the image DiT (runtime wiring,
merge/validate_controlnet.py), scene-matched prompts, seed 43:
outputs are excellent β but driven by the prompt, not the control. Base (no control) vs
controlled at the same seed:
PSNR(base, ctrl) per condition β large changes, none structurally aligned with the control
(consistent with the hint-collapse diagnostics in assets/diag2/):
| canny | depth | hed | mlsd | pose | scribble | layout | gray |
|---|---|---|---|---|---|---|---|
| 10.3 | 14.3 | 13.0 | 11.1 | 19.7 | 11.0 | 13.0 | 14.5 dB |
All 24 images (controls, base, controlled): assets/image-dit-official/.
Notes
- The ControlNet is a runtime branch of the MiniMax-H3 video DiT (10 blocks injecting
zero-gated skips at layers 0,5,β¦,45) β it cannot be folded into the base weights, and the
image model is not a ControlNet out of the box: on the H3-Image DiT (PDD-distilled, 6
steps) the branch attaches but does not steer without adaptation fine-tuning. Evidence and
diagnostics:
assets/validation*,assets/diag*. - For image use via the video pipeline, static repeated-frame controls transfer for canny and gray (HED partially); the other conditions need real moving control videos.
Sources & license
- Base: https://huggingface.co/MiniMaxAI/MiniMax-H3-Image Β· ControlNet: https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 Β· Code: https://github.com/aigc-apps/VideoX-Fun
- Both upstream models are under the MiniMax-H3 Community License Agreement (
LICENSE,LICENSE.controlnet); the same terms apply to this unofficial community repackaging. Not endorsed by MiniMax or the VideoX-Fun authors.
- Downloads last month
- 154











