MiniMax-H3 + Fun ControlNet-Union 2.0 β€” video-model control, validated

TL;DR: this repo is the ControlNet-Union-2.0 branch working with the MiniMax-H3 video model β€” all 9 official demos reproduced, every condition validated, a ComfyUI workflow, and a repeated-frame recipe that turns the video stack into a still-image controlnet (canny/gray/HED).

What we tried that did NOT work: driving the MiniMax-H3-Image model (the fast 6-step PDD-distilled image DiT) with this branch. The runtime wiring is verified mathematically exact (scale-0 output is bit-identical to base), but the video-trained control blocks collapse the spatial signal on the image model's distilled trajectory β€” it perturbs, never steers (full evidence: assets/diag*, assets/image-dit-official/).

Future work: fine-tune the control branch on the H3-Image DiT (frozen DiT, zero-gated start β€” all machinery in merge/).

Contents: MiniMaxAI/MiniMax-H3-Image packaged with the alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 control branch (8 union conditions + inpainting), plus verified, reproducible scripts and samples.

Showcase: extreme poses β†’ man in a suit

pose showcase grid

Each row, left to right: a text-to-video source pose (woman in athletic wear), its canny map driving the ControlNet-Union-2.0 video stack (22-frame repeated control, 40 steps, guidance_scale=1.0, seed 904231), the no-control baseline at the same seed, and the controlled output β€” the same extreme pose as a man in a tailored business suit in a different setting. Structure comes from the control; identity, clothing and setting from the prompt. Honest caveat (row 4): when the control carries a strong body silhouette, identity can follow the source instead of the prompt. Full-res frames: assets/showcase2/.

Official card demos, reproduced

All 9 official demos from the upstream card, reproduced end-to-end: the official control videos (alibaba-pai/.../asset/) at exact card settings β€” 40 steps, guidance_scale=1.0, control_context_scale=1.0, seed 43, 704Γ—1280 budget, 24 fps β€” one H200, one model load (merge/card_repro_job.py). The upstream card publishes no prompts, so ours are scene-matched. Mid-frames shown; compare with the official result videos on the upstream card.

Condition Control Reproduced frame
Canny astronaut edges g
Depth dog + cat depth g
HED turtle edges g
MLSD architectural lines g
Pose dancer skeleton g
Scribble 3-student sketch g
Layout colored boxes g
Gray coral luminance g
Inpaint source + mask g

First frames, extracted control maps and every sample: assets/card-repro/. More samples (same-seed on/off proof, static-frame coverage, diagnostics): assets/.

Reproduce / use

ComfyUI

ComfyUI β‰₯ 0.35 ships the core Apply MiniMax H3 Fun ControlNet node β€” no custom nodes needed. Drag comfyui/workflows/minimax-h3-fun-controlnet-union-2.0.json onto the canvas (the official template, adapted to load the 2.0 union patch).

File Goes in
minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) models/diffusion_models/
minimax_h3_fun_controlnet_union_2.0_pruned_int8_convrot.safetensors (4.5 GB) β€” the 2.0 patch (bf16 also available) models/model_patches/
qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) models/text_encoders/
minimax_h3_video_vae_int8_convrot.safetensors + minimax_h3_audio_vae_fp32.safetensors models/vae/

Usage notes:

  • Weights caveat: ComfyUI's pruned bases need the curve-form patches from Comfy-Org (table above). The checkpoint in this repo's controlnet/ is the original full-width AdaLN release β€” use it with the VideoX-Fun pipeline (merge/ scripts), not with ComfyUI's pruned models.
  • strength = control_context_scale (1.0 strongest, 0 off); keep guidance at 1.0. Control is windowable via start_percent/end_percent.
  • Frames snap to the 17n+5 grid at 24 fps. For single-image control, repeat the control image into a 22-frame clip β€” static frames steer for canny/gray/HED (see coverage below); the other conditions want real motion.
  • Preprocessors are built in: Canny (Detect Edges), Depth (Depth Anything 3 / MoGe), Pose (SDPose, wired in the template). Inpainting: connect mask (+ optional source_video).

What's in this repo

controlnet/                 ControlNet-Union-2.0 branch (13.6 GB) + config + license
transformer_diffusers/      H3-Image DiT (PDD, 6-step), unmodified
vae_diffusers/, tokenizer/, processor/, scheduler/, modular_model_index.json
merge/                      wiring + reproduction + coverage scripts
comfyui/workflows/          ComfyUI template adapted for the 2.0 union patch
assets/                     samples: card-repro, validation, diagnostics

Official controls on the H3-Image DiT (negative result)

The same 8 official control frames through the image DiT (runtime wiring, merge/validate_controlnet.py), scene-matched prompts, seed 43: outputs are excellent β€” but driven by the prompt, not the control. Base (no control) vs controlled at the same seed:

base_canny ctrl_canny
b c

PSNR(base, ctrl) per condition β€” large changes, none structurally aligned with the control (consistent with the hint-collapse diagnostics in assets/diag2/):

canny depth hed mlsd pose scribble layout gray
10.3 14.3 13.0 11.1 19.7 11.0 13.0 14.5 dB

All 24 images (controls, base, controlled): assets/image-dit-official/.

Notes

  • The ControlNet is a runtime branch of the MiniMax-H3 video DiT (10 blocks injecting zero-gated skips at layers 0,5,…,45) β€” it cannot be folded into the base weights, and the image model is not a ControlNet out of the box: on the H3-Image DiT (PDD-distilled, 6 steps) the branch attaches but does not steer without adaptation fine-tuning. Evidence and diagnostics: assets/validation*, assets/diag*.
  • For image use via the video pipeline, static repeated-frame controls transfer for canny and gray (HED partially); the other conditions need real moving control videos.

Sources & license

Downloads last month
154
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0