Camera-Depth-Model-D435 — transformers format (original code)

Unofficial transformers port of the Camera Depth Model (CDM) depth-anything/camera-depth-model-d435 (cdm_d435.ckpt) that runs the original code. Loads with trust_remote_code. model.safetensors holds the checkpoint's model tensors under their original names (every tensor checked equal). Trained for the Intel RealSense D435 (camera-specific noise model).

Not an official release. Camera Depth Models were created by Minghuan Liu, Zhengbang Zhu, Xiaoshen Han, Peng Hu, Haotong Lin, Xinyao Li, Jingxiao Chen, Jiafeng Xu, Yichu Yang, Yunfeng Lin, Xinghang Li, Yong Yu, Weinan Zhang, Tao Kong and Bingyi Kang (ByteDance Seed) — https://github.com/ByteDance-Seed/manip-as-in-sim-suite, paper "Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots" (arXiv 2509.02530).

What it is

Denoised metric depth from an RGB image plus the raw depth of a depth camera. CameraDepthModelForDepthEstimation holds the official RGBDDepth's submodules (pretrained / depth_pretrained: two DINOv2 ViT-L/14, crossAtts, depth_head_rgbd) and computes depth with that class's own forward, returning the standard DepthEstimatorOutput (predicted_depth: inverse depth; meters = 1 / predicted_depth, as the original infer.py); the official infer_image and image2tensor are kept. Code vendored from ByteDance-Seed/manip-as-in-sim-suite @ 757b2ea2 (cdm/rgbddepth/) into cdm_dinov2.py / cdm_dpt.py; edits: imports, and Block → NestedTensorBlock where the original imported NestedTensorBlock as Block. CameraDepthModelProcessor (processing_camera_depth_model.py, processor_config.json) wraps CameraDepthModelImageProcessor: infer.py's load_images (raw depth / 1000, values above 25 m set to 0, inverse depth) and image2tensor.

As in the original infer.py and README API, the image is given as RGB and image2tensor applies cv2.COLOR_BGR2RGB to it, so the network sees the channels swapped. This port keeps that behaviour unchanged.

Verified against the official code (same process, V100): preprocessing (image and depth), predicted_depth (also as a tuple), the hidden states, infer_image, the post-processing and fp16 autocast outputs are bit-identical.

Usage

import torch
from transformers import AutoModelForDepthEstimation, AutoProcessor

model = AutoModelForDepthEstimation.from_pretrained("Camera-Depth-Model-D435", trust_remote_code=True).eval().cuda()
processor = AutoProcessor.from_pretrained("Camera-Depth-Model-D435", trust_remote_code=True)
# image: path / PIL / RGB uint8 array; depth: the camera's raw depth png (millimeters, depth_scale=1000) or meters
inputs = processor(images="rgb.png", depth="depth.png", return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model(**inputs)                       # outputs.predicted_depth: (1, H', W') inverse depth
inverse = processor.post_process_depth_estimation(outputs, target_sizes=[(height, width)])[0]["predicted_depth"]
depth_m = 1 / inverse

• AutoModel loads the same class. output_hidden_states=True adds the DINOv2 features of the RGB and depth branches; return_dict=False returns a tuple.
• post_process_depth_estimation resizes as the original infer_image (nearest).
• depth_scale / max_depth of the processor: the original's --depth-scale (1000) and the 25 m cut-off infer.py uses.

Runtime packages beyond torch / transformers: opencv-python, torchvision (as the original).

Licensing

• Weights (depth-anything/camera-depth-model-d435): CC-BY-NC-4.0 (non-commercial) — LICENSE.
• Code (vendored files and the port): Apache-2.0 — LICENSE.code.
• Attribution, sources and the changes made here: NOTICE.

Citation

If you use this model, please cite the original work, as its authors ask:

@article{liu2025manipulation,
  title={Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots},
  author={Liu, Minghuan and Zhu, Zhengbang and Han, Xiaoshen and Hu, Peng and Lin, Haotong and 
          Li, Xinyao and Chen, Jingxiao and Xu, Jiafeng and Yang, Yichu and Lin, Yunfeng and 
          Li, Xinghang and Yu, Yong and Zhang, Weinan and Kong, Tao and Kang, Bingyi},
  journal={arXiv preprint},
  year={2025}
}