Scene3D - Monocular Depth Estimation

Depth estimation is an essential technology for safe operation of self-driving cars, especially in challenging long-tail edge case scenarios. By sensing depth from a single front-view monocular camera, autonomous vehicles can detect obstacles in the scene irrespective of their object class, shape, size, or appearance. Scene3D processes monocular camera images to produce high-resolution per-pixel relative depth maps with sharp object boundaries — visible on foliage, thin structures such as poles, and foreground obstacles — enabling real-time scene geometry understanding on edge hardware.

Scene3D is distilled from DepthAnythingV2 ViT-Large pseudo-labels onto lightweight convolutional architectures specifically optimized for real-time on-road autonomous driving:

  • Scene3D: Encoder-Context-Decoder CNN combining a frozen road-pretrained EfficientNet-B0 backbone (SceneSegNetwork), a DepthContext residual attention module, a progressive upsampling decoder neck (Scene3DNeck), and a dense depth regression head (Scene3DHead).
  • Scene3DLite: Compact variant combining an EfficientNet-B1 encoder, a DeepLabV3+ decoder, and a lightweight regression head, reducing compute by ~29× (7.78 vs. 224 GOPs) while preserving edge sharpness, relative depth ordering, and scale consistency.

Performance Results

Scene3D was trained on 488,535 street-scene images (24,426 held out for validation) curated across BDD100K, Mapillary Vistas, ROADWork, Comma10K, KITTI, DDAD, Driving Stereo, Indian Driving Dataset, Zenesact Open Dataset, and Mapillary Planet Scale Dataset, optimizing a weighted sum of a Scale Invariant robust MAE Loss and a Multi-Scale Edge Preservation Loss.

Validation Set Performance (Scene3D, 640×320)

Total Loss Scale Invariant Loss Edge Preservation Loss
0.08408 0.07004 0.01404

Embedded Inference Performance — Jetson Orin Nano (8GB)

Forward pass includes Host→Device (H2D) + Inference + Device→Host (D2H) at 640×320 input resolution.

Model GOPs Backend Forward [ms] FPS
Scene3D 224 FP32 ONNX 168.4 5.9
Scene3D 224 FP32 TensorRT 99.7 10.0
Scene3DLite 7.78 FP32 ONNX 41.0 24.4
Scene3DLite 7.78 FP32 TensorRT 23.4 42.7
Scene3DLite 7.78 INT8 TensorRT 10.9 91.4

On a laptop GPU (RTX 3060 Mobile / AMD Ryzen 7 5800H, 238.41 GFLOPs), Scene3D achieves 15.1 FPS in FP32 and 23.9 FPS in FP16.

Model variants

The repository provides both the original Scene3D network and the lightweight Scene3DLite variant. We recommend Scene3DLite for embedded/real-time deployment and Scene3D for legacy VisionPilot 0.9 compatibility.

Variant Input frame PyTorch Traced PyTorch ONNX FP32
Scene3DLite 640x320 (2:1)* model weights model weights
Scene3D 640x320 (2:1) model weights model weights model weights

*The Scene3DLite PyTorch architecture supports dynamic input resolutions (e.g., 640×320 or 1024×512); the pre-exported scene3d_lite_fp32.onnx file uses [1, 3, 320, 640].

Quick start

PyTorch (Scene3D)

Clone the Scene3D GitHub repository, install dependencies (pip install -r requirements.txt huggingface_hub), and run:

import sys
from PIL import Image
from huggingface_hub import hf_hub_download

sys.path.append("Models")
from Models.inference.scene_3d_infer import Scene3DNetworkInfer

repo_id = "AutowareFoundation/scene_3d"
ckpt = hf_hub_download(repo_id, "models/scene3d.pth")
model = Scene3DNetworkInfer(ckpt)          # uses CUDA if available
test_img = hf_hub_download(repo_id, "assets/highway.jpg")

img = Image.open(test_img).convert("RGB").resize((640, 320))
depth_map = model.inference(img)           # shape: (320, 640, 1)
print(depth_map.shape, depth_map.min(), depth_map.max())

ONNX Runtime (Scene3DLite / Scene3D, standalone)

import numpy as np
import onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

repo_id = "AutowareFoundation/scene_3d"
onnx_path = hf_hub_download(repo_id, "models/scene3d_lite_fp32.onnx")
test_img = hf_hub_download(repo_id, "assets/highway.jpg")

img = Image.open(test_img).convert("RGB").resize((640, 320))
x = np.asarray(img, dtype=np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
x = ((x - mean) / std).transpose(2, 0, 1)[None, ...]

session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
depth_map = session.run(None, {"input": x})[0]   # shape: (1, 1, 320, 640)
print(depth_map.shape, depth_map.min(), depth_map.max())

Citation

@software{autoware_scene_3d_2026,
  author    = {The Autoware Foundation},
  title     = {Scene3D: Monocular relative depth estimation model},
  year      = {2026},
  url       = {https://github.com/autowarefoundation/scene_3d},
  note      = {Checkpoints: https://huggingface.co/AutowareFoundation/scene_3d}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AutowareFoundation/scene_3d

Quantized
(14)
this model