WeDetect-Base (ONNX)

ONNX export of wedetect_base.pth and wedetect_base_uni.pth from fushh7/WeDetect.

Usage

import numpy as np
import onnxruntime as ort
from PIL import Image
from tokenizers import Tokenizer

providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
text = ort.InferenceSession("onnx/text_model_fp16.onnx", providers=providers)
vision = ort.InferenceSession("onnx/vision_model_fp16.onnx", providers=providers)
tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.enable_padding(pad_id=1, pad_token="<pad>")

labels = ["็Œซ", "็‹—", "ๆฑฝ่ฝฆ"]
enc = tokenizer.encode_batch(labels)
txt_feats = text.run(["txt_feats"], {"input_ids": np.array([e.ids for e in enc], np.int64),
                                      "attention_mask": np.array([e.attention_mask for e in enc], np.int64)})[0]

image = Image.open("image.jpg").convert("RGB")
r = 640 / max(image.size)
w, h = round(image.width * r), round(image.height * r)
canvas = Image.new("RGB", (640, 640), (114, 114, 114))
dx, dy = (640 - w) // 2, (640 - h) // 2
canvas.paste(image.resize((w, h), Image.BILINEAR), (dx, dy))
x = (np.asarray(canvas, np.float32) / 255).transpose(2, 0, 1)[None]

boxes, scores = vision.run(["bboxes", "scores"], {"image": x, "txt_feats": txt_feats})
for i in np.argsort(-scores[0].max(-1))[:5]:
    box = (boxes[0, i] - [dx, dy, dx, dy]) / r
    print(labels[scores[0, i].argmax()], round(float(scores[0, i].max()), 3), box.round(1))

proposal = ort.InferenceSession("onnx/proposal_model_fp16.onnx", providers=providers)
boxes, scores, region_embeddings = proposal.run(None, {"image": x})
top = np.argsort(-scores[0].max(-1))[:5]
print(((boxes[0, top] - [dx, dy, dx, dy]) / r).round(1), region_embeddings[0, top].shape)

Notes

  • NMS is not included in the graph; the Usage above skips it.
  • proposal_model is WeDetect-Base-Uni.
  • Files: *.onnx (fp32), *_fp16 (fp16 weights, float32 I/O), *_int8 (8-bit weight-only MatMulNBits, requires ONNX Runtime; lower accuracy than fp16). Opset 23 (ONNX Runtime 1.22+).
  • Changes from the original: exported the text encoder and the detector as separate graphs and the Uni model as proposal_model; NMS is left out; the computation is unchanged.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AX1Y2JP/WeDetect-Base-ONNX

Base model

fushh7/WeDetect
Quantized
(3)
this model