WeDetect-Base (ONNX)
ONNX export of wedetect_base.pth and wedetect_base_uni.pth from fushh7/WeDetect.
Usage
import numpy as np
import onnxruntime as ort
from PIL import Image
from tokenizers import Tokenizer
providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
text = ort.InferenceSession("onnx/text_model_fp16.onnx", providers=providers)
vision = ort.InferenceSession("onnx/vision_model_fp16.onnx", providers=providers)
tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.enable_padding(pad_id=1, pad_token="<pad>")
labels = ["็ซ", "็", "ๆฑฝ่ฝฆ"]
enc = tokenizer.encode_batch(labels)
txt_feats = text.run(["txt_feats"], {"input_ids": np.array([e.ids for e in enc], np.int64),
"attention_mask": np.array([e.attention_mask for e in enc], np.int64)})[0]
image = Image.open("image.jpg").convert("RGB")
r = 640 / max(image.size)
w, h = round(image.width * r), round(image.height * r)
canvas = Image.new("RGB", (640, 640), (114, 114, 114))
dx, dy = (640 - w) // 2, (640 - h) // 2
canvas.paste(image.resize((w, h), Image.BILINEAR), (dx, dy))
x = (np.asarray(canvas, np.float32) / 255).transpose(2, 0, 1)[None]
boxes, scores = vision.run(["bboxes", "scores"], {"image": x, "txt_feats": txt_feats})
for i in np.argsort(-scores[0].max(-1))[:5]:
box = (boxes[0, i] - [dx, dy, dx, dy]) / r
print(labels[scores[0, i].argmax()], round(float(scores[0, i].max()), 3), box.round(1))
proposal = ort.InferenceSession("onnx/proposal_model_fp16.onnx", providers=providers)
boxes, scores, region_embeddings = proposal.run(None, {"image": x})
top = np.argsort(-scores[0].max(-1))[:5]
print(((boxes[0, top] - [dx, dy, dx, dy]) / r).round(1), region_embeddings[0, top].shape)
Notes
- NMS is not included in the graph; the Usage above skips it.
proposal_modelis WeDetect-Base-Uni.- Files:
*.onnx(fp32),*_fp16(fp16 weights, float32 I/O),*_int8(8-bit weight-onlyMatMulNBits, requires ONNX Runtime; lower accuracy than fp16). Opset 23 (ONNX Runtime 1.22+). - Changes from the original: exported the text encoder and the detector as separate graphs and the Uni model as
proposal_model; NMS is left out; the computation is unchanged.
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support
Model tree for AX1Y2JP/WeDetect-Base-ONNX
Base model
fushh7/WeDetect