Model card: JevBench API board result (#1 of 48) and decision task scores

#2
by Eljaja - opened
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -11,6 +11,24 @@ tags: [jev, systemone, typed-decisions, decision-model, diffusion-lm, calibratio
11
  question in a single read: `choice` (a probability per option), `noul` (P(yes)) and `score` (expected level over an
12
  ordered rubric, with the distribution).
13
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  ## Model
15
  - **Base:** `google/diffusiongemma-26B-A4B-it` (discrete-diffusion MoE, 26B total, 4B active), served from NVIDIA's
16
  NVFP4 checkpoint `nvidia/diffusiongemma-26B-A4B-it-NVFP4`.
 
11
  question in a single read: `choice` (a probability per option), `noul` (P(yes)) and `score` (expected level over an
12
  ordered rubric, with the distribution).
13
 
14
+ ## Results
15
+
16
+ Super Decider is #1 of 48 hosted decision APIs on the [JevBench API leaderboard](https://benchmarkheaven.com/jev-models/api)
17
+ by Benchmark Heaven (board v1.7.48, 24 Oct 2026), measured through our hosted API on 21 Oct 2026 over 1,500 decisions,
18
+ 1,200 of them sealed.
19
+
20
+ | System | Composite | Capability | Cost per 1,000 decisions | Median latency |
21
+ |---|---|---|---|---|
22
+ | Super Decider | **78.5** (#1) | 84.9 | $0.016 | 0.25 s |
23
+ | Super Decider, thinking always | 74.5 (#8) | 85.9 | $0.028 | 0.89 s |
24
+ | kettei-fat | 77.7 (#2) | **86.3** | $0.026 | 0.20 s |
25
+ | Jev 1.13 | 71.4 (#14) | 77.1 | $0.032 | 0.24 s |
26
+
27
+ Decision tasks from the [Jev Decision Index 0.3](https://huggingface.co/spaces/multimodalart/jev-decision-index) suite,
28
+ every row of each task, scored with the Decision Index kit: When2Call 85.6 (Jev 1.14: 81.7), PhishNChips 85.9 (83.8),
29
+ BFCL tool selection 96.9 (96.4). [DecisionBench](https://github.com/Hanno-Labs/decision-bench), 23,900 decisions: 75.1%
30
+ (Jev 1.13: 72.0%).
31
+
32
  ## Model
33
  - **Base:** `google/diffusiongemma-26B-A4B-it` (discrete-diffusion MoE, 26B total, 4B active), served from NVIDIA's
34
  NVFP4 checkpoint `nvidia/diffusiongemma-26B-A4B-it-NVFP4`.