Model card: JevBench API board result (#1 of 48) and decision task scores
#2
by Eljaja - opened
README.md
CHANGED
|
@@ -11,6 +11,24 @@ tags: [jev, systemone, typed-decisions, decision-model, diffusion-lm, calibratio
|
|
| 11 |
question in a single read: `choice` (a probability per option), `noul` (P(yes)) and `score` (expected level over an
|
| 12 |
ordered rubric, with the distribution).
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
## Model
|
| 15 |
- **Base:** `google/diffusiongemma-26B-A4B-it` (discrete-diffusion MoE, 26B total, 4B active), served from NVIDIA's
|
| 16 |
NVFP4 checkpoint `nvidia/diffusiongemma-26B-A4B-it-NVFP4`.
|
|
|
|
| 11 |
question in a single read: `choice` (a probability per option), `noul` (P(yes)) and `score` (expected level over an
|
| 12 |
ordered rubric, with the distribution).
|
| 13 |
|
| 14 |
+
## Results
|
| 15 |
+
|
| 16 |
+
Super Decider is #1 of 48 hosted decision APIs on the [JevBench API leaderboard](https://benchmarkheaven.com/jev-models/api)
|
| 17 |
+
by Benchmark Heaven (board v1.7.48, 24 Oct 2026), measured through our hosted API on 21 Oct 2026 over 1,500 decisions,
|
| 18 |
+
1,200 of them sealed.
|
| 19 |
+
|
| 20 |
+
| System | Composite | Capability | Cost per 1,000 decisions | Median latency |
|
| 21 |
+
|---|---|---|---|---|
|
| 22 |
+
| Super Decider | **78.5** (#1) | 84.9 | $0.016 | 0.25 s |
|
| 23 |
+
| Super Decider, thinking always | 74.5 (#8) | 85.9 | $0.028 | 0.89 s |
|
| 24 |
+
| kettei-fat | 77.7 (#2) | **86.3** | $0.026 | 0.20 s |
|
| 25 |
+
| Jev 1.13 | 71.4 (#14) | 77.1 | $0.032 | 0.24 s |
|
| 26 |
+
|
| 27 |
+
Decision tasks from the [Jev Decision Index 0.3](https://huggingface.co/spaces/multimodalart/jev-decision-index) suite,
|
| 28 |
+
every row of each task, scored with the Decision Index kit: When2Call 85.6 (Jev 1.14: 81.7), PhishNChips 85.9 (83.8),
|
| 29 |
+
BFCL tool selection 96.9 (96.4). [DecisionBench](https://github.com/Hanno-Labs/decision-bench), 23,900 decisions: 75.1%
|
| 30 |
+
(Jev 1.13: 72.0%).
|
| 31 |
+
|
| 32 |
## Model
|
| 33 |
- **Base:** `google/diffusiongemma-26B-A4B-it` (discrete-diffusion MoE, 26B total, 4B active), served from NVIDIA's
|
| 34 |
NVFP4 checkpoint `nvidia/diffusiongemma-26B-A4B-it-NVFP4`.
|