JevAny-Robotics-4B-LoRA (v0.4)

A 4B Jev-style (System One) judge for robot manipulation: one forward pass returns a probability for every option (task progress, 1–5 rollout scores, execution success, failure mode, plan checking, failure localisation, trajectory preference). It is a LoRA adapter and pointer head on Qwen/Qwen3.5-4B, trained with JevAny on judgement questions over real and simulated robot images and videos, built only from train-side splits and decontaminated against every test set below. The previous release stays available as revision v0.2.

Training data and time: 289,311 questions (about 400M tokens), one epoch, about 43 hours on one or two A100-40GB GPUs (69 GPU-hours).

Non-commercial use only. Part of the training data is licensed for non-commercial use only (airoa-moma and AgiBot World 2026: CC BY-NC-SA 4.0; RH20T scenes 6–10: CC BY-NC 4.0) and the VLAC-Cut data carry no license terms, so the weights are released under CC BY-NC-SA 4.0 and must not be used commercially. All training sources and their licenses are listed in ACKNOWLEDGEMENTS.md; the training set is SimpleJev/JevAny-Robotics-Train-v0.4, with the source of every question (its VLAC-Cut rows are rebuilt with a script).

All numbers use the serving configuration we recommend (native resolution, every video frame, JevAny's acceleration flags; see Usage). We ran every other model ourselves, in the same harness and on the same items.

Highlights

• Small and fast: 4B parameters on one A100-40GB; a median 165 ms per RoboRewardBench video, 117 ms per RoboPulse pair and 83 ms per RoboPulse++ pair, faster than every other model we timed on RoboPulse and RoboPulse++.
• Best on real robots: on RoboArena's real-robot rollouts its MAE is 0.516, the lowest of all models we ran (Flora-Rewarder-27B 0.708, Claude Opus 5.5 0.770, RoboReward-8B 0.968), and it orders policies like human raters 91.1% of the time, also the highest.
• RoboRewardBench overall: MAE 0.309, second only to Flora-Rewarder-27B (0.225) and ahead of RoboReward-8B (0.358) and Claude Opus 5.5 (0.492).
• Progress between two observations: RoboPulse AUROC 0.915, behind only Robo-Dopamine, whose training data contain the test episodes; RoboPulse++ AUROC 0.732, statistically tied for first.
• Probabilities you can use: the lowest NLL of the models we measured (0.779), and 36% of the videos get a single-score conformal interval that is right 94.6% of the time.

Robot benchmarks

RoboRewardBench (2,831 videos, 1–5 progress scores)

Model	Overall ↓	RoboArena ↓	OXE ↓	MSE ↓	exact ↑	within 1 ↑	Kendall τ ↑	QWK ↑	NLL ↓
JevAny-Robotics-4B-LoRA v0.4 (this model)	0.309	0.516	0.300	0.567	70.8%	93.4%	0.800	0.863	0.779
Flora-Rewarder-27B	0.225	0.708	0.203	0.656	72.7%	92.7%	0.790	0.844	0.858
RoboReward-8B	0.358	0.968	0.330	1.059	63.5%	86.6%	0.688	0.754	1.388
RoboReward-4B	0.407	0.999	0.380	1.205	62.6%	85.0%	0.654	0.720	1.212
JevAny-Robotics-4B-LoRA v0.2	0.443	0.687	0.432	0.888	62.1%	89.0%	0.711	0.788	1.021
Claude Opus 5.5 (zero-shot)	0.492	0.770	0.480	0.904	54.8%	89.7%	0.728	0.786	–
Claude Haiku 5.5 (zero-shot)	0.674	0.848	0.666	1.237	46.5%	85.2%	0.655	0.675	–
Claude Sonnet 5.5 (zero-shot)	0.685	0.857	0.677	1.243	44.7%	84.4%	0.666	0.700	–
JevAny-Qwen3.8-27B-LoRA	0.692	0.867	0.684	1.553	51.4%	82.7%	0.609	0.667	1.158
Robometer-4B ᶜ	0.777	0.834	0.774	1.275	40.6%	83.7%	0.549	0.652	–
OneJev-27B	0.805	0.919	0.800	1.769	47.3%	77.5%	0.593	0.657	1.252
Qwen3-VL-4B-Instruct zero-shot	0.945	0.872	0.948	1.736	36.9%	76.1%	0.505	0.470	1.978
TOPReward (Qwen3-VL-8B-Instruct) ᶜ	0.965	0.968	0.965	1.602	30.9%	76.3%	0.437	0.501	–
Robo-Dopamine-GRM-2.0-4B ᵉ	0.976	0.950	0.977	1.620	30.9%	79.2%	0.445	0.491	–
Qwen3.8-27B zero-shot	0.992	1.106	0.987	2.389	41.4%	70.7%	0.543	0.557	1.540
Robo-Dopamine-GRM-2.0-8B ᵉ	1.023	0.928	1.028	1.691	30.4%	77.6%	0.416	0.451	–
RynnValue-4B ᶜ	1.193	1.031	1.200	2.033	24.0%	69.2%	0.264	0.259	–
DenseReward-1frame ᶜ	1.239	1.219	1.240	2.576	25.2%	69.4%	0.197	0.217	–

Overall: the official group-wise MAE (mean over RoboArena and the 22 OXE subsets), from each model's most likely score; the other columns are over all 2,831 videos. ᶜ ᵉ: 1–5 levels cut from a continuous output (see How the other models were run). Flora-Rewarder-27B leads on Overall (Flora − this model: −0.084 [−0.110, −0.018]) because of the OXE clips cut before the end (below); per video the two tie (MAE 0.373 vs 0.377), and on RoboArena this model is clearly better (this model − Flora: −0.192 [−0.273, −0.118]). Against every other model in the table this model is significantly better on Overall and Kendall τ (paired cluster bootstrap over task groups).

Item type (videos)	This model	Flora-Rewarder-27B	RoboReward-8B
RoboArena, real robot (1,000)	0.516 / 60.2%	0.708 / 53.5%	0.968 / 42.2%
OXE, successful clip (371)	0.189 / 87.1%	0.189 / 86.8%	0.310 / 81.4%
OXE, relabelled with another instruction (669)	0.191 / 85.2%	0.143 / 86.8%	0.223 / 81.9%
OXE, cut before the end (791)	0.431 / 64.3%	0.243 / 78.4%	0.418 / 66.5%

On RoboArena, Flora-Rewarder-27B scores 19.8% of the failed or partial rollouts as successes (this model 7.6%), and this model scores more of the real successes below 5 (25.0% vs 9.0%).

Robot reward models

Model	Size	RoboPulse acc ↑	RoboPulse AUROC ↑	RoboPulse++ AUROC ↑	RoboPulse++ bal. acc ↑	RoboRewardBench Overall ↓	RoboArena MAE ↓
JevAny-Robotics-4B-LoRA v0.4 (this model)	4B	80.9	0.915	0.732	63.4	0.309	0.516
Flora-Rewarder-27B	27B	60.7ᶠ	0.676ᶠ	0.590ᶠ	57.2ᶠ	0.225	0.708
RoboReward-8B	8B	49.9ᵃ	0.499ᵃ	0.489ᵃ	50.0ᵃ	0.358	0.968
RoboReward-4B	4B	49.6ᵃ	0.501ᵃ	0.507ᵃ	51.6ᵃ	0.407	0.999
Robometer-4B	4B	63.6ᵇ	0.700ᵇ	0.630ᵇ	58.6ᵇ	0.777ᶜ	0.834ᶜ
Robo-Dopamine-GRM-2.0-8B	8B	90.1ʳ	0.953ʳ	0.717ᵈ	68.6ᵈ	1.023ᵉ	0.928ᵉ
Robo-Dopamine-GRM-2.0-4B	4B	91.5ʳ	0.956ʳ	0.728ᵈ	69.8ᵈ	0.976ᵉ	0.950ᵉ
RynnValue-4B	4B	70.5	0.781	0.711	63.7	1.193ᶜ	1.031ᶜ
TOPReward (Qwen3-VL-8B-Instruct)	8B	55.6	0.576	0.553	53.6	0.965ᶜ	0.968ᶜ
DenseReward-1frame	4B	56.6	0.598	0.571	55.9	1.239ᶜ	1.219ᶜ

ᵃ ᶠ Whole-rollout scorers: pairs scored as a 2-frame video in both orders, an out-of-scope use. ᵇ Progress difference, both orders. ᶜ ᵉ 1–5 levels cut from a continuous output. ᵈ Incremental-hop readout (position-debiased: 8B 0.730, 4B 0.747). ʳ In-distribution: Robo-Dopamine's public training data contain the target episode of 1,796 of the 1,800 RoboPulse items. In paired bootstraps (AUROC over episodes, Overall MAE over task groups) this model is significantly better than every model here except Robo-Dopamine on RoboPulse, Robo-Dopamine and RynnValue-4B on RoboPulse++ (ties) and Flora-Rewarder-27B on Overall MAE.

Ranking policies (RoboArena)

Model	Agreement with human ranking [95% CI]
JevAny-Robotics-4B-LoRA v0.4	91.1% [88.4, 93.6]
OneJev-27B	88.8% [85.7, 91.8]
Qwen3.8-27B zero-shot	88.7% [86.1, 91.3]
Flora-Rewarder-27B	88.7% [85.6, 91.7]
JevAny-Robotics-4B-LoRA v0.2	88.4% [85.3, 91.4]
JevAny-Qwen3.8-27B-LoRA	88.3% [85.2, 91.3]
RoboReward-8B	85.3% [81.9, 88.5]
Qwen3-VL-4B-Instruct zero-shot	84.3% [81.3, 87.6]
Claude Opus 5.5	84.2% [81.6, 86.6]
RoboReward-4B	83.7% [79.9, 87.4]
Claude Haiku 5.5	82.5% [79.7, 85.4]
Claude Sonnet 5.5	80.4% [77.6, 83.4]
Robometer-4B (last-frame progress)	79.6% [75.4, 83.7]
TOPReward (Qwen3-VL-8B-Instruct; success readout)	79.3% [75.9, 83.0]
Robo-Dopamine-8B (progress hop, first → last frame)	78.3% [74.2, 82.3]
Robo-Dopamine-4B (progress hop, first → last frame)	78.0% [73.9, 81.8]
RynnValue-4B (progress from the remaining time)	67.1% [61.7, 72.5]
DenseReward-1frame (last-frame reward)	63.6% [58.9, 68.0]

805 pairs of policies run on the same task in the same session (134 sessions), with different human scores. Each policy gets the mean over its camera views of the expected score (models with probabilities), the continuous output (models scored through cut points) or the score; ties count one half; CIs from a cluster bootstrap over sessions. The lead is significant against every model except the four 27B models.

Other judges and base VLMs

Jev-style judges and zero-shot base VLMs on the three benchmarks

Model	Size	RoboPulse acc ↑	RoboPulse AUROC ↑	RoboPulse++ AUROC ↑	RoboPulse++ bal. acc ↑	RoboRewardBench Overall ↓
JevAny-Robotics-4B-LoRA v0.4 (this model)	4B	80.9	0.915	0.732	63.4	0.309
JevAny-Robotics-4B-LoRA v0.2 (previous release)	4B	71.1	0.806	0.728	65.9	0.443
JevAny-Qwen3.5-4B-LoRA	4B	52.7	0.588	0.641	59.1	1.057
Qwen3.5-4B zero-shot (constrained choice)	4B	49.9	0.519	0.592	50.0	1.261
OneJev-4B	4B	53.9	0.586	0.617	58.3	1.005
Intern-Decision-4B	4B	51.6	0.547	0.653	60.5	0.820
OmniJev-4B	4B	49.7	0.536	0.556	54.6	0.811*
OneJev-9B	9B	57.6	0.595	0.564	54.0	0.927
Jev-Omni-12B	12B	49.4	0.492	0.537	52.3	1.085
Qwen3.8-27B zero-shot (constrained choice)	27B	56.7	0.604	0.569	55.2	0.992
JevAny-Qwen3.8-27B-LoRA	27B	56.6	0.627	0.692	55.7	0.692
OneJev-27B	27B	60.4	0.651	0.700	62.7	0.805

RoboPulse: 1,800 pairs; RoboPulse++: 7,720 consecutive-step pairs (81% rising); native resolution. *OmniJev-4B's training data overlap RoboRewardBench. In paired bootstraps (RoboPulse accuracy and AUROC, RoboPulse++ AUROC, RoboRewardBench Overall) this model is significantly better than every judge we tested, except RoboPulse++ AUROC against v0.2 and the two 27B judges (ties); OmniJev-4B, OneJev-9B, Jev-Omni-12B and Qwen3.8-27B zero-shot were not tested pairwise. Reports with all slices: eval/comparison/.

How the other models were run

Inference, readouts and adaptations (all fixed before scoring the test sets)

• RoboReward-8B / 4B (teetone/RoboReward-*): model-card prompt verbatim, assistant turn prefilled with ANSWER: , softmax over the digit tokens 1–5; every frame of the same derived clips. Free generation gives the same RoboRewardBench MAE (0.357 vs 0.358 for 8B, 0 unparsable). ᵃ RoboReward scores whole rollouts; for the pairwise benchmarks we score the 2-frame video (before, after) minus (after, before). It was not trained for pairs, so treat these as an out-of-scope use.
• Robometer-4B (robometer/Robometer-4B, official code): ᵇ progress difference between the two observations, with the two frames also scored in reversed order to remove a position bias. ᶜ 1–5 levels from the last-frame progress (for the other models marked ᶜ, from the continuous output given in their entries), with cut points fitted on a label-stratified subset of the RoboReward validation split (never the test set). Its training data include the RoboArena test videos of RoboRewardBench.
• Robo-Dopamine-GRM-2.0-8B (tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview): the inference of PRM-as-a-Judge's GRMInference (vLLM, same prompt, sampling and model length), its native 8-image layout (reference start / end + three views before and after). ʳ RoboPulse is in-distribution for this model: its public training data (GRM json_sel) contain the target episode of 1,796 of the 1,800 items, 440 of them with the same frame pair, and RoboPulse uses its training input format; we found no clean subset. ᵈ RoboPulse++: hop from the earlier to the later observation, reference start = the episode's first observation, blank goal image (the model card's default when no goal image exists), missing views repeated as in the model card; 146 of 23,559 requests exceeded the 8,192-token limit and were rerun with images downscaled to ≤76,800 px. Its other readouts: forward 0.698, position-debiased 0.730. ᵉ RoboRewardBench: hop from the first to the last frame, mapped to 1–5 with cut points fitted on the RoboReward validation subset.
• Robo-Dopamine-GRM-2.0-4B (tanhuajie2001/Robo-Dopamine-GRM-2.0-4B-Preview): exactly as the 8B (same code, prompt, sampling, layouts, readouts and its own validation cut points), trained on the same public GRM data, so RoboPulse is in-distribution for it too (ʳ). Its other readouts: RoboPulse position-debiased 0.962; RoboPulse++ forward 0.697, position-debiased 0.747. BAAI/RoboBrain2.5-4B, whose model card points to Robo-Dopamine for progress estimation, ships byte-identical weights; we ran it as well and it reproduces these numbers within vLLM's sampling noise (RoboPulse AUROC 0.956, RoboPulse++ 0.731).
• Flora-Rewarder-27B (nvidia/Flora-Rewarder, Qwen3.5-27B, NVIDIA OneWay Noncommercial License): the model repository's own terminal-reward inference (batch_utils, vLLM, two A100-40GB with tensor parallelism): its score-only prompt with thinking off, its Score: 1-5 structured output, P(1)…P(5) read from the logprobs at the score token, its frame sampler (up to 64 frames over the episode, about 10 per second of video; our clips have about 1 per second, so frames repeat) and its frame size of about 100k pixels. To load the checkpoint in vLLM 0.30 we skip its 256 Transformer-Engine _extra_state entries (FP8 metadata, no weights). ᶠ Pairwise benchmarks: the earlier and the later observation (first view) as a 2-frame, 1-fps video in both orders, P(rising) = sigmoid((E_fwd − E_rev) / 0.5), the rule we use for RoboReward. Like RoboReward, Flora scores whole rollouts and was not built for pairs: both orders get the same top score for 86% of the RoboPulse and 94% of the RoboPulse++ pairs. Treat these as an out-of-scope use, not as a measure of its progress judgement. Its training data are not itemized (open-source, licensed and in-house robot video), so we cannot check them for overlap with these test sets. Its paper reports a RoboRewardBench MAE of 0.400; we measure 0.377 over all 2,831 videos (micro average) and 0.225 as the official group-wise Overall.
• RynnValue-4B (Alibaba-DAMO-Academy/RynnValue-4B): the model card's programmatic inference (causal SDPA attention, BF16, process_episode, every frame of the clip), remaining time to completion per frame from the value heads. The checkpoint requires a robot or camera description; ours is the same generic one for every item. RoboRewardBench: progress 1 − rem_last / rem_first, mapped to 1–5 with cut points as above. Pairs: the two observations as a 2-frame episode in both orders, P(rising) = sigmoid(((rem₀ − rem₁) − (rem′₀ − rem′₁)) / 2 s). Its training mixture includes Open X-Embodiment, AgiBot and RoboMIND.
• DenseReward-1frame (densereward/densereward-1frame): the model card's quickstart per frame (its system prompt, image plus "Task: …", greedy decoding of a reward in [0, 1]). RoboRewardBench: the reward of the last frame, cut points as above. Pairs: P(rising) = sigmoid((r_later − r_earlier) / 0.1).
• TOPReward (arXiv 2602.19313) on Qwen/Qwen3-VL-8B-Instruct, zero-shot: LeRobot's defaults (no chat template, log-probability of the final "True" after the statement that the video completes the task, at most 16 frames). RoboRewardBench: the mean over the last three prefix anchors (the paper's success readout), cut points as above. Pairs: the two observations as a 2-frame, 1-fps episode in both orders, P(rising) = sigmoid((lp_fwd − lp_rev) / 0.5).

Claude 5.5. We asked Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 to score all 2,831 RoboRewardBench videos through the Claude Code CLI (claude -p, default effort, so extended thinking is on; no tools), with our system prompt carrying the suite's instructions and 1–5 rubric, every frame of the clip as an image at native resolution (the same visual input as this model), one sample per video and a final Score: N line. Claude returns no probabilities, so its numbers use its single answer. All three score too low on average (mean prediction 2.28–2.45 vs a mean label of 2.81). Claude Code reported an API-equivalent cost of $2.19 (Haiku), $35.92 (Sonnet) and $75.52 (Opus) for the 2,831 videos.

Other baselines.

• Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice): the untrained base VLM with the same judge prompt; options are lettered and the answer is restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM structured-output choice decoding (structured_outputs.choice + letter logprobs); the robot suites were run with the transformers implementation.
• OneJev / Intern-Decision / OmniJev / Jev-Omni: released checkpoints run with their own inference code inside the same harness and items.
• JevAny-Qwen3.8-27B-LoRA and JevAny-Qwen3.5-4B-LoRA: released JevAny checkpoints, same media settings as this model.
• Qwen3-VL-4B-Instruct zero-shot: RoboReward's prompt and digit readout, as a check of our harness against the published leaderboard.

Our harness measures the RoboReward models with lower error than published on the official leaderboard (8B: 0.358 vs 0.665; 4B: 0.407 vs 0.845) on the same 2,831 test videos (HF test split, revision 469b9af, all labels identical). Free generation reproduces our number and a zero-shot Qwen3-VL-4B-Instruct control lands near its published value (0.945 vs 1.032), so we report what we measured and do not compare with leaderboard entries.

Calibration and uncertainty

Model	Temperature T	NLL ↓ raw → T	ECE ↓ raw → T	Brier ↓ raw → T	RoboArena NLL ↓ raw → T
JevAny-Robotics-4B-LoRA v0.4	1.29	0.779 → 0.742	0.069 → 0.030	0.399 → 0.390	0.981 → 0.962
Flora-Rewarder-27B	1.60	0.858 → 0.730	0.115 → 0.032	0.401 → 0.376	1.519 → 1.222
RoboReward-8B	2.63	1.388 → 0.910	0.227 → 0.055	0.563 → 0.467	2.082 → 1.369
RoboReward-4B	2.09	1.212 → 0.942	0.175 → 0.043	0.532 → 0.476	1.649 → 1.330

Each temperature was fitted by NLL on 2,000 held-out RoboRewardBench-style items (the RoboReward validation split). The raw probabilities of this model have the lowest NLL of the 13 models we measured; after their own temperature Flora-Rewarder-27B is slightly better (0.730 vs 0.742). Against Flora the raw ECE is significantly lower (−0.047 [−0.076, −0.013]) and NLL and Brier are not significantly different. The temperature does not change the top score; use it only for reported probabilities.

Split-conformal intervals (Sheng et al., EMNLP 2025), α = 0.1, calibrated on the same 2,000 held-out items and tested on all of RoboRewardBench (coverage P5 / P95 from 1,000 cluster bootstraps). Every method, R2CCP included, uses the paper's boundary adjustment for whole scores: interval ends are rounded inwards (l′ = ⌈l⌉, u′ = ⌊u⌋), and an interval left with no whole score becomes the rounded expected score; for R2CCP we first take the hull of its output set.

Method	Coverage [P5, P95]	Mean width (scores)	Width 1 / 2 / 3 / 4 / 5	Single-score intervals correct
CQR (recommended)	95.1% [93.0, 96.7]	2.15	36 / 29 / 22 / 10 / 3%	94.6%
R2CCP	87.3% [84.2, 90.0]	1.66	47 / 41 / 11 / 1 / 0%	90.2%
absolute residual	89.2% [86.8, 91.2]	2.17	0 / 83 / 17 / 0 / 0%	–

With CQR the interval is the range from the 5% to the 95% quantile of the raw distribution (calibration offset 0). A single-score interval means the top score holds more than 90% of the probability: accept those scores automatically and send wide intervals to review. Calibrated on random halves of the RoboRewardBench task groups, only Flora-Rewarder-27B gets narrower intervals (CQR 2.02 vs 2.15 scores, R2CCP 1.92 vs 1.99); every other model is wider (RoboReward-8B 2.53–3.15, Claude Opus 5.5 2.61–3.35). Full tables: eval/final_eval.md and eval/new_models.md.

Choosing a point score

Use	Point score	RoboRewardBench (2,831)
Report a score, exact-score accuracy (our tables)	top score (most likely)	Overall 0.309, exact 70.8%, MSE 0.567
Lowest absolute error per video	median of the distribution	MAE 0.367 (top score 0.373; not significant), MSE 0.531
Ranking, reward signal, squared error	expected score (probability-weighted mean of 1–5)	MSE 0.458, Spearman 0.882, policy ranking 91.1%
Avoid	midpoint of an interval	Overall 0.386 (R2CCP) – 0.540 (absolute residual)

Speed

Model	Parameters	GPUs	RoboRewardBench video (RoboArena / OXE)	RoboPulse pair	RoboPulse++ pair
This model, JevAny acceleration flags	4B	1	165 ms (281 / 91)	117 ms	83 ms
This model, JevAny reference path	4B	1	372 ms (661 / 298)	291 ms	202 ms
Flora-Rewarder-27B (vLLM, tensor parallel)	27B	2	971 ms (1,053 / 903)	769 ms	754 ms
RoboReward-4B, one forward pass	4B	1	142 ms (204 / 108)	280 ms	292 ms
RoboReward-8B, one forward pass	8B	1	160 ms (224 / 130)	292 ms	345 ms
Robometer-4B	4B	1	319 ms (620 / 168)	134 ms	191 ms
RynnValue-4B	4B	1	342 ms (684 / 295)	132 ms	206 ms
TOPReward (Qwen3-VL-8B-Instruct)	8B	1	657 ms (726 / 340)	213 ms	262 ms
DenseReward-1frame	4B	1	869 ms (901 / 841)	516 ms	627 ms
Robo-Dopamine-GRM-2.0-4B (vLLM)	4B	1	–	210 ms	168 ms
Robo-Dopamine-GRM-2.0-8B (vLLM)	8B	1	–	307 ms	254 ms
Claude Haiku / Sonnet / Opus 5.5 (Claude Code CLI)	–	–	3.5 / 2.5 / 3.7 s	–	–

Batch 1, median end-to-end time per item: decoding, preprocessing and every model call the item needs (both orders of a pair where the rule uses them), on one A100-40GB unless noted. Flora-Rewarder-27B's BF16 weights do not fit one 40 GB GPU. DenseReward-1frame's RoboRewardBench time covers three scored frames; its readout uses one. Sample sizes: eval/new_models.md.

General judgement

Benchmark	JevAny-Qwen3.5-4B-LoRA	This model (v0.4)	v0.4 − JevAny-4B
Transfer v9 (1,046) acc / NLL	78.68% / 0.587	68.64% / 0.821	acc −10.0 [−12.8, −7.3]
JevBench public (local rebuild, 231) acc	77.92%	75.76%	−2.2 [−6.9, +2.6]
JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc	0.201 / 52.6%	0.147 / 48.7%	acc −3.9 [−5.4, −2.4]

v0.4 is trained from the base model, not from the JevAny parent, so it does not keep the parent's general judgement: use it for robot judging and SimpleJev/JevAny-Qwen3.5-4B-LoRA for anything else. JevBench is a local rebuild of the public suite (not comparable with the official 80.09%); these numbers use JevAny's reference path.

Training

• Init: Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; new LoRA adapter (r = 8 on all projection modules) and pointer head (head dim 256 + residual 256), BF16.
• Data: JevAny-Robotics-Train-v0.4, 289,311 questions including 11% general-judgement replay (about 400M tokens); 1 epoch, 9,041 steps, global batch 32, lr 2e-4 (OneCycle, 10% warm-up).
• Compute: 43 hours on one or two A100-40GB GPUs (17 h on one, 26 h on two; 69 GPU-hours, including in-training evaluations), peak memory 20 GiB per GPU.
• Media: images at native resolution, every frame of the derived clips (about 1–2 fps, 8–32 frames), up to 16,384 packed tokens.
• training_config.json, training_metrics.json and eval/training_eval_history.jsonl hold the arguments and in-training evaluations.

Limitations

Built for robot judging: its general (non-robot) judgement is weaker than the JevAny parent's. Its weakest case is a long multi-step clip cut before the end, which it tends to score as nearly done. The test sets share source datasets with the training data (overlapping episodes and frames were removed), so check thresholds and conformal intervals on your own data.

Usage

# --media full and the acceleration flags are on the full-media-accel branch of the fork (pending JevAny PRs):
git clone -b full-media-accel https://github.com/BruceSheng1202/JevAny && cd JevAny
pip install -e '.[serve,multimodal,fast]'          # fast = flash-linear-attention
pip install --no-build-isolation causal-conv1d    # optional, see JevAny's docs/DEPLOYMENT.md
JEVANY_MERGE_BF16=1 jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16 \
    --media full --fused-kernels --media-cuda-graphs --max-state-tokens 16384 --max-packed-tokens 32768
# the previous release: --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA@v0.2  (or revision="v0.2" when downloading)

• --media full (or JEVANY_MEDIA=full): images at native resolution and every frame of a video, as in training and evaluation. JevAny's default (capped: 512×512 pixels per request, 8 frames per video) costs accuracy, most of all on video (RoboRewardBench Overall 0.456).
• Acceleration flags (JEVANY_MERGE_BF16=1, --fused-kernels, --media-cuda-graphs) give the speeds above. Without them JevAny uses its reference path: the same results up to BF16 rounding (99.1% identical top scores), 2.3–2.5× slower.
• Send short clips like the evaluation's (about 1 frame per second, 8–32 frames).

Reading the five probabilities of a 1–5 question (options in score order):

import itertools, math

def summarise(p, T=1.29):
    """p: probabilities of scores 1..5 (raw, as returned)."""
    cdf = list(itertools.accumulate(p))
    quantile = lambda q: 1 + next(i for i, c in enumerate(cdf) if c >= q - 1e-12)
    lo, hi = quantile(0.05), quantile(0.95)          # 90% conformal interval (CQR, calibration offset 0)
    w = [x ** (1 / T) for x in p]                       # temperature only for reporting probabilities
    return {
        "top": 1 + max(range(5), key=p.__getitem__),    # reported score
        "median": quantile(0.5),                        # lowest absolute error
        "expected": sum((i + 1) * x for i, x in enumerate(p)),  # ranking / reward signal
        "interval": (lo, hi),
        "confident": lo == hi,                          # single-score interval: 94.6% correct on RoboRewardBench
        "calibrated": [x / sum(w) for x in w],
    }

The repository contains the LoRA adapter and JevAny readout metadata, not the base weights; the base model's license (Apache-2.0) and the terms of the training data sources also apply.