v1.9.1: V-JEPA2 re-measured on public SDK 2.33 (docs only, no code change)
Browse files- CHANGELOG.md +18 -0
- README.md +55 -5
CHANGELOG.md
CHANGED
|
@@ -1,5 +1,23 @@
|
|
| 1 |
# Changelog
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
## v1.9 (2026-10)
|
| 4 |
|
| 5 |
* **New: padding masks on the kernel path for `NeuronXLMRobertaModel`.** v1.8 sent any batch with a padding
|
|
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
+
## v1.9.1 (2026-11-08)
|
| 4 |
+
|
| 5 |
+
Documentation only. No code change: the kernels and layers are byte-identical to v1.9.
|
| 6 |
+
|
| 7 |
+
* **V-JEPA2 re-measured on the public SDK 2.33 DLAMI** (`torch_neuronx_2_14_0` venv: torch 2.14.0,
|
| 8 |
+
torch-neuronx 2.14.3.0, NKI 0.7.0, neuronx-cc 2.28.7631; transformers 5.22.2, kernels 0.17.3) on trn2.3xlarge.
|
| 9 |
+
The kernel source needed no change for this toolchain.
|
| 10 |
+
* Per instance: 44.5 to **159.6 clips/s (3.59x)** at LNC=2, batch 1 per core. On Beta 6 this was 4.27x.
|
| 11 |
+
* Single core, LNC=2, batch 1: 82.6 to **23.8 ms/clip (3.48x)** wall-clock; 23.54 ms device time at 45.1% MFU.
|
| 12 |
+
On Beta 6 this was 3.93x.
|
| 13 |
+
* The speedup narrowed because the stock path got faster on 2.33. The kernels themselves are unchanged or slightly faster.
|
| 14 |
+
* Accuracy: rel_l2 vs fp32 is 3.9e-2 in all four configurations, identical to Beta 6.
|
| 15 |
+
* Faster than the original `aws-neuron/vjepa2-neuron-kernels` v1 layer on the same instance: 23.8 vs 26.7 ms/clip.
|
| 16 |
+
* Documented that the stock V-JEPA2 graph at LNC=1, batch 1 is 2.1e-1 from fp32 on SDK 2.33. The LNC=2 and
|
| 17 |
+
LNC=1 batch-2 graphs are 1.4e-2.
|
| 18 |
+
* Requirements: SDK 2.33 install notes (transformers 5.22.2 requires `kernels>=0.17.2,<0.18`), and use the
|
| 19 |
+
`forward` graft rather than `KernelConfig`. DINOv3 is still verified on Beta 6 only.
|
| 20 |
+
|
| 21 |
## v1.9 (2026-10)
|
| 22 |
|
| 23 |
* **New: padding masks on the kernel path for `NeuronXLMRobertaModel`.** v1.8 sent any batch with a padding
|
README.md
CHANGED
|
@@ -38,7 +38,9 @@ have been validated.
|
|
| 38 |
|
| 39 |
## Performance (trn2.3xlarge, bf16, `torch.compile(backend="neuron")`)
|
| 40 |
|
| 41 |
-
|
|
|
|
|
|
|
| 42 |
Real Hugging Face weights. "Stock" is the same model compiled with
|
| 43 |
`torch.compile(backend="neuron")` without these kernels: DINOv3 cast to bf16, V-JEPA2 as its fp32
|
| 44 |
checkpoint (stock V-JEPA2 is the encoder as shipped, and its accuracy is in the table below).
|
|
@@ -52,7 +54,7 @@ checkpoint (stock V-JEPA2 is the encoder as shipped, and its accuracy is in the
|
|
| 52 |
| DINOv3 ViT-L/16 | 448 | 201.1 | **491.9** | 2.45x |
|
| 53 |
| DINOv3 ViT-H+/16 | 224 | 400.0 | **626.6** | 1.57x |
|
| 54 |
| DINOv3 ViT-H+/16 | 448 | 106.8 | **200.3** | 1.88x |
|
| 55 |
-
| V-JEPA2 ViT-L, 16 frames x 256 px | - | 36.8 clips/s | **157.3 clips/s** | 4.27x |
|
| 56 |
|
| 57 |
Configuration: LNC=1 (8 logical cores) for every row except V-JEPA2 stock, whose best was LNC=2
|
| 58 |
(4 cores). DINOv3 kernels run at batch 8 (224 px) or batch 2 (448 px). DINOv3 stock runs at batch 4
|
|
@@ -67,12 +69,47 @@ Configuration: LNC=1 (8 logical cores) for every row except V-JEPA2 stock, whose
|
|
| 67 |
| DINOv3 ViT-L/16 | 448 | 25.1 img/s | **83.0 img/s** | 3.31x |
|
| 68 |
| DINOv3 ViT-H+/16 | 224 | 21.4 img/s | **88.2 img/s** | 4.12x |
|
| 69 |
| DINOv3 ViT-H+/16 | 448 | 17.7 img/s | **32.4 img/s** | 1.83x |
|
| 70 |
-
| V-JEPA2 ViT-L, 16 frames x 256 px | - | 101.5 ms/clip | **25.8 ms/clip** | 3.93x |
|
| 71 |
|
| 72 |
The gain grows with resolution: attention cost grows with token count, and these kernels keep
|
| 73 |
attention and its surroundings on chip. At 224 px with large batches the gap narrows. For example,
|
| 74 |
stock ViT-L/16 at LNC=2 batch 4 is 252 img/s and the kernels give 321 img/s (1.27x).
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
## Accuracy
|
| 77 |
|
| 78 |
Every result above passes a per-image check against the fp32 CPU model (rel_l2 <= 5e-2).
|
|
@@ -85,10 +122,16 @@ LayerNorm statistics stay in fp32:
|
|
| 85 |
| DINOv3 ViT-L/16 | 1.3-1.6e-2 | 0.8-1.0e-2 |
|
| 86 |
| DINOv3 ViT-H+/16 | 1.0-1.7e-2 | 0.6-1.0e-2 |
|
| 87 |
| V-JEPA2 ViT-L | 1.4e-2 (fp32 model, compiler-chosen precision) | 3.9e-2 (bf16 matmuls) |
|
|
|
|
| 88 |
|
| 89 |
For V-JEPA2 the stock path compiles the fp32 checkpoint and keeps more of it in higher precision,
|
| 90 |
so it is closer to fp32 than the kernels (which run every matmul in bf16).
|
| 91 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
## Quick start
|
| 93 |
|
| 94 |
```python
|
|
@@ -205,9 +248,16 @@ RoPE pairings via `rope=`:
|
|
| 205 |
|
| 206 |
* trn2 (verified on trn2.3xlarge, LNC=1 and LNC=2).
|
| 207 |
* PyTorch Native (TorchNeuron) with NKI 0.7 and `nkilib`. Verified on Beta 6 (torch 2.13.0,
|
| 208 |
-
torch-neuronx 2.13.3, NKI 0.7.0b1, neuronx-cc 2.0.404056) and
|
| 209 |
`torch_neuronx_2_14_0` venv (torch 2.14.0, torch-neuronx 2.14.3.0, NKI 0.7.0, neuronx-cc 2.28.7631).
|
| 210 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 211 |
|
| 212 |
## Repository layout
|
| 213 |
|
|
|
|
| 38 |
|
| 39 |
## Performance (trn2.3xlarge, bf16, `torch.compile(backend="neuron")`)
|
| 40 |
|
| 41 |
+
The two tables below were measured 2026-10 on PyTorch Native Beta 6 (torch 2.13.0, torch-neuronx
|
| 42 |
+
2.13.3, NKI 0.7.0b1). V-JEPA2 has been re-measured on the public SDK 2.33 release; see
|
| 43 |
+
[V-JEPA2 on SDK 2.33](#v-jepa2-on-sdk-233) below. The DINOv3 rows have not been re-measured on 2.33.
|
| 44 |
Real Hugging Face weights. "Stock" is the same model compiled with
|
| 45 |
`torch.compile(backend="neuron")` without these kernels: DINOv3 cast to bf16, V-JEPA2 as its fp32
|
| 46 |
checkpoint (stock V-JEPA2 is the encoder as shipped, and its accuracy is in the table below).
|
|
|
|
| 54 |
| DINOv3 ViT-L/16 | 448 | 201.1 | **491.9** | 2.45x |
|
| 55 |
| DINOv3 ViT-H+/16 | 224 | 400.0 | **626.6** | 1.57x |
|
| 56 |
| DINOv3 ViT-H+/16 | 448 | 106.8 | **200.3** | 1.88x |
|
| 57 |
+
| V-JEPA2 ViT-L, 16 frames x 256 px | - | 36.8 clips/s | **157.3 clips/s** | 4.27x (SDK 2.33: 3.59x, below) |
|
| 58 |
|
| 59 |
Configuration: LNC=1 (8 logical cores) for every row except V-JEPA2 stock, whose best was LNC=2
|
| 60 |
(4 cores). DINOv3 kernels run at batch 8 (224 px) or batch 2 (448 px). DINOv3 stock runs at batch 4
|
|
|
|
| 69 |
| DINOv3 ViT-L/16 | 448 | 25.1 img/s | **83.0 img/s** | 3.31x |
|
| 70 |
| DINOv3 ViT-H+/16 | 224 | 21.4 img/s | **88.2 img/s** | 4.12x |
|
| 71 |
| DINOv3 ViT-H+/16 | 448 | 17.7 img/s | **32.4 img/s** | 1.83x |
|
| 72 |
+
| V-JEPA2 ViT-L, 16 frames x 256 px | - | 101.5 ms/clip | **25.8 ms/clip** | 3.93x (SDK 2.33: 3.48x, below) |
|
| 73 |
|
| 74 |
The gain grows with resolution: attention cost grows with token count, and these kernels keep
|
| 75 |
attention and its surroundings on chip. At 224 px with large batches the gap narrows. For example,
|
| 76 |
stock ViT-L/16 at LNC=2 batch 4 is 252 img/s and the kernels give 321 img/s (1.27x).
|
| 77 |
|
| 78 |
+
### V-JEPA2 on SDK 2.33
|
| 79 |
+
|
| 80 |
+
Re-measured 2026-11-08 with v1.9 (`get_kernel(..., version=1)`). The environment was the public
|
| 81 |
+
SDK 2.33 DLAMI `torch_neuronx_2_14_0` venv on trn2.3xlarge: torch 2.14.0, torch-neuronx 2.14.3.0,
|
| 82 |
+
NKI 0.7.0, neuronx-cc 2.28.7631.0, transformers 5.22.2, kernels 0.17.3. The input was 16 frames
|
| 83 |
+
x 256 px, from the fp32 checkpoint. Stock is that model compiled with
|
| 84 |
+
`torch.compile(backend="neuron")`. The kernel source needed no change for this toolchain.
|
| 85 |
+
|
| 86 |
+
| | Stock | **These kernels** | Speedup |
|
| 87 |
+
|---|---|---|---|
|
| 88 |
+
| Per instance, best config | 44.5 clips/s | **159.6 clips/s** | **3.59x** |
|
| 89 |
+
| Single core, LNC=2, batch 1, wall-clock | 82.6 ms/clip | **23.8 ms/clip** | 3.48x |
|
| 90 |
+
| Single core, LNC=2, batch 1, device time (neuron-explorer) | 82.57 ms, 12.7% MFU | **23.54 ms, 45.1% MFU** | 3.51x |
|
| 91 |
+
| Single core, LNC=1, batch 1, wall-clock | 165.2 ms/clip | **51.8 ms/clip** | 3.19x |
|
| 92 |
+
|
| 93 |
+
**Per-instance runs** (one process per logical core, clips/s):
|
| 94 |
+
|
| 95 |
+
| Batch per core | Stock LNC=2 (4 cores) | Stock LNC=1 (8 cores) | Kernels LNC=2 (4 cores) | Kernels LNC=1 (8 cores) |
|
| 96 |
+
|---|---|---|---|---|
|
| 97 |
+
| 1 | 44.5 | 44.7 | **159.6** | 149.8 |
|
| 98 |
+
| 2 | 44.3 | 45.1 | 143.1 | 147.4 |
|
| 99 |
+
| 4 | 42.2 | 44.7 | - | - |
|
| 100 |
+
|
| 101 |
+
- **Best configuration changed.** On SDK 2.33 the kernels do best at LNC=2 with batch 1. On Beta 6
|
| 102 |
+
the best was LNC=1 with 8 cores.
|
| 103 |
+
- **Stock is flat.** It stays at 42-45 clips/s across every batch size and LNC setting.
|
| 104 |
+
- **Compared with Beta 6:**
|
| 105 |
+
- The stock path got faster: 36.8 to 44.5 clips/s per instance, and 101.5 to 82.6 ms single-core.
|
| 106 |
+
- The kernels stayed about the same: 157.3 to 159.6 clips/s, and 25.8 to 23.8 ms.
|
| 107 |
+
- The speedup therefore narrowed, from 4.27x to 3.59x per instance and from 3.93x to 3.48x
|
| 108 |
+
single-core.
|
| 109 |
+
- **This repo is faster than the original.** On the same SDK 2.33 instance, wall-clock and same
|
| 110 |
+
input, the original [`aws-neuron/vjepa2-neuron-kernels`](https://huggingface.co/kernels/aws-neuron/vjepa2-neuron-kernels)
|
| 111 |
+
v1 layer is 26.7 ms/clip. This repo's v1.9 is 23.8 ms/clip.
|
| 112 |
+
|
| 113 |
## Accuracy
|
| 114 |
|
| 115 |
Every result above passes a per-image check against the fp32 CPU model (rel_l2 <= 5e-2).
|
|
|
|
| 122 |
| DINOv3 ViT-L/16 | 1.3-1.6e-2 | 0.8-1.0e-2 |
|
| 123 |
| DINOv3 ViT-H+/16 | 1.0-1.7e-2 | 0.6-1.0e-2 |
|
| 124 |
| V-JEPA2 ViT-L | 1.4e-2 (fp32 model, compiler-chosen precision) | 3.9e-2 (bf16 matmuls) |
|
| 125 |
+
| V-JEPA2 ViT-L, SDK 2.33 | 1.4e-2 (LNC=2, and LNC=1 batch 2); **2.1e-1 at LNC=1 batch 1** | 3.9e-2 (all four configs, identical to Beta 6) |
|
| 126 |
|
| 127 |
For V-JEPA2 the stock path compiles the fp32 checkpoint and keeps more of it in higher precision,
|
| 128 |
so it is closer to fp32 than the kernels (which run every matmul in bf16).
|
| 129 |
|
| 130 |
+
On SDK 2.33, the stock V-JEPA2 graph compiled at LNC=1 with batch 1 is 2.1e-1 from fp32. That
|
| 131 |
+
result was the same across 4 runs and 2 processes. The LNC=2 graph and the LNC=1 batch-2 graph
|
| 132 |
+
are 1.4e-2. The kernels are 3.9e-2 in every configuration. If you compare against stock at LNC=1,
|
| 133 |
+
use batch 2 or LNC=2 as the reference.
|
| 134 |
+
|
| 135 |
## Quick start
|
| 136 |
|
| 137 |
```python
|
|
|
|
| 248 |
|
| 249 |
* trn2 (verified on trn2.3xlarge, LNC=1 and LNC=2).
|
| 250 |
* PyTorch Native (TorchNeuron) with NKI 0.7 and `nkilib`. Verified on Beta 6 (torch 2.13.0,
|
| 251 |
+
torch-neuronx 2.13.3, NKI 0.7.0b1, neuronx-cc 2.0.404056) and on the **SDK 2.33 DLAMI**
|
| 252 |
`torch_neuronx_2_14_0` venv (torch 2.14.0, torch-neuronx 2.14.3.0, NKI 0.7.0, neuronx-cc 2.28.7631).
|
| 253 |
+
SDK 2.33 coverage: XLM-RoBERTa since v1.8, and V-JEPA2 since v1.9.1. DINOv3 has been verified only
|
| 254 |
+
on Beta 6.
|
| 255 |
+
* Transformers with DINOv3 / V-JEPA2 (verified on 4.57.6 and 5.22.2); `kernels` for `get_kernel`.
|
| 256 |
+
The SDK 2.33 venv does not include either package. One way to add them without touching its torch:
|
| 257 |
+
`pip install --target ~/xpkgs --no-deps "transformers==5.22.2" "kernels>=0.17.2,<0.18" tokenizers safetensors regex typer-slim accelerate`,
|
| 258 |
+
then `export PYTHONPATH=~/xpkgs`. `transformers` 5.22.2 rejects `kernels` 0.18.x.
|
| 259 |
+
* Use the `forward` graft shown in the quick start. `transformers.KernelConfig` cannot map
|
| 260 |
+
`VJEPA2Encoder`, because it is not registered with `use_kernel_forward_from_hub`.
|
| 261 |
|
| 262 |
## Repository layout
|
| 263 |
|