Spaces:
Running
Running
Commit History
-
Move Nemotron text generation to WebGPU
b9bfeb6
-
Cache status + delete for WebLLM and Transformers.js (like wllama)
The model bar already shows "✓ downloaded" + a delete button for any engine that exposes cachedSet()/deleteCached(); wire those up for the other two: - WebLLM: MLC's own hasModelInCache() / deleteModelAllInfoInCache() over prebuiltAppConfig (matches the q4f16/q4f32 variant we'd load). - Transformers.js: inspect the 'transformers-cache' Cache API store; only count a model as cached when its *.onnx weights are present (Transformers.js sometimes caches the config/tokenizer JSONs but not the big model file — known upstream issue), so the badge is honest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>b0f48f8 -
Transformers.js: pre-flight WebGPU device, else WASM (fixes load + caching)
Verified headlessly: Transformers.js DOES cache (writes a transformers-cache store) — the real problem was it never loaded. navigator.gpu can exist and requestAdapter() succeed, yet the WebGPU backend still throws "no available backend", and a failed WebGPU attempt poisons the in-context WASM retry. Now we pick the device up front via requestDevice() and only use WebGPU when a real device is obtained; otherwise WASM from the start (which loads and caches reliably). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>bed3298 -
Fix stale JS (no-cache on /web) + Transformers.js WebGPU→WASM fallback
- "stops at 220": the browser was serving a stale cached personaPanel.js (old token cap) because /web assets had no Cache-Control. Set Cache-Control: no-cache so modules always revalidate (cheap 304 when unchanged). Model weights come from huggingface.co, so this doesn't touch their caching. - Transformers.js picked device:webgpu whenever navigator.gpu existed, but if the adapter request fails (headless / flaky drivers) it threw "no available backend" instead of falling back — so the model never loaded and never cached. Catch and retry on WASM; backendLabel now reflects the device actually used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>559041f -
Fix Qwen3 parsing (/no_think) + WebLLM double-load guard & progress clarity
Qwen3 is a thinking model: with a 220-token budget it spent everything on a <think> block and never emitted the JSON/answer, so persona parsing failed after the thinking. Append Qwen3's `/no_think` soft switch for these structured tasks (persona + diary) so it goes straight to the answer. Helper noThink(modelId) keyed on the model id; no-op for non-Qwen3. WebLLM "downloads twice / no cache": - ensure() guarded on _loadedId (only set AFTER load), so a re-entrant ensure during a slow download could start a SECOND load. Guard on _loadingId (set up front) instead — same fix applied to the Transformers.js engine. - MLC runs two phases through one progress callback (Fetching from network, then Loading from cache into GPU) — both 0→100%, which reads as "downloading twice". Pass MLC's progress text through so the status line shows which phase it is; the second pass is a cache-load, not a re-download. (Caching = MLC Cache API + our storage.persist(); evictable under quota pressure.…ab87288 -
Multi-engine benchmark: wllama vs Transformers.js vs WebLLM (engine selector)
Add a runtime facade (runtime.js) over three in-browser LLM engines so the same model can be A/B'd and tok/s compared: - engineWllama.js — llama.cpp WASM + WebGPU (GGUF from HF; 🦙 + local-first) - engineTransformers.js — HF Transformers.js (ONNX, WebGPU/WASM) - engineWebllm.js — MLC WebLLM (WebGPU only; greyed out without GPU) - genStats.js — shared tok/s tracker (decode rate, TTFT) for fair benchmarking modelBar.js gains an Engine selector (unavailable engines disabled); persona + diary panels now import from runtime.js. Removes the old wllamaLlm.js (folded into engineWllama.js). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>f8d0843