openPangu-2.0-Flash support for llama.cpp v0.7.0
This repository contains a patch that adds support for openPangu-2.0-Flash
(92B MoE, ~6B active parameters, 512K context) to the latest llama.cpp release
v0.7.0 (commit 620d2225cc436ab0b001e3200e590e3b286ef9ab, 2026-10-15).
The support was ported from mrexodia/llama.cpp-openPangu-2.0-Flash (which tracks an older llama.cpp base) and rebased on top of v0.7.0.
Files:
llama.cpp-openpangu-v0.7.0.diff— the full patch (48 files, +3340 / −368)patch.sh— downloads the pinned llama.cpp commit, applies the patch and prints build/run instructionsPROMPT_REPRODUCE.md— task brief for reproducing this project with a coding agent (port + A/B experiment protocol + validation + upload)LICENSE— GPL-3.0 (this project) with the references/licenses of the upstream projects it is built on
中文說明
本 repo 是將 openPangu-2.0-Flash（92B MoE、約 6B 激活參數、512K 上下文） 的 llama.cpp 支援，從 mrexodia 的 fork 移植到最新的 llama.cpp v0.7.0 上。
用法：
- 執行
./patch.sh（會自動下載 v0.7.0 原始碼並套用 patch） cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9abcmake -B build -S . -DCMAKE_BUILD_TYPE=Release && cmake --build build -j- 下載 GGUF 模型（見下方連結）後即可執行
llama-cli/llama-server
What is included
- The
openpangu-v2architecture: absorbed MLA attention, DSA sparse attention (lightning-indexer top-2048 selection on the global layers), per-layer sliding-window attention via a hybrid iSWA cache, 4-stream manifold hyper-connections (mHC), MoME causal convolutions with recurrent cross-batch state, and 128 learned cache-resident attention sinks per layer - Fused CPU/CUDA kernels for the mHC mixing chain (
GGML_OP_SINKHORN,GGML_OP_HC_MIX), with automatic fallback to the unfused ops on backends without the kernels - Gather-based sparse attention at decode: DSA layers attend only the top-2048
selected tokens (plus sinks), with a fused indexer-scoring kernel
(
GGML_OP_DSA_SCORE) - Fused DSA indexer scoring at prefill (tiled kernel that skips the multi-GB
scoring intermediate) plus a radix-select top-k for
GGML_OP_TOP_K - Multi-token-prediction self-speculative decoding (
--spec-type draft-mtp, with the split MTP draft GGUF) - Chat support: tool calling and
thinkingreasoning parsing for the openPangu template - HF → GGUF conversion (
conversion/pangu.py) and a base/MTP split tool (conversion/split_pangu_mtp.py)
Ported and adapted to v0.7.0 APIs, plus fixes on top of the original fork:
- adapted to the v0.7.0 common chat parser API (
build_chat_peg_parser), per-layern_ff_exp(), and the newbuild_attn_mha(..., n_kv_max, ...)signature - CPU AMX fix: the AMX weight packer only implements 2D layouts; batched
(3D/4D) quantized weights — e.g. the MLA
wk_b/wv_btensors — were corrupted when routed through the AMX extra buffer, producing gibberish output. The AMX backend now rejects batched tensors and they run on the plain CPU path - the absorbed-MLA
wk_b/wv_binputs are kept contiguous so they are also correct with weight repacking on other backends
Requirements
- A C/C++ build environment (CMake ≥ 3.14, C++17 compiler)
- No GPU required (CPU-only works); CUDA/ Metal/ Vulkan also supported
- Get the GGUF models at
mrexodia/openPangu-2.0-Flash-GGUF:
openPangu-2.0-Flash-base-Q3_K_M.gguf— 40.7 GiBopenPangu-2.0-Flash-base-Q4_K_M.gguf— 51.5 GiBopenPangu-2.0-Flash-base-Q8_0.gguf— 90.8 GiBopenPangu-2.0-Flash-base-BF16.gguf— 170.7 GiBopenPangu-2.0-Flash-mtp-Q8_0.gguf— 9.2 GiB (optional MTP head)
Quick start
./patch.sh
cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server
CUDA build:
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
Run
./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf -c 65536 --jinja -t $(nproc)
./build/bin/llama-server -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
-c 65536 --jinja --host 0.0.0.0 --port 8080
Context scales up to -c 524288; the compressed MLA KV cache needs only ~12 GB
at the full 512K.
Multi-token prediction (self-speculative decoding)
./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
--spec-draft-model openPangu-2.0-Flash-mtp-Q8_0.gguf \
--spec-type draft-mtp -c 65536 --jinja
Notes:
- The MTP draft adds ~10 GB of RAM usage on top of the base model
- On CPUs with AMX, weight repacking (
--repack, enabled by default) costs ~16 GiB of extra RAM; if the combined footprint does not fit, run with-nr(no repack) - On CPU-only setups the draft overhead can make generation slower than plain
decoding — measure on your hardware (
llama-benchdoes not support draft models; usellama-cliand compare the reported t/s)
Test environment（測試環境）
All performance numbers below were measured on a real machine:
- CPU: Intel Xeon Platinum 8559C (Emerald Rapids, supports AMX); the test container is capped at 16 vCPUs (host has 2 sockets × 48 cores × 2 SMT)
- RAM: 2.0 TiB host; the test environment enforces a 64 GiB per-process
memory cap, which is why all testing used the 40.7 GiB
Q3_K_Mquant - GPU: none — CPU-only inference, 16 threads (
-t 16) - OS/toolchain: Linux, gcc 14.2.0, CMake 3.31.6
- Model:
openPangu-2.0-Flash-base-Q3_K_M.gguf(40.72 GiB, 91.6 B params)
Measured resident memory of llama-cli with this model (-c 512, -n 1):
| configuration | peak RSS |
|---|---|
| default (AMX weight repacking enabled) | ~57.1 GiB |
-nr (no repack) |
~41.2 GiB |
-nr + MTP draft (Q8_0) |
~52 GiB |
| default + MTP draft (Q8_0) | exceeds 64 GiB — could not run under the test cap |
Tests
- full
ctestsuite (61 tests): 61/61 pass — includingtest-generate-modelsandtest-llama-context, which now cover openpangu-v2 as a first-class arch (synthetic model fixtures, KV-cache cases) and round-trip its K-sink-prefix metadata through the model saver test-backend-ops -b CPU -o SINKHORN -o HC_MIX -o DSA_SCORE -o TOP_K: 534/534 tests passtest-chat,test-chat-template,test-chat-peg-parser,test-chat-auto-parser: all pass
Comparison with the original fork
Measured against the original
mrexodia/llama.cpp-openPangu-2.0-Flash
(HEAD 76c92350b, llama.cpp base 6f4f53f2b from 2026-06-29), same machine,
same model, same flags.
Compatibility on this machine (AMX CPU)
| original fork | this port (v0.7.0) | |
|---|---|---|
| default settings | aborts — GGML_ASSERT(*cur_backend_id != -1) during sched_reserve (AMX weight repacking) |
runs |
-nr (weight repacking disabled) |
runs | runs |
The original fork cannot be used with default settings on a CPU with AMX.
Its llama-bench has no option to disable repacking at all — the original
numbers below required a local source edit. This port fixes the underlying
AMX packing bug (the AMX packer silently corrupted batched/3D weights).
llama-bench, Q3_K_M, -t 16 -r 3 (2026-10-16)
Both builds with weight repacking disabled (-nr, the only common
runnable configuration on this machine):
| test | original fork | this port | change |
|---|---|---|---|
| pp128 | 41.51 ± 2.21 | 60.47 ± 4.63 | +46 % |
| pp512 | 42.62 ± 0.83 | 64.75 ± 2.26 | +52 % |
| tg32 | 11.65 ± 0.19 | 11.52 ± 0.35 | ≈ |
| tg128 | 11.47 ± 0.21 | 11.82 ± 0.72 | ≈ |
This port with the default AMX weight repacking (original fork cannot run this configuration):
| test | this port (default) |
|---|---|
| pp128 | 56.84 ± 1.32 |
| pp512 | 62.65 ± 0.54 |
| tg32 | 11.94 ± 0.71 |
| tg128 | 12.02 ± 0.30 |
Reading these numbers:
- Prefill is ~1.5× faster than the original fork, mostly from the v0.7.0 base (3.5 months of upstream CPU optimizations). Generation is on par.
- On this CPU, AMX weight repacking is performance-neutral and costs
~16 GiB of extra RAM, so
-nris usually the better choice. - The host is shared and measurements vary between runs (see the ±).
Real inference (llama-cli end-to-end)
Actual generation with a real prompt (3,271 tokens of English text),
llama-cli -c 8192 -n 128 -t 16 --no-warmup -st, two runs each
(prompt eval / generation speed as reported by llama-cli):
| configuration | prompt eval | generation |
|---|---|---|
original fork, -nr |
29.7 / 28.9 t/s | 6.1 / 6.1 t/s |
this port, -nr |
41.0 / 41.0 t/s | 6.0 / 6.0 t/s |
| this port, default (AMX repack) | 37.7 / 38.1 t/s | 5.6 / 5.5 t/s |
- real prompt processing is ~+40 % faster than the original fork (+29 % in the default AMX repack configuration)
- generation with a ~3K-token context is ~6 t/s in all configurations (the DSA indexer adds per-token work over the cached context, so this is lower than the tg128 llama-bench numbers on a fresh 128-token run)
- run-to-run variance on this host is small for these end-to-end runs
MTP (self-speculative decoding)
Same prompt, llama-cli -n 128 -nr:
| generation | |
|---|---|
| original fork, MTP draft | 4.9 t/s |
| this port, MTP draft | 4.7 t/s |
either fork, plain decoding (llama-bench tg128) |
11.5–12.0 t/s |
On CPU the MTP draft head is a net loss in both forks; it is included for completeness and may pay off on GPUs where the draft forward pass is cheap.
CPU optimization experiments（CPU 優化實驗記錄）
Upstream CPU-speedup candidates (open pull requests of llama.cpp) are
benchmarked one by one on the machine described above. Because this shared
host is noisy, each candidate is measured with an alternating A/B
protocol: two binaries built from the same source tree (baseline vs
baseline + candidate patch) are run interleaved in the same session
(llama-bench -t 16 -r 3 -p 512 -n 128, default settings, 3–8 rounds each);
the table reports the medians over all rounds. Every number is
incremental: each candidate is compared against the previous merged
state, not against the original port.
Methodology fix (2026-10-16): a first attempt measured
-nrvia theLLAMA_ARG_REPACK=0env var.llama-benchignores that variable (it has its own parser; only common-arg tools such asllama-cli/llama-perplexityhonor it), so those runs were silently default-mode. The-nrentry for #27590 was re-measured withllama-bench --repack 0, the flag the tool actually supports.
Only candidates that win beyond the run-to-run noise are merged into the patch (✅ 有幫助); candidates without a real gain are recorded here with their data and are not merged (❌ 沒幫助).
| candidate | content | verdict | pp512 t/s (median) | tg128 t/s (median) |
|---|---|---|---|---|
| upstream PR #27024 by @jinzihao | ggml-cpu/amx: fix the (dead) block_q8_K VNNI quantization and enable the VNNI path |
✅ 有幫助 — merged | 57.25 → 59.20 (+3.4 %) | 10.94 → 11.24 (+2.7 %) |
| upstream PR #22525 by @HyeongiJeon | ggml-cpu: optimize ggml_gemm_q4_K_8x8_q8_K interleaving/staging for AVX-512 (and AVX2) |
✅ 有幫助（prefill）— merged | 55.78 → 57.42 (+2.9 %) | 10.70 → 10.87 (+1.5 %, neutral) |
| upstream PR #19706 by @Manogna-Sree | Q6_K block interleaving for x86 SIMD (AVX-512/AVX2) | ❌ 沒幫助 — not merged | 58.16 → 58.34 (+0.3 %) | 11.06 → 11.07 (+0.1 %) |
| upstream PR #27590 by @SongXiaoXi | ggml-cpu: add AVX-512/VNNI paths for Q5_K/Q6_K dot products | ✅ 有幫助（-nr）— merged |
-nr: 65.91 → 66.35 (+0.7 %) |
-nr: 11.16 → 11.32 (+1.4 %) |
| upstream PR #18698 by @zzjianhui | ggml-cpu: repack weights in parallel (OpenMP, NUMA-local thread placement) | ❌ 沒幫助（推論退化）— not merged | 56.93 → 54.58 (−4.1 %) | 10.67 → 10.45 (−2.1 %) |
| draft PR #27032 by @lexasub | CPU optimizations (GET_ROWS+ADD fusion, get_rows fast paths, tokenizer, KV-cell rework) — only the ggml-kernel subset is portable onto this tree |
❌ 沒幫助（不適用於本模型）— not merged | 56.64 → 56.88 (+0.4 %) | 10.82 → 10.73 (−0.8 %) |
| draft PR #27396 by @lexasub | speculative draft-model prefill (zero-copy embedding, event-based sync) for EAGLE3/DFlash/GLM4-style draft heads | ❌ 沒幫助（不適用；未移植）— not merged | — | — |
Round-by-round for #27024 (candidate vs baseline, t/s):
| round | pp512 | tg128 |
|---|---|---|
| 1 | 57.66 vs 57.69 | 11.09 vs 10.57 |
| 2 | 59.20 vs 57.25 | 11.05 vs 10.94 |
| 3 | 58.66 vs 56.57 | 11.47 vs 11.27 |
| 4 | 62.13 vs 55.67 | 11.62 vs 10.83 |
| 5 | 61.76 vs 65.98 | 11.24 vs 12.01 |
The candidate won 4 of 5 pairwise rounds on both metrics; the single losing round (r5) is a transient quiet window on the host (both absolute values spike upward). Median deltas stay positive (+3.4 % / +2.7 %), so the change is merged. All values t/s; measured 2026-10-16.
Round-by-round for #22525 (candidate vs baseline, t/s), default settings.
Note: Q3_K_M does contain 134 Q4_K tensors (the MoE expert weights),
so the Q4_K GEMM path this PR optimizes is actually exercised:
| round | pp512 | tg128 |
|---|---|---|
| 1 | 56.69 vs 57.18 | 10.97 vs 10.97 |
| 2 | 57.30 vs 57.15 | 10.88 vs 10.78 |
| 3 | 57.30 vs 55.90 | 10.30 vs 10.94 |
| 4 | 56.26 vs 55.66 | 10.85 vs 9.51 |
| 5 | 57.91 vs 55.07 | 10.55 vs 10.62 |
| 6 | 57.55 vs 56.81 | 10.60 vs 10.19 |
| 7 | 57.84 vs 55.52 | 11.04 vs 10.43 |
| 8 | 59.24 vs 55.06 | 10.97 vs 11.05 |
The candidate won 6 of 8 pairwise rounds on prefill (1 loss, 1 tie within 0.5 %); decode is neutral (4 W / 3 L / 1 T). The PR only changes the AVX-512 interleaving/staging of the Q4_K GEMM (the prefill path), so the measured profile matches the expectation. Merged. Measured 2026-10-16.
Round-by-round for #19706 (candidate vs baseline, t/s), default settings:
| round | pp512 | tg128 |
|---|---|---|
| 1 | 58.42 vs 53.67 | 11.67 vs 11.45 |
| 2 | 59.46 vs 58.68 | 12.04 vs 10.84 |
| 3 | 54.57 vs 60.04 | 10.54 vs 11.82 |
| 4 | 62.32 vs 57.06 | 10.95 vs 11.16 |
| 5 | 57.66 vs 58.59 | 11.10 vs 10.86 |
| 6 | 57.26 vs 57.92 | 11.05 vs 11.17 |
| 7 | 59.71 vs 56.95 | 10.98 vs 10.96 |
| 8 | 58.25 vs 58.41 | 11.11 vs 10.83 |
Verdict ❌ 沒幫助 (no measurable gain). Medians differ by +0.3 %
(prefill) and +0.1 % (decode), with a 4 W / 3 L / 1 T split on both — pure
noise. The base Q3_K_M model contains only one single Q6_K tensor,
so a faster Q6_K kernel has almost no weight to act on. The PR is not
merged; the data is recorded here for the record. Measured 2026-10-16.
Round-by-round for #27590 (candidate vs baseline, t/s), -nr mode
(llama-bench --repack 0). The Q5_K/Q6_K dots this PR changes are used on
the plain CPU path selected by -nr; with default repacking those weights go
through the repack gemv/gemm kernels instead (default-mode incremental
measured ~neutral, within noise):
| round | pp512 | tg128 |
|---|---|---|
| 1 | 67.88 vs 67.80 | 11.17 vs 11.15 |
| 2 | 66.08 vs 66.46 | 11.46 vs 11.18 |
| 3 | 66.01 vs 65.34 | 11.32 vs 11.19 |
| 4 | 66.63 vs 65.36 | 11.33 vs 11.12 |
Verdict ✅ 有幫助 (-nr 模式, +0.7 % prefill / +1.4 % decode medians;
decode won 4 of 4 rounds). In this model the PR affects Q5_K (2 tensors,
the dense ffn_down of layers 0–1) and Q6_K (1 tensor — output.weight,
the 388 M-param head that is read on every token), so the consistent decode
win matches the expectation. One caveat: the upstream PR is closed but never
merged (only an automated AI-content flag, no review), so it received extra
local validation before merging:
- perplexity on a fixed corpus: 1.0354 → 1.0354 (identical)
test-backend-ops -b CPU -o MUL_MAT -p 'type=q5_K|type=q6_K'passes
Merged. Re-measured 2026-10-16 with the correct --repack 0 flag (see
methodology note above).
Round-by-round for #18698 (candidate vs baseline = current HEAD i.e. all merged patches, t/s), default settings. This PR repacks the quantized weights in parallel with OpenMP and pins each repack thread NUMA-local. (Ported by hand: the year-old PR conflicts with the current tree and depends on the never-merged PR #18239.)
| round | pp512 | tg128 |
|---|---|---|
| 1 | 54.24 vs 56.40 | 10.48 vs 10.35 |
| 2 | 55.50 vs 57.92 | 10.37 vs 10.67 |
| 3 | 54.58 vs 56.93 | 10.45 vs 10.87 |
Verdict ❌ 沒幫助 (clear prefill regression, not merged). Medians: prefill −4.1 % (0 W / 3 L) and decode −2.1 % (1 W / 2 L); an earlier 5-round block against the same effective baseline agreed on prefill (−2.7 %, 1 W / 4 L) with noisy decode. Side effect worth recording: the parallel repack cuts model load wall time (load + tiny bench, warm page cache) from 24 s (2/2 runs) to 10 s (2/2 runs) — but the inference phase itself gets slower, so per the merge rule (real tok/s gains) the candidate is not merged. Correctness: perplexity 1.0361 → 1.0361 (identical), i.e. the parallel repack produces identical weights; the regression is presumably a NUMA page-placement effect, not a data error. Measured 2026-10-16.
Round-by-round for draft #27032 (candidate vs baseline = current HEAD, t/s),
default settings. The draft bundles several optimizations (GET_ROWS+ADD
fusion, get_rows fast paths, tokenizer, KV-cell rework). The KV/batch and
tokenizer parts are either not portable (rooted in a pre-refactor
KV-cell design that conflicts with this tree) or not measurable with this
protocol; only the disjoint ggml CPU-kernel subset could be ported and
tested:
| round | pp512 | tg128 |
|---|---|---|
| 1 | 57.49 vs 56.64 | 10.73 vs 10.82 |
| 2 | 56.88 vs 58.68 | 10.85 vs 10.75 |
| 3 | 55.71 vs 56.50 | 10.35 vs 10.95 |
Verdict ❌ 沒幫助 (not applicable to this model, not merged). ≈±1 % on
both metrics, PPL identical (1.0361). The reason follows from the graph and
kernel dispatch: (1) the GET_ROWS+ADD fusion needs an ADD node immediately
after a GET_ROWS node — this model's graph has no such pattern; (2) the
optimized Q4_K dequant path is not used (token_embd is Q3_K); (3) the
remaining get_rows cache-hit path only helps repeated 1-row gathers, which
this workload barely has. Recorded for the record. Measured 2026-10-16.
Recorded verdict for draft #27396 (not ported, not measured): the draft
optimizes the speculative draft-model prefill path (zero-copy embedding,
event-based sync) for EAGLE3 / DFlash / GLM4-style draft heads, and it
touches core files (llama-batch, llama-context, llama-graph, ...) that
have diverged from its base — 7 of 17 files conflict and a faithful port
would require redesigning the speculative prefill plumbing. This CPU setup
does not use speculative decoding at all — the bundled MTP head measured
net-negative on CPU (see the inference section above) — so the draft has no
applicable surface here. Recorded 2026-10-16.
Combined effect of the merged set
Interleaved A/B of the shipped tree (original port + all merged candidates) against the original port, same session, 3 rounds each, medians:
| config | pp512 | tg128 |
|---|---|---|
| default, original port → shipped | 55.65 → 56.30 (+1.2 %, won 3/3) | 10.50 → 10.86 (+3.4 %, won 3/3) |
-nr, original port → shipped |
60.96 → 64.87 (+6.4 %, won 3/3) | 10.20 → 10.26 (+0.6 %, noisy) |
The per-mechanism incremental numbers in the sections above are the
authoritative records; this table is the same data rolled up against the
original-port baseline in a single session. Note the separate, much larger
-nr vs default finding below.
-nr vs default on this machine
Side observation from the re-anchoring session: on this AMX machine the
plain-CPU (-nr) path is clearly faster than the default repack path.
HEAD (all merged patches): pp512 56.93 → 66.35 (+16.6 %) and tg128
10.67 → 11.32 (+6.1 %) with --repack 0 (3–4 interleaved rounds each,
2026-10-16). The repack kernels used by default have no AMX path, while the
plain Q4_K×Q8_K kernels use AMX (boosted further by #27024). On AMX-class
machines it is worth testing -nr for ~+15 % prefill.
License
This project (the patch set, patch.sh and documentation) is licensed
under GPL-3.0 — see LICENSE. The patch is a derivative of
llama.cpp; the MIT-licensed portions remain under their original licenses,
with the references listed in the LICENSE header and repeated here:
- llama.cpp — MIT — © ggml-org
contributors (base commit
620d2225cc436ab0b001e3200e590e3b286ef9ab) - mrexodia/llama.cpp-openPangu-2.0-Flash — MIT — original openPangu-2.0-Flash port
- upstream PR #27024 by @jinzihao — MIT (merged)
- upstream PR #22525 by @HyeongiJeon — MIT (merged)
- upstream PR #27590 by @SongXiaoXi — MIT (merged; closed upstream unmerged, verified locally)
- ggml — MIT (bundled with llama.cpp)