README: usage instructions, features, and real test-environment details

README.md

@@ -0,0 +1,147 @@
+# openPangu-2.0-Flash support for llama.cpp v0.7.0
+
+This repository contains a patch that adds support for **openPangu-2.0-Flash**
+(92B MoE, ~6B active parameters, 512K context) to the latest llama.cpp release
+**v0.7.0** (commit `620d2225cc436ab0b001e3200e590e3b286ef9ab`, 2026-10-15).
+
+The support was ported from
+[mrexodia/llama.cpp-openPangu-2.0-Flash](https://github.com/mrexodia/llama.cpp-openPangu-2.0-Flash)
+(which tracks an older llama.cpp base) and rebased on top of v0.7.0.
+
+Files:
+
+- `llama.cpp-openpangu-v0.7.0.diff` — the full patch (40 files, +2845 / −49)
+- `patch.sh` — downloads the pinned llama.cpp commit, applies the patch and
+  prints build/run instructions
+
+## 中文說明
+
+本 repo 是將 openPangu-2.0-Flash（92B MoE、約 6B 激活參數、512K 上下文）
+的 llama.cpp 支援，從 mrexodia 的 fork 移植到最新的 llama.cpp **v0.7.0** 上。
+
+用法：
+
+1. 執行 `./patch.sh`（會自動下載 v0.7.0 原始碼並套用 patch）
+2. `cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab`
+3. `cmake -B build -S . -DCMAKE_BUILD_TYPE=Release && cmake --build build -j`
+4. 下載 GGUF 模型（見下方連結）後即可執行 `llama-cli` / `llama-server`
+
+## What is included
+
+- The `openpangu-v2` architecture: absorbed MLA attention, DSA sparse attention
+  (lightning-indexer top-2048 selection on the global layers), per-layer
+  sliding-window attention via a hybrid iSWA cache, 4-stream manifold
+  hyper-connections (mHC), MoME causal convolutions with recurrent cross-batch
+  state, and 128 learned cache-resident attention sinks per layer
+- Fused CPU/CUDA kernels for the mHC mixing chain (`GGML_OP_SINKHORN`,
+  `GGML_OP_HC_MIX`), with automatic fallback to the unfused ops on backends
+  without the kernels
+- Gather-based sparse attention at decode: DSA layers attend only the top-2048
+  selected tokens (plus sinks), with a fused indexer-scoring kernel
+  (`GGML_OP_DSA_SCORE`)
+- Fused DSA indexer scoring at prefill (tiled kernel that skips the multi-GB
+  scoring intermediate) plus a radix-select top-k for `GGML_OP_TOP_K`
+- Multi-token-prediction self-speculative decoding (`--spec-type draft-mtp`,
+  with the split MTP draft GGUF)
+- Chat support: tool calling and ` thinking` reasoning parsing for the openPangu
+  template
+- HF → GGUF conversion (`conversion/pangu.py`) and a base/MTP split tool
+  (`conversion/split_pangu_mtp.py`)
+
+Ported and adapted to v0.7.0 APIs, plus fixes on top of the original fork:
+
+- adapted to the v0.7.0 common chat parser API (`build_chat_peg_parser`),
+  per-layer `n_ff_exp()`, and the new `build_attn_mha(..., n_kv_max, ...)`
+  signature
+- CPU AMX fix: the AMX weight packer only implements 2D layouts; batched
+  (3D/4D) quantized weights — e.g. the MLA `wk_b`/`wv_b` tensors — were
+  corrupted when routed through the AMX extra buffer, producing gibberish
+  output. The AMX backend now rejects batched tensors and they run on the
+  plain CPU path
+- the absorbed-MLA `wk_b`/`wv_b` inputs are kept contiguous so they are also
+  correct with weight repacking on other backends
+
+## Requirements
+
+- A C/C++ build environment (CMake ≥ 3.14, C++17 compiler)
+- No GPU required (CPU-only works); CUDA/ Metal/ Vulkan also supported
+- Get the GGUF models at
+  [mrexodia/openPangu-2.0-Flash-GGUF](https://huggingface.co/mrexodia/openPangu-2.0-Flash-GGUF):
+  - `openPangu-2.0-Flash-base-Q3_K_M.gguf` — 40.7 GiB
+  - `openPangu-2.0-Flash-base-Q4_K_M.gguf` — 51.5 GiB
+  - `openPangu-2.0-Flash-base-Q8_0.gguf` — 90.8 GiB
+  - `openPangu-2.0-Flash-base-BF16.gguf` — 170.7 GiB
+  - `openPangu-2.0-Flash-mtp-Q8_0.gguf` — 9.2 GiB (optional MTP head)
+
+## Quick start
+
+```sh
+./patch.sh
+cd llama.cpp-620d2225cc436ab0b001e3200e590e3b286ef9ab
+cmake -B build -S . -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j --target llama-cli llama-server
+```
+
+CUDA build:
+
+```sh
+cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
+```
+
+## Run
+
+```sh
+./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf -c 65536 --jinja -t $(nproc)
+
+./build/bin/llama-server -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
+    -c 65536 --jinja --host 0.0.0.0 --port 8080
+```
+
+Context scales up to `-c 524288`; the compressed MLA KV cache needs only ~12 GB
+at the full 512K.
+
+### Multi-token prediction (self-speculative decoding)
+
+```sh
+./build/bin/llama-cli -m openPangu-2.0-Flash-base-Q3_K_M.gguf \
+    --spec-draft-model openPangu-2.0-Flash-mtp-Q8_0.gguf \
+    --spec-type draft-mtp -c 65536 --jinja
+```
+
+Notes:
+
+- The MTP draft adds ~10 GB of RAM usage on top of the base model
+- On CPUs with AMX, weight repacking (`--repack`, enabled by default) costs
+  ~16 GiB of extra RAM; if the combined footprint does not fit, run with `-nr`
+  (no repack)
+- On CPU-only setups the draft overhead can make generation slower than plain
+  decoding — measure on your hardware (`llama-bench` does not support draft
+  models; use `llama-cli` and compare the reported t/s)
+
+## Test environment（測試環境）
+
+All performance numbers below were measured on a real machine:
+
+- **CPU**: Intel Xeon Platinum 8559C (Emerald Rapids, supports AMX); the test
+  container is capped at **16 vCPUs** (host has 2 sockets × 48 cores × 2 SMT)
+- **RAM**: 2.0 TiB host; the test environment enforces a **64 GiB per-process
+  memory cap**, which is why all testing used the 40.7 GiB `Q3_K_M` quant
+- **GPU**: none — CPU-only inference, 16 threads (`-t 16`)
+- **OS/toolchain**: Linux, gcc 14.2.0, CMake 3.31.6
+- **Model**: `openPangu-2.0-Flash-base-Q3_K_M.gguf` (40.72 GiB, 91.6 B params)
+
+Measured resident memory of `llama-cli` with this model (`-c 512`, `-n 1`):
+
+| configuration | peak RSS |
+|---|---|
+| default (AMX weight repacking enabled) | ~57.1 GiB |
+| `-nr` (no repack) | ~41.2 GiB |
+| `-nr` + MTP draft (Q8_0) | ~52 GiB |
+| default + MTP draft (Q8_0) | exceeds 64 GiB — could not run under the test cap |
+
+## Tests
+
+- `test-backend-ops -b CPU -o SINKHORN -o HC_MIX -o DSA_SCORE -o TOP_K`:
+  534/534 tests pass
+- `test-chat`, `test-chat-template`, `test-chat-peg-parser`,
+  `test-chat-auto-parser`: all pass