Upload README.md with huggingface_hub

1d89903 verified 14 days ago

1.91 kB

---
license: mit
tags:
  - llama-cpp-python
  - llama.cpp
  - cuda
  - gguf
  - embeddings
---
# llama-cpp-python 0.3.36 (CUDA 13) prebuilt wheel

Prebuilt `llama-cpp-python` with CUDA, so you can install in seconds instead of compiling for 10-20 minutes. Made to run `unsloth/embeddinggemma-2-GGUF` (architecture `gemma-embedding2`), which the PyPI build cannot load.

## Install

!pip install https://huggingface.co/vinamrag-code/llama-cpp-python-cuda-wheels/resolve/main/llama_cpp_python-0.3.36%2Bcu130.gemma2-py3-none-linux_x86_64.whl


Note: the `+` in the file name is written as `%2B` in the URL. `llama_cpp.__version__` prints `0.3.36`; `pip show llama-cpp-python` shows `0.3.36+cu130.gemma2`.

## Works on
- Linux x86_64
- Python 3.13 (tested; the wheel is tagged `py3`, other 3.x versions are untested)
- CUDA 13 runtime (built with CUDA 13.0)
- GPUs with compute capability 7.5 (Tesla T4, as in Google Colab)
## Does not work on
- Windows, macOS, ARM
- Other GPU generations (built only with `-DCMAKE_CUDA_ARCHITECTURES=75`)
## What is inside
- llama-cpp-python 0.3.36
- llama.cpp commit `fc9ce6b9d52a8504edcb262abc92737c2289f96c` (2026-10-08)
- Built with `CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=75"`
- Two small fixes to the Python bindings, needed because that llama.cpp commit changed its C structs:
  1. Added the missing `moe_cache_size` field to `llama_context_params` (without it, `embed()` crashes with a NULL pointer error).
  2. `embed()` now uses `llama_model_n_embd_out` for the vector size (otherwise embeddinggemma-2 returns 512 numbers instead of 768).
## Tested
`embeddinggemma-2-BF16.gguf` with `embedding=True, n_gpu_layers=-1` on a Colab T4: 768-number embeddings, all layers offloaded to the GPU.
## Credits
Built from [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) and [llama.cpp](https://github.com/ggml-org/llama.cpp), both MIT licensed.

	---
	license: mit
	tags:
	- llama-cpp-python
	- llama.cpp
	- cuda
	- gguf
	- embeddings
	---

	# llama-cpp-python 0.3.36 (CUDA 13) prebuilt wheel

	Prebuilt `llama-cpp-python` with CUDA, so you can install in seconds instead of compiling for 10-20 minutes. Made to run `unsloth/embeddinggemma-2-GGUF` (architecture `gemma-embedding2`), which the PyPI build cannot load.

	## Install

	!pip install https://huggingface.co/vinamrag-code/llama-cpp-python-cuda-wheels/resolve/main/llama_cpp_python-0.3.36%2Bcu130.gemma2-py3-none-linux_x86_64.whl


	Note: the `+` in the file name is written as `%2B` in the URL. `llama_cpp.__version__` prints `0.3.36`; `pip show llama-cpp-python` shows `0.3.36+cu130.gemma2`.

	## Works on
	- Linux x86_64
	- Python 3.13 (tested; the wheel is tagged `py3`, other 3.x versions are untested)
	- CUDA 13 runtime (built with CUDA 13.0)
	- GPUs with compute capability 7.5 (Tesla T4, as in Google Colab)

	## Does not work on
	- Windows, macOS, ARM
	- Other GPU generations (built only with `-DCMAKE_CUDA_ARCHITECTURES=75`)

	## What is inside
	- llama-cpp-python 0.3.36
	- llama.cpp commit `fc9ce6b9d52a8504edcb262abc92737c2289f96c` (2026-10-08)
	- Built with `CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=75"`
	- Two small fixes to the Python bindings, needed because that llama.cpp commit changed its C structs:
	1. Added the missing `moe_cache_size` field to `llama_context_params` (without it, `embed()` crashes with a NULL pointer error).
	2. `embed()` now uses `llama_model_n_embd_out` for the vector size (otherwise embeddinggemma-2 returns 512 numbers instead of 768).

	## Tested
	`embeddinggemma-2-BF16.gguf` with `embedding=True, n_gpu_layers=-1` on a Colab T4: 768-number embeddings, all layers offloaded to the GPU.

	## Credits
	Built from [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) and [llama.cpp](https://github.com/ggml-org/llama.cpp), both MIT licensed.