Lighteval documentation
Installation
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v0.13.0).
Installation
Lighteval can be installed from PyPI or from source. This guide covers all installation options and dependencies.
System Requirements
- Python: 3.10 or higher
- PyTorch: 2.0 or higher (but less than 3.0)
- CUDA: Optional, for GPU acceleration
From PyPI
The simplest way to install Lighteval is from PyPI:
pip install lighteval
This installs the core package with all essential dependencies for basic evaluation tasks.
From Source
Source installation is recommended for developers who want to contribute to Lighteval or need the latest features:
git clone https://github.com/huggingface/lighteval.git
cd lighteval
pip install -e .Optional Dependencies (Extras)
Lighteval provides several optional dependency groups that you can install based on your needs. Use the format pip install lighteval[<group>] or pip install -e .[<group>] for source installation.
Backend Extras
| Extra | Description | Dependencies |
|---|---|---|
vllm | Use VLLM as backend for high-performance inference | vllm>=0.10.0, ray, more_itertools |
tgi | Use Text Generation Inference API | text-generation>=0.6.0 |
litellm | Use LiteLLM for unified API access | litellm, diskcache |
optimum | Use Optimum for optimized models | optimum==1.12.0 |
quantization | Evaluate quantized models | bitsandbytes>=0.41.0, auto-gptq>=0.4.2 |
adapters | Evaluate adapter models (PEFT, Delta) | peft==0.3.0 |
nanotron | Evaluate Nanotron models | nanotron, tensorboardX |
Task and Feature Extras
| Extra | Description | Dependencies |
|---|---|---|
extended_tasks | Extended evaluation tasks | langdetect, openai>1.87, tiktoken |
multilingual | Multilingual evaluation support | stanza, spacy[ja,ko,th], jieba, pyvi |
math | Mathematical reasoning tasks | latex2sympy2_extended==1.0.6 |
Storage and Logging Extras
| Extra | Description | Dependencies |
|---|---|---|
s3 | Upload results to S3 | s3fs |
tensorboardX | Upload results to TensorBoard | tensorboardX |
wandb | Log results to Weights & Biases | wandb |
trackio | Log results to Trackio | trackio |
Development Extras
| Extra | Description | Dependencies |
|---|---|---|
quality | Code quality tools | ruff>=v0.11.0, pre-commit |
tests | Testing dependencies | pytest>=7.4.0, deepdiff |
docs | Documentation building | hf-doc-builder, watchdog |
dev | All development dependencies | Includes accelerate, quality, tests, multilingual, math, extended_tasks, vllm |