Text Ranking
sentence-transformers
Safetensors
Transformers
multilingual
t5gemma2
text2text-generation
reranker
encoder-decoder
FBNL
matryoshka
retrieval
RAG
Instructions to use KaLM-Embedding/KaLM-Reranker-V1-Small-R2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-R2 with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("KaLM-Embedding/KaLM-Reranker-V1-Small-R2") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores) - Transformers
How to use KaLM-Embedding/KaLM-Reranker-V1-Small-R2 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KaLM-Embedding/KaLM-Reranker-V1-Small-R2") model = AutoModelForMultimodalLM.from_pretrained("KaLM-Embedding/KaLM-Reranker-V1-Small-R2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update vllm_support/README.md
Browse files- vllm_support/README.md +15 -15
vllm_support/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
-
# KaLM-Reranker-V1-Small vLLM Support
|
| 2 |
|
| 3 |
This directory contains the experimental vLLM 0.19.1 adapter for
|
| 4 |
-
`KaLM-Embedding/KaLM-Reranker-V1-Small`. It supports offline Python and CLI
|
| 5 |
reranking plus an optional FastAPI service.
|
| 6 |
|
| 7 |
The adapter does not modify or retrain the checkpoint. It reads the original
|
|
@@ -35,9 +35,9 @@ conda activate kalm-vllm
|
|
| 35 |
pip install "vllm==0.19.1" "transformers==5.6.2"
|
| 36 |
pip install "fastapi>=0.136,<0.137" "uvicorn>=0.46,<0.47"
|
| 37 |
|
| 38 |
-
hf download KaLM-Embedding/KaLM-Reranker-V1-Small \
|
| 39 |
-
--local-dir ./KaLM-Reranker-V1-Small
|
| 40 |
-
pip install ./KaLM-Reranker-V1-Small/vllm_support --no-deps
|
| 41 |
export VLLM_PLUGINS=kalm_t5gemma2
|
| 42 |
```
|
| 43 |
|
|
@@ -45,10 +45,10 @@ The model can also be loaded directly by its Hugging Face ID. In that case,
|
|
| 45 |
only download the `vllm_support` directory before installing the plugin:
|
| 46 |
|
| 47 |
```bash
|
| 48 |
-
hf download KaLM-Embedding/KaLM-Reranker-V1-Small \
|
| 49 |
--include "vllm_support/**" \
|
| 50 |
-
--local-dir ./KaLM-Reranker-V1-Small
|
| 51 |
-
pip install ./KaLM-Reranker-V1-Small/vllm_support --no-deps
|
| 52 |
```
|
| 53 |
|
| 54 |
## Offline Python API
|
|
@@ -64,7 +64,7 @@ documents = [
|
|
| 64 |
|
| 65 |
pairs = [(query, document) for document in documents]
|
| 66 |
with KaLMVLLMReranker(
|
| 67 |
-
"KaLM-Embedding/KaLM-Reranker-V1-Small",
|
| 68 |
query_max_length=512,
|
| 69 |
document_max_length=1024,
|
| 70 |
encoder_chunk_size=4,
|
|
@@ -97,7 +97,7 @@ Score JSONL input:
|
|
| 97 |
|
| 98 |
```bash
|
| 99 |
kalm-vllm-rerank \
|
| 100 |
-
--input-jsonl ./KaLM-Reranker-V1-Small/vllm_support/examples/sample_pairs.jsonl \
|
| 101 |
--output-jsonl ./scores.jsonl \
|
| 102 |
--return-margin
|
| 103 |
```
|
|
@@ -114,7 +114,7 @@ Start one model instance:
|
|
| 114 |
kalm-vllm-serve \
|
| 115 |
--host 0.0.0.0 \
|
| 116 |
--port 8000 \
|
| 117 |
-
--model KaLM-Embedding/KaLM-Reranker-V1-Small \
|
| 118 |
--encoder-chunk-size 4
|
| 119 |
```
|
| 120 |
|
|
@@ -123,7 +123,7 @@ variables:
|
|
| 123 |
|
| 124 |
```bash
|
| 125 |
CUDA_VISIBLE_DEVICES=0 PORT=8000 \
|
| 126 |
-
./KaLM-Reranker-V1-Small/vllm_support/examples/start_online_server.sh
|
| 127 |
```
|
| 128 |
|
| 129 |
In a second terminal, check health and send built-in demo requests:
|
|
@@ -140,7 +140,7 @@ one query against multiple documents:
|
|
| 140 |
```bash
|
| 141 |
kalm-vllm-client \
|
| 142 |
--endpoint rerank \
|
| 143 |
-
--json-file ./KaLM-Reranker-V1-Small/vllm_support/examples/rerank_request.json \
|
| 144 |
--return-margin \
|
| 145 |
--top-k 10
|
| 146 |
```
|
|
@@ -150,7 +150,7 @@ Use `/score` for a batch of independent query-document pairs:
|
|
| 150 |
```bash
|
| 151 |
kalm-vllm-client \
|
| 152 |
--endpoint score \
|
| 153 |
-
--json-file ./KaLM-Reranker-V1-Small/vllm_support/examples/score_request.json \
|
| 154 |
--return-margin
|
| 155 |
```
|
| 156 |
|
|
@@ -247,7 +247,7 @@ Reinstall the package and ensure the environment variable includes its entry
|
|
| 247 |
point name:
|
| 248 |
|
| 249 |
```bash
|
| 250 |
-
pip install ./KaLM-Reranker-V1-Small/vllm_support --no-deps --force-reinstall
|
| 251 |
export VLLM_PLUGINS=kalm_t5gemma2
|
| 252 |
```
|
| 253 |
|
|
|
|
| 1 |
+
# KaLM-Reranker-V1-Small-R2 vLLM Support
|
| 2 |
|
| 3 |
This directory contains the experimental vLLM 0.19.1 adapter for
|
| 4 |
+
`KaLM-Embedding/KaLM-Reranker-V1-Small-R2`. It supports offline Python and CLI
|
| 5 |
reranking plus an optional FastAPI service.
|
| 6 |
|
| 7 |
The adapter does not modify or retrain the checkpoint. It reads the original
|
|
|
|
| 35 |
pip install "vllm==0.19.1" "transformers==5.6.2"
|
| 36 |
pip install "fastapi>=0.136,<0.137" "uvicorn>=0.46,<0.47"
|
| 37 |
|
| 38 |
+
hf download KaLM-Embedding/KaLM-Reranker-V1-Small-R2 \
|
| 39 |
+
--local-dir ./KaLM-Reranker-V1-Small-R2
|
| 40 |
+
pip install ./KaLM-Reranker-V1-Small-R2/vllm_support --no-deps
|
| 41 |
export VLLM_PLUGINS=kalm_t5gemma2
|
| 42 |
```
|
| 43 |
|
|
|
|
| 45 |
only download the `vllm_support` directory before installing the plugin:
|
| 46 |
|
| 47 |
```bash
|
| 48 |
+
hf download KaLM-Embedding/KaLM-Reranker-V1-Small-R2 \
|
| 49 |
--include "vllm_support/**" \
|
| 50 |
+
--local-dir ./KaLM-Reranker-V1-Small-R2
|
| 51 |
+
pip install ./KaLM-Reranker-V1-Small-R2/vllm_support --no-deps
|
| 52 |
```
|
| 53 |
|
| 54 |
## Offline Python API
|
|
|
|
| 64 |
|
| 65 |
pairs = [(query, document) for document in documents]
|
| 66 |
with KaLMVLLMReranker(
|
| 67 |
+
"KaLM-Embedding/KaLM-Reranker-V1-Small-R2",
|
| 68 |
query_max_length=512,
|
| 69 |
document_max_length=1024,
|
| 70 |
encoder_chunk_size=4,
|
|
|
|
| 97 |
|
| 98 |
```bash
|
| 99 |
kalm-vllm-rerank \
|
| 100 |
+
--input-jsonl ./KaLM-Reranker-V1-Small-R2/vllm_support/examples/sample_pairs.jsonl \
|
| 101 |
--output-jsonl ./scores.jsonl \
|
| 102 |
--return-margin
|
| 103 |
```
|
|
|
|
| 114 |
kalm-vllm-serve \
|
| 115 |
--host 0.0.0.0 \
|
| 116 |
--port 8000 \
|
| 117 |
+
--model KaLM-Embedding/KaLM-Reranker-V1-Small-R2 \
|
| 118 |
--encoder-chunk-size 4
|
| 119 |
```
|
| 120 |
|
|
|
|
| 123 |
|
| 124 |
```bash
|
| 125 |
CUDA_VISIBLE_DEVICES=0 PORT=8000 \
|
| 126 |
+
./KaLM-Reranker-V1-Small-R2/vllm_support/examples/start_online_server.sh
|
| 127 |
```
|
| 128 |
|
| 129 |
In a second terminal, check health and send built-in demo requests:
|
|
|
|
| 140 |
```bash
|
| 141 |
kalm-vllm-client \
|
| 142 |
--endpoint rerank \
|
| 143 |
+
--json-file ./KaLM-Reranker-V1-Small-R2/vllm_support/examples/rerank_request.json \
|
| 144 |
--return-margin \
|
| 145 |
--top-k 10
|
| 146 |
```
|
|
|
|
| 150 |
```bash
|
| 151 |
kalm-vllm-client \
|
| 152 |
--endpoint score \
|
| 153 |
+
--json-file ./KaLM-Reranker-V1-Small-R2/vllm_support/examples/score_request.json \
|
| 154 |
--return-margin
|
| 155 |
```
|
| 156 |
|
|
|
|
| 247 |
point name:
|
| 248 |
|
| 249 |
```bash
|
| 250 |
+
pip install ./KaLM-Reranker-V1-Small-R2/vllm_support --no-deps --force-reinstall
|
| 251 |
export VLLM_PLUGINS=kalm_t5gemma2
|
| 252 |
```
|
| 253 |
|