File size: 14,284 Bytes
238fe38 b8f4463 238fe38 624fa44 238fe38 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 | ---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- biology
- single-cell
- scRNA-seq
- cell-type-annotation
---
# CellScribe
📜 **[Paper (bioRxiv 2026)](https://www.biorxiv.org/content/10.64898/2026.10.07.757352)**:
*From Closed-Set Annotation to In-Context Discovery: Biologically Grounded LLMs for Cellular
Profiling*
CellScribe is a generative framework that describes a cell along two complementary axes in a
single model: its *identity* (lineage) and its *state* (active transcriptional programs). It
conditions a pretrained language model
([Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)) on frozen embeddings
of a single-cell foundation model ([scFoundation](https://www.nature.com/articles/s41592-024-02305-7))
through adaptive layer normalization (AdaLN): the cell embedding sets the scale and shift of every
layer normalization of the language model. Depending on the task-specific prompt, the same model
generates:
- **Cell typing**: an ontology-grounded cell type (Cell Ontology vocabulary of CELLxGENE).
Example: `"The cell type is B cell."`
- **Cell state**: a quantified meta-program profile. Meta-programs are the 41 recurrent
malignant programs identified across 24 cancer types with the Curated Cancer Cell Atlas (3CA)
by [Gavish et al., *Hallmarks of transcriptional intratumour heterogeneity across a thousand tumours*, Nature 2023](https://www.nature.com/articles/s41586-023-06130-4), and
grouped into 11 hallmarks. CellScribe lists the active programs with their activity scores
and the dominant hallmark.
Example: `"Active programs: Cell Cycle - G2/M (1.9), EMT-II (1.1). Dominant hallmark: Cell cycle."`
The paper also describes reinforcement learning against cell-ontology distance and
meta-program agreement. Beyond these two tasks, CellScribe identifies cell types outside its
supervised vocabulary through in-context learning: the prompt lists a few labelled example
cells, each conditioning its own line, and the model infers the type of the query cell.
## Model
- **Cell encoder**: [scFoundation](https://www.nature.com/articles/s41592-024-02305-7)
(Hao et al., 2024). The encoder embeds the cell's raw counts over 19,264 genes.
- **Conditioning**: a linear projection followed by a layer normalization turns the embedding
into a conditioning vector. Through adaptive layer normalization (AdaLN,
[Peebles and Xie, 2023](https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_Diffusion_Models_with_Transformers_ICCV_2023_paper.html)), that vector sets the scale and
shift of every layer normalization of the language model. The AdaLN projections are
initialised to zero (AdaLN-Zero), so training starts from the unmodified language model.
- **Language model**: [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
([Qwen Team, 2024](https://qwenlm.github.io/blog/qwen2.5/)), prompted with a task-specific system
prompt and question.
The model was trained on [CELLxGENE Census](https://chanzuckerberg.github.io/cellxgene-census/)
datasets (cell typing) and on the [Curated Cancer Cell Atlas, 3CA](https://www.weizmann.ac.il/sites/3CA/)
(cell typing and cell state).
## Usage
```bash
pip install "torch>=2.6" "transformers>=4.51,!=5.3.0" accelerate pandas
```
The model code is in this repository and is loaded with `trust_remote_code=True`. The
examples below use the example cells of this repository (`examples/`, raw counts with gene
symbols as columns), all held out of training:
- `cell_typing_sample.csv`: four CELLxGENE immune cells, for cell typing.
- `cell_state_sample.csv`: four malignant cells from the Curated Cancer Cell Atlas (3CA),
for cell state.
- `icl_query_sample.csv` and `icl_examples_sample.csv`, for in-context learning: a query
cell of a cell type held out of training, and labelled example cells (first column
`cell_type`).
### Using the model directly
The model is a `torch.nn.Module`, which can be used in a custom inference loop or fine-tuned.
#### Loading
```python
import pandas as pd
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("InstaDeepAI/CellScribe", trust_remote_code=True).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("InstaDeepAI/CellScribe", trust_remote_code=True)
def load_example(name: str) -> pd.DataFrame:
path = hf_hub_download("InstaDeepAI/CellScribe", f"examples/{name}.csv")
return pd.read_csv(path, index_col=0)
```
#### Cell typing
```python
cells = load_example("cell_typing_sample")
scrnaseq_input = model.preprocess(cells.values, list(cells.columns)) # raw counts -> encoder inputs
prompt = model.build_prompt(tokenizer, task="cell_typing")
inputs = tokenizer([prompt] * len(cells), return_tensors="pt").to(model.device)
generated = model.generate(
inputs.input_ids, inputs.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input
)
answers = tokenizer.batch_decode(generated, skip_special_tokens=True)
print([model.parse_cell_type(answer) for answer in answers])
# ['CD4-positive, alpha-beta cytotoxic T cell', 'immature B cell',
# 'CD16-negative, CD56-bright natural killer cell, human', 'CD14-low, CD16-positive monocyte']
```
#### Cell state
```python
cells = load_example("cell_state_sample")
scrnaseq_input = model.preprocess(cells.values, list(cells.columns))
prompt = model.build_prompt(tokenizer, task="cell_state")
inputs = tokenizer([prompt] * len(cells), return_tensors="pt").to(model.device)
generated = model.generate(
inputs.input_ids, inputs.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input
)
for answer in tokenizer.batch_decode(generated, skip_special_tokens=True):
programs, hallmark = model.parse_cell_state(answer)
print(programs, hallmark)
# [('Stress (in vitro)', 0.9), ('Stress', 0.8), ('Translation initiation', 0.7), ('MYC-targets', 0.6)] Stress or hypoxia
# [('Unfolded protein response', 1.1), ('immunoglobulins (IGs)', 1.0), ('Protein maturation', 0.8), ('Interferon/MHC-II (I)', 0.8), ('Stress', 0.7)] Protein regulation
# [('Cell Cycle - G1/S', 1.8), ('Cell Cycle - G2/M', 1.7), ('Cell cycle (HMG-rich)', 1.4), ('MYC-targets', 0.9), ('EMT-II', 0.8)] Cell cycle
# [('MYC-targets', 1.0), ('Cell Cycle - G2/M', 0.9), ('Cell cycle (HMG-rich)', 0.7), ('Respiration', 0.6)] Oncogenic
```
#### In-context learning
Each example line of the prompt is conditioned by its own cell, the rest of the prompt and
the answer by the query cell.
```python
query = load_example("icl_query_sample")
examples = load_example("icl_examples_sample")
example_labels = examples.pop("cell_type").tolist()
with torch.no_grad():
query_vectors = model.embed_cells(model.preprocess(query.values, list(query.columns)))
example_vectors = model.embed_cells(model.preprocess(examples.values, list(examples.columns)))
prompt_ids, spans = model.build_in_context_prompt(tokenizer, example_labels)
plan = query_vectors.expand(len(prompt_ids), -1).clone() # the query cell on every token,
for k, (start, end) in enumerate(spans):
plan[start:end] = example_vectors[k] # except the example lines
input_ids = torch.tensor([prompt_ids], device=model.device)
generated = model.generate(
input_ids,
torch.ones_like(input_ids),
conditioning=plan[None], # batch of one query
conditioning_default=query_vectors, # conditions the generated answer
)
print(tokenizer.batch_decode(generated, skip_special_tokens=True))
# ['The cell type is kidney loop of Henle thick ascending limb epithelial cell.']
```
#### Training
A teacher-forced loss on the answers (prompt tokens and padding masked out):
```python
cells = load_example("cell_typing_sample")
cell_types = [
"CD4-positive, alpha-beta cytotoxic T cell",
"immature B cell",
"CD16-negative, CD56-bright natural killer cell, human",
"CD14-low, CD16-positive monocyte",
]
scrnaseq_input = model.preprocess(cells.values, list(cells.columns))
prompt = model.build_prompt(tokenizer, task="cell_typing")
answers = [f"The cell type is {cell_type}.<|im_end|>\n" for cell_type in cell_types]
tokenizer.padding_side = "right"
batch = tokenizer([prompt + answer for answer in answers], return_tensors="pt", padding=True)
batch = batch.to(model.device)
labels = batch.input_ids.clone()
labels[:, : len(tokenizer(prompt).input_ids)] = -100
labels[batch.attention_mask == 0] = -100
model.train()
loss = model(
batch.input_ids, batch.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input, labels=labels
).loss
loss.backward()
```
### Using CellScribePredictor
`CellScribePredictor` wraps the model, tokenizer and preprocessing: raw counts in, parsed
answers out. It runs in float32 by default (pass `autocast_dtype="auto"` to any method for
bfloat16 on GPU, as in the paper's evaluation).
```python
predictor = model.predictor(tokenizer) # model and tokenizer loaded as above
```
#### Cell typing
```python
cells = load_example("cell_typing_sample")
print(predictor.predict_cell_type(cells.values, list(cells.columns)))
# ['CD4-positive, alpha-beta cytotoxic T cell', 'immature B cell',
# 'CD16-negative, CD56-bright natural killer cell, human', 'CD14-low, CD16-positive monocyte']
```
#### Cell state
```python
cells = load_example("cell_state_sample")
for prediction in predictor.predict_cell_state(cells.values, list(cells.columns)):
print(prediction.programs, prediction.hallmark)
# [('Stress (in vitro)', 0.9), ('Stress', 0.8), ('Translation initiation', 0.7), ('MYC-targets', 0.6)] Stress or hypoxia
# [('Unfolded protein response', 1.1), ('immunoglobulins (IGs)', 1.0), ('Protein maturation', 0.8), ('Interferon/MHC-II (I)', 0.8), ('Stress', 0.7)] Protein regulation
# [('Cell Cycle - G1/S', 1.8), ('Cell Cycle - G2/M', 1.7), ('Cell cycle (HMG-rich)', 1.4), ('MYC-targets', 0.9), ('EMT-II', 0.8)] Cell cycle
# [('MYC-targets', 1.0), ('Cell Cycle - G2/M', 0.9), ('Cell cycle (HMG-rich)', 0.7), ('Respiration', 0.6)] Oncogenic
```
An `AnnData` object holding raw counts can be used directly with
`predictor.predict_anndata(adata, task="cell_typing")` (or `task="cell_state"`).
#### In-context learning
```python
query = load_example("icl_query_sample")
examples = load_example("icl_examples_sample")
example_labels = examples.pop("cell_type").tolist()
# Without examples: the held-out cell type cannot be predicted
print(predictor.predict_cell_type(query.values, list(query.columns)))
# ['kidney distal convoluted tubule epithelial cell']
# With labelled examples
print(
predictor.predict_cell_type_in_context(
query.values, list(query.columns), examples.values, list(examples.columns), example_labels
)
)
# ['kidney loop of Henle thick ascending limb epithelial cell']
```
## Input
- **Counts**: raw UMI counts, not normalised values. The model applies its own normalisation
(CP10k + log1p).
- **Gene names**: HGNC symbols. For CELLxGENE files, set
`adata.var_names = adata.var["feature_name"]`.
- **Genes**: the cell encoder reads a fixed list of 19,264 genes (`gene_list.txt`). Datasets
rarely measure exactly these genes, so `model.preprocess` maps the input genes onto this
list: listed genes absent from the input are given a count of zero, and genes outside the
list are dropped.
## Files
| File | Content |
|---|---|
| `config.json`, `model.safetensors` | model hyper-parameters and weights |
| `*.py` | the model code, loaded with `trust_remote_code=True` |
| tokenizer files | Qwen2.5 tokenizer |
| `gene_list.txt` | the 19,264 genes read by the cell encoder, in order |
| `cell_types.txt` | cell-type vocabulary used to read cell-typing answers |
| `examples/` | example cells of this card (raw counts, gene symbols as columns) |
| `LICENSE.md`, `NOTICE` | license terms (see below) |
## License
CellScribe is released by InstaDeep Ltd and CentraleSupélec under the
[Apache License 2.0](LICENSE.md), except for its cell encoder: the scFoundation encoder weights
(`scrna_encoder.*`, unmodified) remain under the
[scFoundation Model License](https://github.com/biomap-research/scFoundation/blob/main/MODEL_LICENSE)
of BioMap, which allows non-commercial research use only and excludes any commercial, military
or illegal purpose. The language model is derived from
[Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) (Apache 2.0). See
[`NOTICE`](NOTICE) for the third-party terms, including the scFoundation Model License.
The example cells in `examples/` are from [CELLxGENE](https://cellxgene.cziscience.com/) and the
[Curated Cancer Cell Atlas (3CA)](https://www.weizmann.ac.il/sites/3CA/) and remain under their
sources' terms.
## Citation
```bibtex
@article{gelard2026closed,
title={From Closed-Set Annotation to In-Context Discovery: Biologically Grounded LLMs for Cellular Profiling},
author={Gelard, Maxence and Benkirane, Hakim and Pierrot, Thomas and Cournede, Paul-Henry and P. de Almeida, Bernardo},
journal={bioRxiv},
pages={2026--10},
year={2026},
publisher={Cold Spring Harbor Laboratory}
}
```
The cell encoder is scFoundation:
```bibtex
@article{hao2024large,
title={Large-scale foundation model on single-cell transcriptomics},
author={Hao, Minsheng and Gong, Jing and Zeng, Xin and Liu, Chiming and Guo, Yucheng and Cheng, Xingyi and Wang, Taifeng and Ma, Jianzhu and Zhang, Xuegong and Song, Le},
journal={Nature Methods},
volume={21},
number={8},
pages={1481--1491},
year={2024},
publisher={Nature Publishing Group}
}
```
The language model is Qwen2.5-0.5B-Instruct:
```bibtex
@article{qwen2024qwen25,
title={{Qwen2.5} Technical Report},
author={{Qwen Team}},
journal={arXiv preprint},
year={2024}
}
```
The cell-state task uses the meta-programs of the Curated Cancer Cell Atlas:
```bibtex
@article{gavish2023hallmarks,
title={Hallmarks of transcriptional intratumour heterogeneity across a thousand tumours},
author={Gavish, Avishai and Tyler, Michael and Greenwald, Alissa C. and Hoefflin, Rouven and Simkin, Dor and Tschernichovsky, Roi and Galili Darnell, Noam and Somech, Einav and Barbolin, Chaya and Antman, Tomer and others},
journal={Nature},
volume={618},
number={7965},
pages={598--606},
year={2023},
publisher={Nature Publishing Group}
}
```
|