File size: 14,284 Bytes
238fe38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b8f4463
238fe38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
624fa44
 
 
 
238fe38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
---
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - biology
  - single-cell
  - scRNA-seq
  - cell-type-annotation
---

# CellScribe

📜 **[Paper (bioRxiv 2026)](https://www.biorxiv.org/content/10.64898/2026.10.07.757352)**:
*From Closed-Set Annotation to In-Context Discovery: Biologically Grounded LLMs for Cellular
Profiling*

CellScribe is a generative framework that describes a cell along two complementary axes in a
single model: its *identity* (lineage) and its *state* (active transcriptional programs). It
conditions a pretrained language model
([Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)) on frozen embeddings
of a single-cell foundation model ([scFoundation](https://www.nature.com/articles/s41592-024-02305-7))
through adaptive layer normalization (AdaLN): the cell embedding sets the scale and shift of every
layer normalization of the language model. Depending on the task-specific prompt, the same model
generates:

- **Cell typing**: an ontology-grounded cell type (Cell Ontology vocabulary of CELLxGENE).
  Example: `"The cell type is B cell."`
- **Cell state**: a quantified meta-program profile. Meta-programs are the 41 recurrent
  malignant programs identified across 24 cancer types with the Curated Cancer Cell Atlas (3CA)
  by [Gavish et al., *Hallmarks of transcriptional intratumour heterogeneity across a thousand tumours*, Nature 2023](https://www.nature.com/articles/s41586-023-06130-4), and
  grouped into 11 hallmarks. CellScribe lists the active programs with their activity scores
  and the dominant hallmark.
  Example: `"Active programs: Cell Cycle - G2/M (1.9), EMT-II (1.1). Dominant hallmark: Cell cycle."`

The paper also describes reinforcement learning against cell-ontology distance and
meta-program agreement. Beyond these two tasks, CellScribe identifies cell types outside its
supervised vocabulary through in-context learning: the prompt lists a few labelled example
cells, each conditioning its own line, and the model infers the type of the query cell.

## Model

- **Cell encoder**: [scFoundation](https://www.nature.com/articles/s41592-024-02305-7)
  (Hao et al., 2024). The encoder embeds the cell's raw counts over 19,264 genes.
- **Conditioning**: a linear projection followed by a layer normalization turns the embedding
  into a conditioning vector. Through adaptive layer normalization (AdaLN,
  [Peebles and Xie, 2023](https://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_Diffusion_Models_with_Transformers_ICCV_2023_paper.html)), that vector sets the scale and
  shift of every layer normalization of the language model. The AdaLN projections are
  initialised to zero (AdaLN-Zero), so training starts from the unmodified language model.
- **Language model**: [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
  ([Qwen Team, 2024](https://qwenlm.github.io/blog/qwen2.5/)), prompted with a task-specific system
  prompt and question.

The model was trained on [CELLxGENE Census](https://chanzuckerberg.github.io/cellxgene-census/)
datasets (cell typing) and on the [Curated Cancer Cell Atlas, 3CA](https://www.weizmann.ac.il/sites/3CA/)
(cell typing and cell state).

## Usage

```bash
pip install "torch>=2.6" "transformers>=4.51,!=5.3.0" accelerate pandas
```

The model code is in this repository and is loaded with `trust_remote_code=True`. The
examples below use the example cells of this repository (`examples/`, raw counts with gene
symbols as columns), all held out of training:

- `cell_typing_sample.csv`: four CELLxGENE immune cells, for cell typing.
- `cell_state_sample.csv`: four malignant cells from the Curated Cancer Cell Atlas (3CA),
  for cell state.
- `icl_query_sample.csv` and `icl_examples_sample.csv`, for in-context learning: a query
  cell of a cell type held out of training, and labelled example cells (first column
  `cell_type`).

### Using the model directly

The model is a `torch.nn.Module`, which can be used in a custom inference loop or fine-tuned.

#### Loading

```python
import pandas as pd
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("InstaDeepAI/CellScribe", trust_remote_code=True).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("InstaDeepAI/CellScribe", trust_remote_code=True)


def load_example(name: str) -> pd.DataFrame:
    path = hf_hub_download("InstaDeepAI/CellScribe", f"examples/{name}.csv")
    return pd.read_csv(path, index_col=0)
```

#### Cell typing

```python
cells = load_example("cell_typing_sample")
scrnaseq_input = model.preprocess(cells.values, list(cells.columns))  # raw counts -> encoder inputs
prompt = model.build_prompt(tokenizer, task="cell_typing")
inputs = tokenizer([prompt] * len(cells), return_tensors="pt").to(model.device)
generated = model.generate(
    inputs.input_ids, inputs.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input
)
answers = tokenizer.batch_decode(generated, skip_special_tokens=True)
print([model.parse_cell_type(answer) for answer in answers])
# ['CD4-positive, alpha-beta cytotoxic T cell', 'immature B cell',
#  'CD16-negative, CD56-bright natural killer cell, human', 'CD14-low, CD16-positive monocyte']
```

#### Cell state

```python
cells = load_example("cell_state_sample")
scrnaseq_input = model.preprocess(cells.values, list(cells.columns))
prompt = model.build_prompt(tokenizer, task="cell_state")
inputs = tokenizer([prompt] * len(cells), return_tensors="pt").to(model.device)
generated = model.generate(
    inputs.input_ids, inputs.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input
)
for answer in tokenizer.batch_decode(generated, skip_special_tokens=True):
    programs, hallmark = model.parse_cell_state(answer)
    print(programs, hallmark)
# [('Stress (in vitro)', 0.9), ('Stress', 0.8), ('Translation initiation', 0.7), ('MYC-targets', 0.6)] Stress or hypoxia
# [('Unfolded protein response', 1.1), ('immunoglobulins (IGs)', 1.0), ('Protein maturation', 0.8), ('Interferon/MHC-II (I)', 0.8), ('Stress', 0.7)] Protein regulation
# [('Cell Cycle - G1/S', 1.8), ('Cell Cycle - G2/M', 1.7), ('Cell cycle (HMG-rich)', 1.4), ('MYC-targets', 0.9), ('EMT-II', 0.8)] Cell cycle
# [('MYC-targets', 1.0), ('Cell Cycle - G2/M', 0.9), ('Cell cycle (HMG-rich)', 0.7), ('Respiration', 0.6)] Oncogenic
```

#### In-context learning

Each example line of the prompt is conditioned by its own cell, the rest of the prompt and
the answer by the query cell.

```python
query = load_example("icl_query_sample")
examples = load_example("icl_examples_sample")
example_labels = examples.pop("cell_type").tolist()

with torch.no_grad():
    query_vectors = model.embed_cells(model.preprocess(query.values, list(query.columns)))
    example_vectors = model.embed_cells(model.preprocess(examples.values, list(examples.columns)))
prompt_ids, spans = model.build_in_context_prompt(tokenizer, example_labels)
plan = query_vectors.expand(len(prompt_ids), -1).clone()  # the query cell on every token,
for k, (start, end) in enumerate(spans):
    plan[start:end] = example_vectors[k]  # except the example lines
input_ids = torch.tensor([prompt_ids], device=model.device)
generated = model.generate(
    input_ids,
    torch.ones_like(input_ids),
    conditioning=plan[None],  # batch of one query
    conditioning_default=query_vectors,  # conditions the generated answer
)
print(tokenizer.batch_decode(generated, skip_special_tokens=True))
# ['The cell type is kidney loop of Henle thick ascending limb epithelial cell.']
```

#### Training

A teacher-forced loss on the answers (prompt tokens and padding masked out):

```python
cells = load_example("cell_typing_sample")
cell_types = [
    "CD4-positive, alpha-beta cytotoxic T cell",
    "immature B cell",
    "CD16-negative, CD56-bright natural killer cell, human",
    "CD14-low, CD16-positive monocyte",
]
scrnaseq_input = model.preprocess(cells.values, list(cells.columns))
prompt = model.build_prompt(tokenizer, task="cell_typing")
answers = [f"The cell type is {cell_type}.<|im_end|>\n" for cell_type in cell_types]
tokenizer.padding_side = "right"
batch = tokenizer([prompt + answer for answer in answers], return_tensors="pt", padding=True)
batch = batch.to(model.device)
labels = batch.input_ids.clone()
labels[:, : len(tokenizer(prompt).input_ids)] = -100
labels[batch.attention_mask == 0] = -100
model.train()
loss = model(
    batch.input_ids, batch.attention_mask, scrnaseq_preprocessed_input=scrnaseq_input, labels=labels
).loss
loss.backward()
```

### Using CellScribePredictor

`CellScribePredictor` wraps the model, tokenizer and preprocessing: raw counts in, parsed
answers out. It runs in float32 by default (pass `autocast_dtype="auto"` to any method for
bfloat16 on GPU, as in the paper's evaluation).

```python
predictor = model.predictor(tokenizer)  # model and tokenizer loaded as above
```

#### Cell typing

```python
cells = load_example("cell_typing_sample")
print(predictor.predict_cell_type(cells.values, list(cells.columns)))
# ['CD4-positive, alpha-beta cytotoxic T cell', 'immature B cell',
#  'CD16-negative, CD56-bright natural killer cell, human', 'CD14-low, CD16-positive monocyte']
```

#### Cell state

```python
cells = load_example("cell_state_sample")
for prediction in predictor.predict_cell_state(cells.values, list(cells.columns)):
    print(prediction.programs, prediction.hallmark)
# [('Stress (in vitro)', 0.9), ('Stress', 0.8), ('Translation initiation', 0.7), ('MYC-targets', 0.6)] Stress or hypoxia
# [('Unfolded protein response', 1.1), ('immunoglobulins (IGs)', 1.0), ('Protein maturation', 0.8), ('Interferon/MHC-II (I)', 0.8), ('Stress', 0.7)] Protein regulation
# [('Cell Cycle - G1/S', 1.8), ('Cell Cycle - G2/M', 1.7), ('Cell cycle (HMG-rich)', 1.4), ('MYC-targets', 0.9), ('EMT-II', 0.8)] Cell cycle
# [('MYC-targets', 1.0), ('Cell Cycle - G2/M', 0.9), ('Cell cycle (HMG-rich)', 0.7), ('Respiration', 0.6)] Oncogenic
```

An `AnnData` object holding raw counts can be used directly with
`predictor.predict_anndata(adata, task="cell_typing")` (or `task="cell_state"`).

#### In-context learning

```python
query = load_example("icl_query_sample")
examples = load_example("icl_examples_sample")
example_labels = examples.pop("cell_type").tolist()

# Without examples: the held-out cell type cannot be predicted
print(predictor.predict_cell_type(query.values, list(query.columns)))
# ['kidney distal convoluted tubule epithelial cell']

# With labelled examples
print(
    predictor.predict_cell_type_in_context(
        query.values, list(query.columns), examples.values, list(examples.columns), example_labels
    )
)
# ['kidney loop of Henle thick ascending limb epithelial cell']
```

## Input

- **Counts**: raw UMI counts, not normalised values. The model applies its own normalisation
  (CP10k + log1p).
- **Gene names**: HGNC symbols. For CELLxGENE files, set
  `adata.var_names = adata.var["feature_name"]`.
- **Genes**: the cell encoder reads a fixed list of 19,264 genes (`gene_list.txt`). Datasets
  rarely measure exactly these genes, so `model.preprocess` maps the input genes onto this
  list: listed genes absent from the input are given a count of zero, and genes outside the
  list are dropped.

## Files

| File | Content |
|---|---|
| `config.json`, `model.safetensors` | model hyper-parameters and weights |
| `*.py` | the model code, loaded with `trust_remote_code=True` |
| tokenizer files | Qwen2.5 tokenizer |
| `gene_list.txt` | the 19,264 genes read by the cell encoder, in order |
| `cell_types.txt` | cell-type vocabulary used to read cell-typing answers |
| `examples/` | example cells of this card (raw counts, gene symbols as columns) |
| `LICENSE.md`, `NOTICE` | license terms (see below) |

## License

CellScribe is released by InstaDeep Ltd and CentraleSupélec under the
[Apache License 2.0](LICENSE.md), except for its cell encoder: the scFoundation encoder weights
(`scrna_encoder.*`, unmodified) remain under the
[scFoundation Model License](https://github.com/biomap-research/scFoundation/blob/main/MODEL_LICENSE)
of BioMap, which allows non-commercial research use only and excludes any commercial, military
or illegal purpose. The language model is derived from
[Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) (Apache 2.0). See
[`NOTICE`](NOTICE) for the third-party terms, including the scFoundation Model License.

The example cells in `examples/` are from [CELLxGENE](https://cellxgene.cziscience.com/) and the
[Curated Cancer Cell Atlas (3CA)](https://www.weizmann.ac.il/sites/3CA/) and remain under their
sources' terms.

## Citation

```bibtex
@article{gelard2026closed,
  title={From Closed-Set Annotation to In-Context Discovery: Biologically Grounded LLMs for Cellular Profiling},
  author={Gelard, Maxence and Benkirane, Hakim and Pierrot, Thomas and Cournede, Paul-Henry and P. de Almeida, Bernardo},
  journal={bioRxiv},
  pages={2026--10},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}
```

The cell encoder is scFoundation:

```bibtex
@article{hao2024large,
  title={Large-scale foundation model on single-cell transcriptomics},
  author={Hao, Minsheng and Gong, Jing and Zeng, Xin and Liu, Chiming and Guo, Yucheng and Cheng, Xingyi and Wang, Taifeng and Ma, Jianzhu and Zhang, Xuegong and Song, Le},
  journal={Nature Methods},
  volume={21},
  number={8},
  pages={1481--1491},
  year={2024},
  publisher={Nature Publishing Group}
}
```

The language model is Qwen2.5-0.5B-Instruct:

```bibtex
@article{qwen2024qwen25,
  title={{Qwen2.5} Technical Report},
  author={{Qwen Team}},
  journal={arXiv preprint},
  year={2024}
}
```

The cell-state task uses the meta-programs of the Curated Cancer Cell Atlas:

```bibtex
@article{gavish2023hallmarks,
  title={Hallmarks of transcriptional intratumour heterogeneity across a thousand tumours},
  author={Gavish, Avishai and Tyler, Michael and Greenwald, Alissa C. and Hoefflin, Rouven and Simkin, Dor and Tschernichovsky, Roi and Galili Darnell, Noam and Somech, Einav and Barbolin, Chaya and Antman, Tomer and others},
  journal={Nature},
  volume={618},
  number={7965},
  pages={598--606},
  year={2023},
  publisher={Nature Publishing Group}
}
```