File size: 25,758 Bytes
4ba57bc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9d9a312
4ba57bc
9d9a312
 
4ba57bc
9d9a312
 
2419a3b
4ba57bc
9d9a312
 
2419a3b
4ba57bc
9d9a312
 
2419a3b
4ba57bc
 
 
9d9a312
4ba57bc
9d9a312
 
4ba57bc
 
 
2419a3b
9d9a312
 
 
2419a3b
4ba57bc
 
 
9d9a312
4ba57bc
9d9a312
4ba57bc
 
 
 
2419a3b
4ba57bc
 
 
2419a3b
4ba57bc
 
 
 
 
 
 
 
 
9d9a312
 
2419a3b
 
 
4ba57bc
 
 
2419a3b
9d9a312
 
 
2419a3b
 
 
 
9d9a312
2419a3b
 
4ba57bc
 
 
9d9a312
 
 
4ba57bc
 
 
 
9d9a312
4ba57bc
 
 
9d9a312
4ba57bc
 
 
 
 
 
9d9a312
 
4ba57bc
 
 
 
 
 
9d9a312
 
 
 
 
 
 
 
 
4ba57bc
 
 
 
 
 
9d9a312
4ba57bc
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9d9a312
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4ba57bc
 
 
 
 
 
9d9a312
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4ba57bc
 
 
9d9a312
4ba57bc
9d9a312
 
4ba57bc
9d9a312
 
4ba57bc
9d9a312
 
 
4ba57bc
 
 
9d9a312
4ba57bc
9d9a312
 
 
4ba57bc
 
 
9d9a312
4ba57bc
 
 
 
 
9d9a312
2419a3b
 
9d9a312
2419a3b
 
 
 
9d9a312
4ba57bc
2419a3b
 
4ba57bc
 
9d9a312
4ba57bc
 
 
9d9a312
4ba57bc
9d9a312
4ba57bc
9d9a312
 
 
 
 
 
 
4ba57bc
9d9a312
2419a3b
 
 
9d9a312
2419a3b
9d9a312
2419a3b
 
 
 
9d9a312
 
 
 
 
 
2419a3b
9d9a312
 
 
 
 
 
2419a3b
 
9d9a312
 
 
 
 
 
 
 
 
 
 
 
 
2419a3b
 
9d9a312
2419a3b
 
9d9a312
 
2419a3b
 
 
 
 
 
 
 
 
 
 
 
 
4ba57bc
 
 
9d9a312
2419a3b
 
9d9a312
 
 
2419a3b
 
9d9a312
 
 
4ba57bc
 
 
 
 
9d9a312
2419a3b
4ba57bc
9d9a312
4ba57bc
 
 
 
9d9a312
4ba57bc
 
 
 
 
 
 
 
 
 
9d9a312
2419a3b
9d9a312
 
 
 
2419a3b
 
 
 
 
 
9d9a312
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language:
- en
- pt
tags:
- mongodb
- code
- fim
- autocomplete
- qwen2
- synthetic-data
model-index:
- name: KapiCoder-Mongo-Small
  results:
  - task:
      type: text-generation
      name: MongoDB autocomplete with the cursor inside a call (editor-context-v2)
    dataset:
      name: Internal evaluation on synthetic data, cursor-in-call suite v1 (n=1200, held-out schemas)
      type: kapicoder-cursor-in-call-v1-synthetic
    metrics:
    - type: suffix_consistent
      name: Document parses after inserting the suggestion (64 tokens, v2 post-processing)
      value: 0.9492
      verified: false
    - type: op_valid_in_position
      name: Every operator is legal where it is (64 tokens, v2 post-processing)
      value: 0.9483
      verified: false
    - type: field_grounded_full_context
      name: Cited fields exist in the target collection, schema in the header (context_kind=full)
      value: 0.9917
      verified: false
  - task:
      type: text-generation
      name: MongoDB fill-in-the-middle autocomplete (previous editor-context-v1 benchmark)
    dataset:
      name: KapiCoder v2c validation split, internal and synthetic (n=2000)
      type: kapicoder-v2c-val-synthetic
    metrics:
    - type: fim_accuracy
      name: FIM accuracy (normalized exact match)
      value: 0.399
      verified: false
    - type: exec_valid
      name: Exec-valid (completion parses and runs read-only)
      value: 0.5815
      verified: false
  - task:
      type: text-generation
      name: MongoDB fill-in-the-middle autocomplete on unseen schemas (editor-context-v1)
    dataset:
      name: Internal evaluation, synthetic out-of-distribution homologation v1 (n=6030, LLM- and grammar-written; not fully independent of the training data)
      type: kapicoder-homolog-v1-synthetic-ood
    metrics:
    - type: fim_accuracy
      name: FIM accuracy (normalized exact match), homologation
      value: 0.4461
      verified: false
    - type: exec_valid
      name: Exec-valid (completion parses and runs read-only), homologation
      value: 0.8426
      verified: false
---

# KapiCoder-Mongo-Small (1.5B)

> **Status: PREVIEW / experimental, public preview.** Published for inspection and feedback, not for production use.
> **Internal evaluation on synthetic data; no blind human-written set.** Read [Limitations and risks](#limitations-and-risks) before using it.

KapiCoder-Mongo-Small is a small code-completion model (about 1.5 B parameters) fine-tuned from
[`Qwen/Qwen2.5-Coder-1.5B`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B) to **complete MongoDB queries at the cursor** (fill-in-the-middle, FIM),
including with the cursor **inside** a `find`, `update` or `aggregate` call. It was built for the *Kapibara Studio* editor, which sends it the text
before and after the cursor plus a context header (the `editor-context-v2` contract described below). This is version **v4b** of the weights
(repository tag `v0.3.0-preview`). It replaces v4 (same contract), which stays available at tag [`v0.2.0-preview`](https://huggingface.co/esilva/KapiCoder-Mongo-Small/tree/v0.2.0-preview);
v3 (`editor-context-v1`) stays available at tag [`v0.1.0-preview`](https://huggingface.co/esilva/KapiCoder-Mongo-Small/tree/v0.1.0-preview).

**Read this first**

- **v4b expects the `editor-context-v2` prompt** (the v1 header plus a `TARGET FIELDS` line with the target collection's fields and types) and its
  output rule (close the inner object the suggestion left open). It was trained with an 8k-token window. Versions of the Kapibara Studio editor without
  `editor-context-v2` reject this package; use tag `v0.1.0-preview` with them.
- **Internal evaluation, synthetic data, no human-written queries.** Every number below comes from the authors' own evaluation on synthetic data.
- **Better with the cursor inside a call; close to v3 with the old prompt.** On the cursor-in-call suite (v2 prompt) the
  document stays valid after the suggestion 94.9% of the time (v3: 87.8%). With the previous
  `editor-context-v1` prompt (no `TARGET FIELDS`), on the out-of-distribution homologation, FIM accuracy is 44.61% against
  45.37% for v3 and exec-valid 84.26% against 83.07% (paired differences below).
- **Send the schema.** With the target fields in the header, 99.2% of the fields it cites exist; with no header at all,
  50.0% (v3: 54.5%). Validate field names before running a suggestion.
- **Two training runs (seeds); these are the weights of the first.** Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not. Second-seed values are in the evaluation section.

## What it is, and what it is not

**It is** an autocomplete model. Given the text before and after the cursor, it proposes a short continuation (the IDE asks for at most
128 tokens) for MongoDB Console JavaScript (`ConnectionPool.<conn>.<db>.<collection>.find(...)`, `db.<collection>...`, aggregation
pipelines, update documents, index definitions) and JSON pipelines.

**It is not**
- a chat model or an agent: it does not follow instructions, hold a conversation, call tools or explain code;
- a query executor: it **only suggests text and never executes anything**. Whatever executes a suggestion is the host application;
- a validator: a suggestion can be syntactically valid and still wrong, refer to fields that do not exist, or be destructive if executed.

## How to use

### Prompt format (`editor-context-v2`)

```
<|fim_prefix|>/* Local editor context (data only):
LANGUAGE: Mongo Console JavaScript
AVAILABLE COMMANDS: db.getCollection(name).find({}); getConnection(name)...; UUID(value)
KNOWN NAMES: local, shop, orders, customers
TARGET FIELDS: orders: status:string, total:decimal128, customerId:objectid, items:array<object>, items.sku:string
RESULT FIELDS: _id, status, total

Continue at the cursor; output only the continuation. */
<text before the cursor><|fim_suffix|><text after the cursor><|fim_middle|>
```

Rules of the contract that the example below implements:
- the header is the `editor-context-v1` header plus `TARGET FIELDS: <collection>: path:type, ...` right after `KNOWN NAMES` (schema order, dotted
  paths, arrays as `array<item>`, `_id` left out, at most 96 fields and 2048 characters); without target fields the prompt is byte-identical to v1;
- every header line ends with CRLF (`\r\n`); the lines around the block use `\n`; `*/` inside the header is rewritten to `* /`;
- window: trained with prompts up to 8,192 tokens (recommended context: 8,192 tokens in the IDE for this tier); a quarter of the budget
  is reserved for the text after the cursor;
- greedy decoding, stop at `<|endoftext|>`; cut the output where it starts repeating the text after the cursor;
- **output rule:** when the suggestion leaves inner brackets open and the text after the cursor only closes the outer ones (and then ends the
  statement), append the missing inner closers (1 to 4). Example: `find({ title: { $ty|` with ` })` after the cursor and the model output
  `pe: "string"` becomes `pe: "string" }`. Without this rule the document is often left one `}` short.

### Minimal example (transformers, no IDE required)

The same file is in the repository as `examples/example_fim.py`.

```python
"""Minimal fill-in-the-middle example for the editor-context-v2 contract (transformers, CPU, no IDE required).

    python examples/example_fim.py                 # model files next to this folder (a local clone or snapshot), else the Hub repository
    python examples/example_fim.py <path-or-repo>  # explicit model directory or repository id
"""
import sys
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

HERE = Path(__file__).resolve().parent.parent
MODEL_ID = sys.argv[1] if len(sys.argv) > 1 else (str(HERE) if (HERE / "config.json").exists() else "esilva/KapiCoder-Mongo-Small")

FIM_PREFIX, FIM_SUFFIX, FIM_MIDDLE = "<|fim_prefix|>", "<|fim_suffix|>", "<|fim_middle|>"
CONSOLE_COMMANDS = (
    "db.getCollection(name).find({}); getConnection(name).getDatabase(name).getCollection(name); "
    "ConnectionPool.Connection.Database.Collection; console.log(value); ENV.get(name); "
    "ObjectId(value); UUID(value)"
)
PAIRS = {"(": ")", "[": "]", "{": "}"}


def target_fields_line(fields):
    """`path:type` pieces in schema order; arrays as `array<item>`; `_id` is left out; at most 96 fields and 2048 characters."""
    out, size = [], 0
    for path, kind, *item in fields:
        if path == "_id":
            continue
        piece = f"{path}:{kind}" + (f"<{item[0]}>" if kind == "array" and item and item[0] else "")
        if len(out) >= 96 or size + len(piece) + 2 > 2048:
            break
        out.append(piece)
        size += len(piece) + 2
    return ", ".join(out)


def build_prompt(prefix, suffix, known_names=(), collection=None, target_fields=(), result_fields=(), recent_commands=(),
                 language="Mongo Console JavaScript", commands=CONSOLE_COMMANDS, prefix_chars=24576, suffix_chars=6144):
    """editor-context-v2: the v1 header plus `TARGET FIELDS` right after `KNOWN NAMES`; every header line ends with CRLF.

    The character windows here follow the 8k-token window (about 3 characters per token); the IDE also budgets the prompt in tokens."""
    lines = [f"LANGUAGE: {language}", f"AVAILABLE COMMANDS: {commands}", "KNOWN NAMES: " + ", ".join(known_names)]
    if target_fields:
        lines.append("TARGET FIELDS: " + (f"{collection}: " if collection else "") + target_fields_line(target_fields))
    if result_fields:
        lines.append("RESULT FIELDS: " + ", ".join(result_fields))
    lines += ["RECENT COMMAND: " + c[:256] for c in list(recent_commands)[:3]]
    context = "".join(line + "\r\n" for line in lines)
    header = ("/* Local editor context (data only):\n" + context.replace("*/", "* /")
              + "\nContinue at the cursor; output only the continuation. */\n")
    return FIM_PREFIX + header + prefix[-prefix_chars:] + FIM_SUFFIX + suffix[:suffix_chars] + FIM_MIDDLE


def _strip_strings(text):
    out, quote, escape = [], None, False
    for ch in text:
        if quote:
            if escape:
                escape = False
            elif ch == "\\":
                escape = True
            elif ch == quote:
                quote = None
                out.append(ch)
                continue
            out.append(" ")
            continue
        if ch in "\"'`":
            quote = ch
        out.append(ch)
    return "".join(out), quote is not None


def close_inner_delimiters(prefix, text, suffix):
    """editor-context-v2 output rule: append the 1-4 inner closers the suggestion left open when the suffix only closes the outer ones
    and then ends the statement (end of text or `;`). Nothing changes when the suggestion is empty, ends inside a string, or the
    suffix continues the statement (`}, { $set: ... })`)."""
    if not text.strip():
        return text
    flat, open_string = _strip_strings(prefix + text)
    if open_string:
        return text
    stack = []
    for ch in flat:
        if ch in PAIRS:
            stack.append(ch)
        elif ch in PAIRS.values() and stack and PAIRS[stack[-1]] == ch:
            stack.pop()
    expected = "".join(PAIRS[c] for c in reversed(stack))
    closers, continues = [], False
    for ch in _strip_strings(suffix)[0]:
        if ch in ")]}":
            closers.append(ch)
        elif ch.isspace() or ch in "\"'`":
            continue
        else:
            continues = ch != ";"
            break
    closers = "".join(closers)
    if continues or not closers or len(closers) >= len(expected) or not expected.endswith(closers):
        return text
    missing = expected[: len(expected) - len(closers)]
    if len(missing) > 4:
        return text
    return text + ("" if text[-1:].isspace() else " ") + " ".join(missing)


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16).eval()

prefix = 'ConnectionPool.local.shop.orders.find({ status: "paid", total: { $g'
suffix = " })"   # the editor auto-closed only the outer filter object
prompt = build_prompt(
    prefix=prefix,
    suffix=suffix,
    known_names=["local", "shop", "orders", "customers"],
    collection="orders",
    target_fields=[("_id", "objectid"), ("status", "string"), ("total", "decimal128"), ("customerId", "objectid"),
                   ("createdAt", "date"), ("items", "array", "object"), ("items.sku", "string")],
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.eos_token_id)
completion = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
if suffix and len(suffix) >= 2 and suffix in completion:   # the IDE cuts an echo of the text after the cursor
    completion = completion[: completion.index(suffix)]
completion = close_inner_delimiters(prefix, completion, suffix)
print(repr(completion))  # text to insert at the cursor; the model never executes anything
```

It runs on CPU. Use `add_special_tokens=False`: the FIM markers are already in the prompt text. Always review the suggestion.

## Training

| | |
|---|---|
| Base model | `Qwen/Qwen2.5-Coder-1.5B`, revision `df3ce67c0e24480f20468b6ef2894622d69eb73b` (Apache-2.0). Training started from the base model, not from v3 |
| Method | LoRA (r=64, alpha=128, dropout 0.05 on the q, k, v, o, gate, up and down projections; 73,859,072 trainable parameters) on the bf16 base, merged into full bf16 weights afterwards. AdamW, peak LR 0.00015, 4,440 steps of 32,768 padded tokens (1.5 epochs) |
| Training data | 205,238 examples, 82,960,670 tokens per epoch; 124,537,220 tokens seen in total |
| Cursor positions | Cuts at typing points: right after `{`, `,`, `:` or `$`, inside a string, in the middle of a field name or operator, after a dotted path, and with the closers the editor auto-inserts already in the text after the cursor |
| Labels | 185,012 short cuts, 102,253 of them (55.3%) with targets written by a 7B teacher (`KapiTeacher-Mongo-7B`, a QLoRA fine-tune of Qwen2.5-Coder-7B) and kept only after a validator check (syntax, API, fields, a structural check of the completed document, read-only execution); the rest keep the generator's reference |
| LLM-written queries | 44,104 of the short cuts (23.8%) come from queries written by Qwen3-14B and validated by execution; compared with v4, more cuts have no context header or an unrelated one |
| Long context | 4,076 examples of 3k to 8k tokens (29.2% of the tokens): related definitions far from the cursor, and the full collection schema in the header with the needed field only there |
| Replay | 12.0% of the tokens (16,150 examples) are permissive JavaScript/TypeScript from npm packages (MIT, Apache-2.0, BSD, ISC), to limit forgetting of generic code |
| Data origin | Synthetic: fictitious schemas and queries generated from templates, a grammar and an LLM. No personal data by construction. No human review of the training samples |
| Hardware | One AMD Radeon RX 7800 XT (16 GB), PyTorch ROCm, transformers 4.57.6 |
| Training time | 20.50 hours of training time |
| Runs | Two training runs with the same data and settings: seed 20261104 (these weights) (the training run was started from a working tree with uncommitted changes) and seed 20261115 (not published) |
| Output | Full model in bf16, safetensors |

The training set is not published.

## Evaluation

**Internal evaluation on synthetic data; no blind human-written set.** Greedy decoding, bf16 weights on the RX 7800 XT, a single run.

### Cursor inside a call (cursor-in-call suite v1, `editor-context-v2`)

1,200 synthetic cases on held-out domains and an unseen schema: cursor right after `(`, `{`, `,`, `:`, `$` or `.`, inside a string, in the middle
of a field name or operator, with empty, auto-closed or longer text after the cursor. Output limit 64 tokens, with the v2 output rule above.
- *Suffix-consistent*: the document parses after inserting the suggestion (no duplicated or missing closer).
- *Operator valid*: the document parses and every operator is legal where it is (`$set` only in an update document, `$gt` only on a field...).
- *Field-grounded*: every field the suggestion cites exists in the target collection.

| Model | Suffix-consistent | Operator valid | Field-grounded (all) | Field-grounded, schema in header | Field-grounded, no header |
|---|---:|---:|---:|---:|---:|
| v3 (`v0.1.0-preview`), same v2 prompt and rule | 87.8% | 85.7% | 86.1% | 96.1% | 54.5% |
| v4 (`v0.2.0-preview`), 2k window | 95.2% | 95.1% | 88.7% | 99.2% | 43.6% |
| **KapiCoder-Mongo-Small** (this model, v4b), 2k window | 94.9% | 94.8% | 90.5% | 99.2% | 50.0% |
| **KapiCoder-Mongo-Small** (this model, v4b), 8k window | 94.9% | 94.8% | 91.1% | | |

| Cursor position (suffix-consistent) | v3 | v4b |
|---|---:|---:|
| inside a string | 41.3% | 100.0% |
| right after an opening bracket | 99.3% | 99.3% |
| in the middle of an operator (`$g|`) | 83.3% | 84.0% |
| right after `$` | 81.3% | 79.3% |

Long context (8k window): with the target field defined about 2k tokens before the cursor, 97.3% of the cited fields exist
(n=37); about 6k tokens before, 97.1% (n=34).

### Previous benchmark (validation split, `editor-context-v1`, 2k context)

2,000 examples of the validation split of the previous synthetic dataset, v1 prompt (no `TARGET FIELDS`), up to 96 new tokens. v4b was not trained on
this prompt format alone, so this table measures how it behaves with an editor that does not send the target fields.

| Model | FIM acc | Exec-valid | Exec ÷ ceiling | Syntax-valid | Nonexistent fields `full` / `irrelevant` / `none` (%) | Generic-JS FIM acc |
|---|---:|---:|---:|---:|---|---:|
| Qwen2.5-Coder-1.5B, zero-shot base | 0.142 | 0.316 | 0.5052 | 0.667 | 3.76 / 10.61 / 12.26 | 0.406 |
| v3 (`v0.1.0-preview`) | 0.410 | 0.602 | 0.9616 | 0.998 | 1.21 / 5.56 / 7.10 | 0.606 |
| v4 (`v0.2.0-preview`) | 0.406 | 0.575 | 0.9185 | 0.990 | 2.00 / 12.63 / 10.97 | 0.628 |
| **KapiCoder-Mongo-Small** (this model, v4b) | 0.399 | 0.582 | 0.9297 | 0.991 | 1.82 / 9.60 / 12.26 | 0.626 |

Validator ceiling for exec-valid: 0.6255. Generic-JS: 500 cuts of permissive npm JavaScript (regression check).

### Out-of-distribution homologation (homologation v1, `editor-context-v1`)

**Internal evaluation, synthetic out-of-distribution homologation. The queries were not written by people.** 6,030 fill-in-the-middle cuts from
28 schemas never used in training, written by Qwen3-14B and a grammar; not fully independent of the training data (same generator and grammar templates
over other schemas). Same v1 prompt, greedy, up to 96 new tokens; exec-valid runs the completed text in mongosh behind a read-only proxy.

| Model | FIM acc (%) | Exec-valid (%) | Exec ÷ ceiling (%) | Syntax-valid (%) | Nonexistent fields `full` / `none` (%) |
|---|---:|---:|---:|---:|---|
| Qwen2.5-Coder-1.5B, zero-shot base | 18.74 | 49.15 | 52.81 | 67.74 | 7.15 / 10.04 |
| v3 (`v0.1.0-preview`) | 45.37 | 83.07 | 89.26 | 99.72 | 7.50 / 8.74 |
| v4 (`v0.2.0-preview`) | 35.94 | 78.51 | 84.36 | 98.54 | 7.58 / 12.12 |
| **KapiCoder-Mongo-Small** (this model, v4b) | 44.61 | 84.26 | 90.53 | 99.62 | 7.34 / 7.01 |

Paired v4b − v3 on the same 6,030 items (95% CI): FIM accuracy -0.8 pp [-1.4, -0.1]; exec-valid +1.2 pp [+0.6, +1.8].
Paired v4b − v4: FIM accuracy +8.7 pp [+7.8, +9.5]; exec-valid +5.8 pp [+4.9, +6.6].
Validator ceiling for exec-valid: 93.07%.

### Second training run (seed 20261115, not published)

Same data, settings and evaluation; only the seed changes. Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not.

| Check | These weights (seed 20261104) | Second seed |
|---|---:|---:|
| Homologation FIM accuracy | 44.61% | 44.73% |
| Cursor-in-call suffix-consistent (2k, v2) | 94.9% | 94.4% |
| Cursor-in-call operator valid (2k, v2) | 94.8% | 94.1% |
| Field-grounded with no header (n=135) | 50.4% | 43.7% |
| Homologation nonexistent fields, no or unrelated schema | 5.71% | 6.06% |

Limits of this evaluation: the authors generated the data; exec-valid means "runs", not "does what was intended"; a single greedy evaluation run per model and seed.

## Limitations and risks

- **Synthetic data only.** Schemas are fictitious; expect lower scores on real collections, naming styles and query shapes.
- **Needs the target fields.** With no header, fewer of the cited fields exist (50.0% on the cursor-in-call suite, v3:
  54.5%); validate field names before running a suggestion.
- **Operators.** About 5% of suggestions on the cursor-in-call suite still leave an operator where it is not legal or a
  document that does not parse; most remaining failures are in the middle of an operator (`$g|`) or right after `$`.
- **Long context.** Trained up to 8k tokens; the farther the needed definition is from the cursor, the more it misses (see the long-context line above).
- **Two seeds only.** Field-grounding without a header varies by several points between the seeds (table above); small differences against v3 or v4
  are not established.
- **Dialects.** Mongo Console JavaScript (`ConnectionPool...` and `db.<collection>` styles) and JSON pipelines only; chat is out of scope.
- **No safety guarantee.** Destructive operations were neither trained nor tested for; during evaluation only read commands were executed. The host
  application must keep a human in the loop and must not run suggestions blindly.

## Formats

| Format | Where | Notes |
|---|---|---|
| safetensors, bf16 | this repository | Reference weights, loaded with `transformers`. |
| ONNX (onnxruntime-genai) | [`esilva/KapiCoder-Mongo-Small-ONNX`](https://huggingface.co/esilva/KapiCoder-Mongo-Small-ONNX) | CPU builds for the Kapibara Studio editor, tag `v0.3.0-preview`. No DirectML build in this release. |

No GGUF is provided. Neither the teacher model nor the training data are part of this repository.

## License, modification notice and attribution

- License: **Apache License 2.0** (see `LICENSE`).
- **Modification notice:** this model is a **modified** version of `Qwen/Qwen2.5-Coder-1.5B` (revision above). Changes: LoRA fine-tuning merged into the weights for MongoDB fill-in-the-middle autocomplete, with part of the labels distilled from a 7B teacher; export to safetensors bf16. It is not a product of Alibaba Cloud or the Qwen team and is not endorsed by them. See `NOTICE.md`.
- Attribution: base model **Qwen2.5-Coder** by the Qwen team (Hui et al., *Qwen2.5-Coder Technical Report*, arXiv:2409.12186), Apache-2.0. The teacher was derived from Qwen2.5-Coder-7B (Apache-2.0); its weights are not part of this repository. Part of the training queries was written by Qwen3-14B (Apache-2.0).
- The generic-code replay used npm packages under MIT, Apache-2.0, BSD-2/3-Clause and ISC licenses. The code itself is not redistributed; the package names, versions and licenses are listed in `REPLAY_ATTRIBUTION.md`.

Related model: [KapiCoder-Mongo-Nano](https://huggingface.co/esilva/KapiCoder-Mongo-Nano) (0.5B), same task and contract.

---

## Resumo em português (pt-BR)

**Status: PREVIEW / experimental, preview público.** KapiCoder-Mongo-Small é um modelo pequeno (cerca de 1,5 B de parâmetros) ajustado a partir do `Qwen/Qwen2.5-Coder-1.5B`
para **completar consultas MongoDB no cursor**, inclusive com o cursor **dentro** de `find`, `update` e `aggregate`, no editor Kapibara Studio. Esta é a
versão **v4b** dos pesos (tag `v0.3.0-preview`), que substitui a v4 (mesmo contrato, tag `v0.2.0-preview`); a v3 continua na tag `v0.1.0-preview`.

- **Contrato:** `editor-context-v2` (o cabeçalho v1 mais a linha `TARGET FIELDS` com os campos da coleção alvo) e a regra de saída que fecha o objeto
  interno deixado aberto. Versões do editor sem `editor-context-v2` recusam este pacote; nelas, use a tag `v0.1.0-preview`.
- **Avaliação:** **avaliação interna sobre dados sintéticos; sem conjunto cego humano.** Na suíte cursor-in-call o documento continua válido em
  94,9% dos casos (v3: 87,8%) e, com o schema no cabeçalho, 99,2% dos campos citados existem.
  Com o prompt antigo (contrato v1, sem `TARGET FIELDS`), na homologação, FIM acc 44,61% contra
  45,37% da v3 e exec-valid 84,26% contra 83,07%. Sem cabeçalho,
  50,0% dos campos citados existem (v3: 54,5%).
- **Treino:** 205.238 exemplos sintéticos, 124.537.220 tokens vistos, 20,50 horas em uma RX 7800 XT.
  **Duas execuções de treino (duas sementes); estes são os pesos da primeira.** Entre as duas sementes, as checagens de cursor contra a v4 mudaram de resultado: com a segunda semente, suffix-consistent e operador válido ficam cerca de 1 ponto abaixo da v4 (além da margem de 1 ponto); com a semente publicada, não. Sem dados pessoais por construção; não houve revisão humana das amostras.
- **Limites:** só dado sintético; mande os campos da coleção no cabeçalho e valide os nomes antes de executar; o modelo só sugere texto e nunca executa.
- **Licença:** Apache-2.0, com **aviso de modificação**: derivado e modificado do Qwen2.5-Coder-1.5B (equipe Qwen, Alibaba Cloud), sem endosso deles.