Text Generation
Transformers
Safetensors
English
Portuguese
qwen2
mongodb
code
fim
autocomplete
synthetic-data
conversational
Eval Results (legacy)
text-generation-inference
Instructions to use esilva/KapiCoder-Mongo-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use esilva/KapiCoder-Mongo-Small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="esilva/KapiCoder-Mongo-Small") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("esilva/KapiCoder-Mongo-Small") model = AutoModelForCausalLM.from_pretrained("esilva/KapiCoder-Mongo-Small", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use esilva/KapiCoder-Mongo-Small with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "esilva/KapiCoder-Mongo-Small" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "esilva/KapiCoder-Mongo-Small", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/esilva/KapiCoder-Mongo-Small
- SGLang
How to use esilva/KapiCoder-Mongo-Small with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "esilva/KapiCoder-Mongo-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "esilva/KapiCoder-Mongo-Small", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "esilva/KapiCoder-Mongo-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "esilva/KapiCoder-Mongo-Small", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use esilva/KapiCoder-Mongo-Small with Docker Model Runner:
docker model run hf.co/esilva/KapiCoder-Mongo-Small
|
Download README.md from esilva/KapiCoder-Mongo-Small: direct link, hf CLI and curl.
- Browser
- Download file 25.8 kB
-
https://huggingface.co/esilva/KapiCoder-Mongo-Small/resolve/main/README.md
- Command line
-
hf download hf://esilva/KapiCoder-Mongo-Small/README.md
-
curl -L -o README.md https://huggingface.co/esilva/KapiCoder-Mongo-Small/resolve/main/README.md
25.8 kB
| license: apache-2.0 | |
| base_model: Qwen/Qwen2.5-Coder-1.5B | |
| base_model_relation: finetune | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| - pt | |
| tags: | |
| - mongodb | |
| - code | |
| - fim | |
| - autocomplete | |
| - qwen2 | |
| - synthetic-data | |
| model-index: | |
| - name: KapiCoder-Mongo-Small | |
| results: | |
| - task: | |
| type: text-generation | |
| name: MongoDB autocomplete with the cursor inside a call (editor-context-v2) | |
| dataset: | |
| name: Internal evaluation on synthetic data, cursor-in-call suite v1 (n=1200, held-out schemas) | |
| type: kapicoder-cursor-in-call-v1-synthetic | |
| metrics: | |
| - type: suffix_consistent | |
| name: Document parses after inserting the suggestion (64 tokens, v2 post-processing) | |
| value: 0.9492 | |
| verified: false | |
| - type: op_valid_in_position | |
| name: Every operator is legal where it is (64 tokens, v2 post-processing) | |
| value: 0.9483 | |
| verified: false | |
| - type: field_grounded_full_context | |
| name: Cited fields exist in the target collection, schema in the header (context_kind=full) | |
| value: 0.9917 | |
| verified: false | |
| - task: | |
| type: text-generation | |
| name: MongoDB fill-in-the-middle autocomplete (previous editor-context-v1 benchmark) | |
| dataset: | |
| name: KapiCoder v2c validation split, internal and synthetic (n=2000) | |
| type: kapicoder-v2c-val-synthetic | |
| metrics: | |
| - type: fim_accuracy | |
| name: FIM accuracy (normalized exact match) | |
| value: 0.399 | |
| verified: false | |
| - type: exec_valid | |
| name: Exec-valid (completion parses and runs read-only) | |
| value: 0.5815 | |
| verified: false | |
| - task: | |
| type: text-generation | |
| name: MongoDB fill-in-the-middle autocomplete on unseen schemas (editor-context-v1) | |
| dataset: | |
| name: Internal evaluation, synthetic out-of-distribution homologation v1 (n=6030, LLM- and grammar-written; not fully independent of the training data) | |
| type: kapicoder-homolog-v1-synthetic-ood | |
| metrics: | |
| - type: fim_accuracy | |
| name: FIM accuracy (normalized exact match), homologation | |
| value: 0.4461 | |
| verified: false | |
| - type: exec_valid | |
| name: Exec-valid (completion parses and runs read-only), homologation | |
| value: 0.8426 | |
| verified: false | |
| # KapiCoder-Mongo-Small (1.5B) | |
| > **Status: PREVIEW / experimental, public preview.** Published for inspection and feedback, not for production use. | |
| > **Internal evaluation on synthetic data; no blind human-written set.** Read [Limitations and risks](#limitations-and-risks) before using it. | |
| KapiCoder-Mongo-Small is a small code-completion model (about 1.5 B parameters) fine-tuned from | |
| [`Qwen/Qwen2.5-Coder-1.5B`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B) to **complete MongoDB queries at the cursor** (fill-in-the-middle, FIM), | |
| including with the cursor **inside** a `find`, `update` or `aggregate` call. It was built for the *Kapibara Studio* editor, which sends it the text | |
| before and after the cursor plus a context header (the `editor-context-v2` contract described below). This is version **v4b** of the weights | |
| (repository tag `v0.3.0-preview`). It replaces v4 (same contract), which stays available at tag [`v0.2.0-preview`](https://huggingface.co/esilva/KapiCoder-Mongo-Small/tree/v0.2.0-preview); | |
| v3 (`editor-context-v1`) stays available at tag [`v0.1.0-preview`](https://huggingface.co/esilva/KapiCoder-Mongo-Small/tree/v0.1.0-preview). | |
| **Read this first** | |
| - **v4b expects the `editor-context-v2` prompt** (the v1 header plus a `TARGET FIELDS` line with the target collection's fields and types) and its | |
| output rule (close the inner object the suggestion left open). It was trained with an 8k-token window. Versions of the Kapibara Studio editor without | |
| `editor-context-v2` reject this package; use tag `v0.1.0-preview` with them. | |
| - **Internal evaluation, synthetic data, no human-written queries.** Every number below comes from the authors' own evaluation on synthetic data. | |
| - **Better with the cursor inside a call; close to v3 with the old prompt.** On the cursor-in-call suite (v2 prompt) the | |
| document stays valid after the suggestion 94.9% of the time (v3: 87.8%). With the previous | |
| `editor-context-v1` prompt (no `TARGET FIELDS`), on the out-of-distribution homologation, FIM accuracy is 44.61% against | |
| 45.37% for v3 and exec-valid 84.26% against 83.07% (paired differences below). | |
| - **Send the schema.** With the target fields in the header, 99.2% of the fields it cites exist; with no header at all, | |
| 50.0% (v3: 54.5%). Validate field names before running a suggestion. | |
| - **Two training runs (seeds); these are the weights of the first.** Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not. Second-seed values are in the evaluation section. | |
| ## What it is, and what it is not | |
| **It is** an autocomplete model. Given the text before and after the cursor, it proposes a short continuation (the IDE asks for at most | |
| 128 tokens) for MongoDB Console JavaScript (`ConnectionPool.<conn>.<db>.<collection>.find(...)`, `db.<collection>...`, aggregation | |
| pipelines, update documents, index definitions) and JSON pipelines. | |
| **It is not** | |
| - a chat model or an agent: it does not follow instructions, hold a conversation, call tools or explain code; | |
| - a query executor: it **only suggests text and never executes anything**. Whatever executes a suggestion is the host application; | |
| - a validator: a suggestion can be syntactically valid and still wrong, refer to fields that do not exist, or be destructive if executed. | |
| ## How to use | |
| ### Prompt format (`editor-context-v2`) | |
| ``` | |
| <|fim_prefix|>/* Local editor context (data only): | |
| LANGUAGE: Mongo Console JavaScript | |
| AVAILABLE COMMANDS: db.getCollection(name).find({}); getConnection(name)...; UUID(value) | |
| KNOWN NAMES: local, shop, orders, customers | |
| TARGET FIELDS: orders: status:string, total:decimal128, customerId:objectid, items:array<object>, items.sku:string | |
| RESULT FIELDS: _id, status, total | |
| Continue at the cursor; output only the continuation. */ | |
| <text before the cursor><|fim_suffix|><text after the cursor><|fim_middle|> | |
| ``` | |
| Rules of the contract that the example below implements: | |
| - the header is the `editor-context-v1` header plus `TARGET FIELDS: <collection>: path:type, ...` right after `KNOWN NAMES` (schema order, dotted | |
| paths, arrays as `array<item>`, `_id` left out, at most 96 fields and 2048 characters); without target fields the prompt is byte-identical to v1; | |
| - every header line ends with CRLF (`\r\n`); the lines around the block use `\n`; `*/` inside the header is rewritten to `* /`; | |
| - window: trained with prompts up to 8,192 tokens (recommended context: 8,192 tokens in the IDE for this tier); a quarter of the budget | |
| is reserved for the text after the cursor; | |
| - greedy decoding, stop at `<|endoftext|>`; cut the output where it starts repeating the text after the cursor; | |
| - **output rule:** when the suggestion leaves inner brackets open and the text after the cursor only closes the outer ones (and then ends the | |
| statement), append the missing inner closers (1 to 4). Example: `find({ title: { $ty|` with ` })` after the cursor and the model output | |
| `pe: "string"` becomes `pe: "string" }`. Without this rule the document is often left one `}` short. | |
| ### Minimal example (transformers, no IDE required) | |
| The same file is in the repository as `examples/example_fim.py`. | |
| ```python | |
| """Minimal fill-in-the-middle example for the editor-context-v2 contract (transformers, CPU, no IDE required). | |
| python examples/example_fim.py # model files next to this folder (a local clone or snapshot), else the Hub repository | |
| python examples/example_fim.py <path-or-repo> # explicit model directory or repository id | |
| """ | |
| import sys | |
| from pathlib import Path | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| HERE = Path(__file__).resolve().parent.parent | |
| MODEL_ID = sys.argv[1] if len(sys.argv) > 1 else (str(HERE) if (HERE / "config.json").exists() else "esilva/KapiCoder-Mongo-Small") | |
| FIM_PREFIX, FIM_SUFFIX, FIM_MIDDLE = "<|fim_prefix|>", "<|fim_suffix|>", "<|fim_middle|>" | |
| CONSOLE_COMMANDS = ( | |
| "db.getCollection(name).find({}); getConnection(name).getDatabase(name).getCollection(name); " | |
| "ConnectionPool.Connection.Database.Collection; console.log(value); ENV.get(name); " | |
| "ObjectId(value); UUID(value)" | |
| ) | |
| PAIRS = {"(": ")", "[": "]", "{": "}"} | |
| def target_fields_line(fields): | |
| """`path:type` pieces in schema order; arrays as `array<item>`; `_id` is left out; at most 96 fields and 2048 characters.""" | |
| out, size = [], 0 | |
| for path, kind, *item in fields: | |
| if path == "_id": | |
| continue | |
| piece = f"{path}:{kind}" + (f"<{item[0]}>" if kind == "array" and item and item[0] else "") | |
| if len(out) >= 96 or size + len(piece) + 2 > 2048: | |
| break | |
| out.append(piece) | |
| size += len(piece) + 2 | |
| return ", ".join(out) | |
| def build_prompt(prefix, suffix, known_names=(), collection=None, target_fields=(), result_fields=(), recent_commands=(), | |
| language="Mongo Console JavaScript", commands=CONSOLE_COMMANDS, prefix_chars=24576, suffix_chars=6144): | |
| """editor-context-v2: the v1 header plus `TARGET FIELDS` right after `KNOWN NAMES`; every header line ends with CRLF. | |
| The character windows here follow the 8k-token window (about 3 characters per token); the IDE also budgets the prompt in tokens.""" | |
| lines = [f"LANGUAGE: {language}", f"AVAILABLE COMMANDS: {commands}", "KNOWN NAMES: " + ", ".join(known_names)] | |
| if target_fields: | |
| lines.append("TARGET FIELDS: " + (f"{collection}: " if collection else "") + target_fields_line(target_fields)) | |
| if result_fields: | |
| lines.append("RESULT FIELDS: " + ", ".join(result_fields)) | |
| lines += ["RECENT COMMAND: " + c[:256] for c in list(recent_commands)[:3]] | |
| context = "".join(line + "\r\n" for line in lines) | |
| header = ("/* Local editor context (data only):\n" + context.replace("*/", "* /") | |
| + "\nContinue at the cursor; output only the continuation. */\n") | |
| return FIM_PREFIX + header + prefix[-prefix_chars:] + FIM_SUFFIX + suffix[:suffix_chars] + FIM_MIDDLE | |
| def _strip_strings(text): | |
| out, quote, escape = [], None, False | |
| for ch in text: | |
| if quote: | |
| if escape: | |
| escape = False | |
| elif ch == "\\": | |
| escape = True | |
| elif ch == quote: | |
| quote = None | |
| out.append(ch) | |
| continue | |
| out.append(" ") | |
| continue | |
| if ch in "\"'`": | |
| quote = ch | |
| out.append(ch) | |
| return "".join(out), quote is not None | |
| def close_inner_delimiters(prefix, text, suffix): | |
| """editor-context-v2 output rule: append the 1-4 inner closers the suggestion left open when the suffix only closes the outer ones | |
| and then ends the statement (end of text or `;`). Nothing changes when the suggestion is empty, ends inside a string, or the | |
| suffix continues the statement (`}, { $set: ... })`).""" | |
| if not text.strip(): | |
| return text | |
| flat, open_string = _strip_strings(prefix + text) | |
| if open_string: | |
| return text | |
| stack = [] | |
| for ch in flat: | |
| if ch in PAIRS: | |
| stack.append(ch) | |
| elif ch in PAIRS.values() and stack and PAIRS[stack[-1]] == ch: | |
| stack.pop() | |
| expected = "".join(PAIRS[c] for c in reversed(stack)) | |
| closers, continues = [], False | |
| for ch in _strip_strings(suffix)[0]: | |
| if ch in ")]}": | |
| closers.append(ch) | |
| elif ch.isspace() or ch in "\"'`": | |
| continue | |
| else: | |
| continues = ch != ";" | |
| break | |
| closers = "".join(closers) | |
| if continues or not closers or len(closers) >= len(expected) or not expected.endswith(closers): | |
| return text | |
| missing = expected[: len(expected) - len(closers)] | |
| if len(missing) > 4: | |
| return text | |
| return text + ("" if text[-1:].isspace() else " ") + " ".join(missing) | |
| tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) | |
| model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16).eval() | |
| prefix = 'ConnectionPool.local.shop.orders.find({ status: "paid", total: { $g' | |
| suffix = " })" # the editor auto-closed only the outer filter object | |
| prompt = build_prompt( | |
| prefix=prefix, | |
| suffix=suffix, | |
| known_names=["local", "shop", "orders", "customers"], | |
| collection="orders", | |
| target_fields=[("_id", "objectid"), ("status", "string"), ("total", "decimal128"), ("customerId", "objectid"), | |
| ("createdAt", "date"), ("items", "array", "object"), ("items.sku", "string")], | |
| ) | |
| inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False) | |
| with torch.no_grad(): | |
| out = model.generate(**inputs, max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.eos_token_id) | |
| completion = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) | |
| if suffix and len(suffix) >= 2 and suffix in completion: # the IDE cuts an echo of the text after the cursor | |
| completion = completion[: completion.index(suffix)] | |
| completion = close_inner_delimiters(prefix, completion, suffix) | |
| print(repr(completion)) # text to insert at the cursor; the model never executes anything | |
| ``` | |
| It runs on CPU. Use `add_special_tokens=False`: the FIM markers are already in the prompt text. Always review the suggestion. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | Base model | `Qwen/Qwen2.5-Coder-1.5B`, revision `df3ce67c0e24480f20468b6ef2894622d69eb73b` (Apache-2.0). Training started from the base model, not from v3 | | |
| | Method | LoRA (r=64, alpha=128, dropout 0.05 on the q, k, v, o, gate, up and down projections; 73,859,072 trainable parameters) on the bf16 base, merged into full bf16 weights afterwards. AdamW, peak LR 0.00015, 4,440 steps of 32,768 padded tokens (1.5 epochs) | | |
| | Training data | 205,238 examples, 82,960,670 tokens per epoch; 124,537,220 tokens seen in total | | |
| | Cursor positions | Cuts at typing points: right after `{`, `,`, `:` or `$`, inside a string, in the middle of a field name or operator, after a dotted path, and with the closers the editor auto-inserts already in the text after the cursor | | |
| | Labels | 185,012 short cuts, 102,253 of them (55.3%) with targets written by a 7B teacher (`KapiTeacher-Mongo-7B`, a QLoRA fine-tune of Qwen2.5-Coder-7B) and kept only after a validator check (syntax, API, fields, a structural check of the completed document, read-only execution); the rest keep the generator's reference | | |
| | LLM-written queries | 44,104 of the short cuts (23.8%) come from queries written by Qwen3-14B and validated by execution; compared with v4, more cuts have no context header or an unrelated one | | |
| | Long context | 4,076 examples of 3k to 8k tokens (29.2% of the tokens): related definitions far from the cursor, and the full collection schema in the header with the needed field only there | | |
| | Replay | 12.0% of the tokens (16,150 examples) are permissive JavaScript/TypeScript from npm packages (MIT, Apache-2.0, BSD, ISC), to limit forgetting of generic code | | |
| | Data origin | Synthetic: fictitious schemas and queries generated from templates, a grammar and an LLM. No personal data by construction. No human review of the training samples | | |
| | Hardware | One AMD Radeon RX 7800 XT (16 GB), PyTorch ROCm, transformers 4.57.6 | | |
| | Training time | 20.50 hours of training time | | |
| | Runs | Two training runs with the same data and settings: seed 20261104 (these weights) (the training run was started from a working tree with uncommitted changes) and seed 20261115 (not published) | | |
| | Output | Full model in bf16, safetensors | | |
| The training set is not published. | |
| ## Evaluation | |
| **Internal evaluation on synthetic data; no blind human-written set.** Greedy decoding, bf16 weights on the RX 7800 XT, a single run. | |
| ### Cursor inside a call (cursor-in-call suite v1, `editor-context-v2`) | |
| 1,200 synthetic cases on held-out domains and an unseen schema: cursor right after `(`, `{`, `,`, `:`, `$` or `.`, inside a string, in the middle | |
| of a field name or operator, with empty, auto-closed or longer text after the cursor. Output limit 64 tokens, with the v2 output rule above. | |
| - *Suffix-consistent*: the document parses after inserting the suggestion (no duplicated or missing closer). | |
| - *Operator valid*: the document parses and every operator is legal where it is (`$set` only in an update document, `$gt` only on a field...). | |
| - *Field-grounded*: every field the suggestion cites exists in the target collection. | |
| | Model | Suffix-consistent | Operator valid | Field-grounded (all) | Field-grounded, schema in header | Field-grounded, no header | | |
| |---|---:|---:|---:|---:|---:| | |
| | v3 (`v0.1.0-preview`), same v2 prompt and rule | 87.8% | 85.7% | 86.1% | 96.1% | 54.5% | | |
| | v4 (`v0.2.0-preview`), 2k window | 95.2% | 95.1% | 88.7% | 99.2% | 43.6% | | |
| | **KapiCoder-Mongo-Small** (this model, v4b), 2k window | 94.9% | 94.8% | 90.5% | 99.2% | 50.0% | | |
| | **KapiCoder-Mongo-Small** (this model, v4b), 8k window | 94.9% | 94.8% | 91.1% | | | | |
| | Cursor position (suffix-consistent) | v3 | v4b | | |
| |---|---:|---:| | |
| | inside a string | 41.3% | 100.0% | | |
| | right after an opening bracket | 99.3% | 99.3% | | |
| | in the middle of an operator (`$g|`) | 83.3% | 84.0% | | |
| | right after `$` | 81.3% | 79.3% | | |
| Long context (8k window): with the target field defined about 2k tokens before the cursor, 97.3% of the cited fields exist | |
| (n=37); about 6k tokens before, 97.1% (n=34). | |
| ### Previous benchmark (validation split, `editor-context-v1`, 2k context) | |
| 2,000 examples of the validation split of the previous synthetic dataset, v1 prompt (no `TARGET FIELDS`), up to 96 new tokens. v4b was not trained on | |
| this prompt format alone, so this table measures how it behaves with an editor that does not send the target fields. | |
| | Model | FIM acc | Exec-valid | Exec ÷ ceiling | Syntax-valid | Nonexistent fields `full` / `irrelevant` / `none` (%) | Generic-JS FIM acc | | |
| |---|---:|---:|---:|---:|---|---:| | |
| | Qwen2.5-Coder-1.5B, zero-shot base | 0.142 | 0.316 | 0.5052 | 0.667 | 3.76 / 10.61 / 12.26 | 0.406 | | |
| | v3 (`v0.1.0-preview`) | 0.410 | 0.602 | 0.9616 | 0.998 | 1.21 / 5.56 / 7.10 | 0.606 | | |
| | v4 (`v0.2.0-preview`) | 0.406 | 0.575 | 0.9185 | 0.990 | 2.00 / 12.63 / 10.97 | 0.628 | | |
| | **KapiCoder-Mongo-Small** (this model, v4b) | 0.399 | 0.582 | 0.9297 | 0.991 | 1.82 / 9.60 / 12.26 | 0.626 | | |
| Validator ceiling for exec-valid: 0.6255. Generic-JS: 500 cuts of permissive npm JavaScript (regression check). | |
| ### Out-of-distribution homologation (homologation v1, `editor-context-v1`) | |
| **Internal evaluation, synthetic out-of-distribution homologation. The queries were not written by people.** 6,030 fill-in-the-middle cuts from | |
| 28 schemas never used in training, written by Qwen3-14B and a grammar; not fully independent of the training data (same generator and grammar templates | |
| over other schemas). Same v1 prompt, greedy, up to 96 new tokens; exec-valid runs the completed text in mongosh behind a read-only proxy. | |
| | Model | FIM acc (%) | Exec-valid (%) | Exec ÷ ceiling (%) | Syntax-valid (%) | Nonexistent fields `full` / `none` (%) | | |
| |---|---:|---:|---:|---:|---| | |
| | Qwen2.5-Coder-1.5B, zero-shot base | 18.74 | 49.15 | 52.81 | 67.74 | 7.15 / 10.04 | | |
| | v3 (`v0.1.0-preview`) | 45.37 | 83.07 | 89.26 | 99.72 | 7.50 / 8.74 | | |
| | v4 (`v0.2.0-preview`) | 35.94 | 78.51 | 84.36 | 98.54 | 7.58 / 12.12 | | |
| | **KapiCoder-Mongo-Small** (this model, v4b) | 44.61 | 84.26 | 90.53 | 99.62 | 7.34 / 7.01 | | |
| Paired v4b − v3 on the same 6,030 items (95% CI): FIM accuracy -0.8 pp [-1.4, -0.1]; exec-valid +1.2 pp [+0.6, +1.8]. | |
| Paired v4b − v4: FIM accuracy +8.7 pp [+7.8, +9.5]; exec-valid +5.8 pp [+4.9, +6.6]. | |
| Validator ceiling for exec-valid: 93.07%. | |
| ### Second training run (seed 20261115, not published) | |
| Same data, settings and evaluation; only the seed changes. Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not. | |
| | Check | These weights (seed 20261104) | Second seed | | |
| |---|---:|---:| | |
| | Homologation FIM accuracy | 44.61% | 44.73% | | |
| | Cursor-in-call suffix-consistent (2k, v2) | 94.9% | 94.4% | | |
| | Cursor-in-call operator valid (2k, v2) | 94.8% | 94.1% | | |
| | Field-grounded with no header (n=135) | 50.4% | 43.7% | | |
| | Homologation nonexistent fields, no or unrelated schema | 5.71% | 6.06% | | |
| Limits of this evaluation: the authors generated the data; exec-valid means "runs", not "does what was intended"; a single greedy evaluation run per model and seed. | |
| ## Limitations and risks | |
| - **Synthetic data only.** Schemas are fictitious; expect lower scores on real collections, naming styles and query shapes. | |
| - **Needs the target fields.** With no header, fewer of the cited fields exist (50.0% on the cursor-in-call suite, v3: | |
| 54.5%); validate field names before running a suggestion. | |
| - **Operators.** About 5% of suggestions on the cursor-in-call suite still leave an operator where it is not legal or a | |
| document that does not parse; most remaining failures are in the middle of an operator (`$g|`) or right after `$`. | |
| - **Long context.** Trained up to 8k tokens; the farther the needed definition is from the cursor, the more it misses (see the long-context line above). | |
| - **Two seeds only.** Field-grounding without a header varies by several points between the seeds (table above); small differences against v3 or v4 | |
| are not established. | |
| - **Dialects.** Mongo Console JavaScript (`ConnectionPool...` and `db.<collection>` styles) and JSON pipelines only; chat is out of scope. | |
| - **No safety guarantee.** Destructive operations were neither trained nor tested for; during evaluation only read commands were executed. The host | |
| application must keep a human in the loop and must not run suggestions blindly. | |
| ## Formats | |
| | Format | Where | Notes | | |
| |---|---|---| | |
| | safetensors, bf16 | this repository | Reference weights, loaded with `transformers`. | | |
| | ONNX (onnxruntime-genai) | [`esilva/KapiCoder-Mongo-Small-ONNX`](https://huggingface.co/esilva/KapiCoder-Mongo-Small-ONNX) | CPU builds for the Kapibara Studio editor, tag `v0.3.0-preview`. No DirectML build in this release. | | |
| No GGUF is provided. Neither the teacher model nor the training data are part of this repository. | |
| ## License, modification notice and attribution | |
| - License: **Apache License 2.0** (see `LICENSE`). | |
| - **Modification notice:** this model is a **modified** version of `Qwen/Qwen2.5-Coder-1.5B` (revision above). Changes: LoRA fine-tuning merged into the weights for MongoDB fill-in-the-middle autocomplete, with part of the labels distilled from a 7B teacher; export to safetensors bf16. It is not a product of Alibaba Cloud or the Qwen team and is not endorsed by them. See `NOTICE.md`. | |
| - Attribution: base model **Qwen2.5-Coder** by the Qwen team (Hui et al., *Qwen2.5-Coder Technical Report*, arXiv:2409.12186), Apache-2.0. The teacher was derived from Qwen2.5-Coder-7B (Apache-2.0); its weights are not part of this repository. Part of the training queries was written by Qwen3-14B (Apache-2.0). | |
| - The generic-code replay used npm packages under MIT, Apache-2.0, BSD-2/3-Clause and ISC licenses. The code itself is not redistributed; the package names, versions and licenses are listed in `REPLAY_ATTRIBUTION.md`. | |
| Related model: [KapiCoder-Mongo-Nano](https://huggingface.co/esilva/KapiCoder-Mongo-Nano) (0.5B), same task and contract. | |
| --- | |
| ## Resumo em português (pt-BR) | |
| **Status: PREVIEW / experimental, preview público.** KapiCoder-Mongo-Small é um modelo pequeno (cerca de 1,5 B de parâmetros) ajustado a partir do `Qwen/Qwen2.5-Coder-1.5B` | |
| para **completar consultas MongoDB no cursor**, inclusive com o cursor **dentro** de `find`, `update` e `aggregate`, no editor Kapibara Studio. Esta é a | |
| versão **v4b** dos pesos (tag `v0.3.0-preview`), que substitui a v4 (mesmo contrato, tag `v0.2.0-preview`); a v3 continua na tag `v0.1.0-preview`. | |
| - **Contrato:** `editor-context-v2` (o cabeçalho v1 mais a linha `TARGET FIELDS` com os campos da coleção alvo) e a regra de saída que fecha o objeto | |
| interno deixado aberto. Versões do editor sem `editor-context-v2` recusam este pacote; nelas, use a tag `v0.1.0-preview`. | |
| - **Avaliação:** **avaliação interna sobre dados sintéticos; sem conjunto cego humano.** Na suíte cursor-in-call o documento continua válido em | |
| 94,9% dos casos (v3: 87,8%) e, com o schema no cabeçalho, 99,2% dos campos citados existem. | |
| Com o prompt antigo (contrato v1, sem `TARGET FIELDS`), na homologação, FIM acc 44,61% contra | |
| 45,37% da v3 e exec-valid 84,26% contra 83,07%. Sem cabeçalho, | |
| 50,0% dos campos citados existem (v3: 54,5%). | |
| - **Treino:** 205.238 exemplos sintéticos, 124.537.220 tokens vistos, 20,50 horas em uma RX 7800 XT. | |
| **Duas execuções de treino (duas sementes); estes são os pesos da primeira.** Entre as duas sementes, as checagens de cursor contra a v4 mudaram de resultado: com a segunda semente, suffix-consistent e operador válido ficam cerca de 1 ponto abaixo da v4 (além da margem de 1 ponto); com a semente publicada, não. Sem dados pessoais por construção; não houve revisão humana das amostras. | |
| - **Limites:** só dado sintético; mande os campos da coleção no cabeçalho e valide os nomes antes de executar; o modelo só sugere texto e nunca executa. | |
| - **Licença:** Apache-2.0, com **aviso de modificação**: derivado e modificado do Qwen2.5-Coder-1.5B (equipe Qwen, Alibaba Cloud), sem endosso deles. | |