esilva's picture
Replace with v4b weights (v0.3.0-preview)
2419a3b verified
|
Raw History Blame Contribute Delete
25.8 kB
metadata
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-1.5B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language:
  - en
  - pt
tags:
  - mongodb
  - code
  - fim
  - autocomplete
  - qwen2
  - synthetic-data
model-index:
  - name: KapiCoder-Mongo-Small
    results:
      - task:
          type: text-generation
          name: >-
            MongoDB autocomplete with the cursor inside a call
            (editor-context-v2)
        dataset:
          name: >-
            Internal evaluation on synthetic data, cursor-in-call suite v1
            (n=1200, held-out schemas)
          type: kapicoder-cursor-in-call-v1-synthetic
        metrics:
          - type: suffix_consistent
            name: >-
              Document parses after inserting the suggestion (64 tokens, v2
              post-processing)
            value: 0.9492
            verified: false
          - type: op_valid_in_position
            name: >-
              Every operator is legal where it is (64 tokens, v2
              post-processing)
            value: 0.9483
            verified: false
          - type: field_grounded_full_context
            name: >-
              Cited fields exist in the target collection, schema in the header
              (context_kind=full)
            value: 0.9917
            verified: false
      - task:
          type: text-generation
          name: >-
            MongoDB fill-in-the-middle autocomplete (previous editor-context-v1
            benchmark)
        dataset:
          name: KapiCoder v2c validation split, internal and synthetic (n=2000)
          type: kapicoder-v2c-val-synthetic
        metrics:
          - type: fim_accuracy
            name: FIM accuracy (normalized exact match)
            value: 0.399
            verified: false
          - type: exec_valid
            name: Exec-valid (completion parses and runs read-only)
            value: 0.5815
            verified: false
      - task:
          type: text-generation
          name: >-
            MongoDB fill-in-the-middle autocomplete on unseen schemas
            (editor-context-v1)
        dataset:
          name: >-
            Internal evaluation, synthetic out-of-distribution homologation v1
            (n=6030, LLM- and grammar-written; not fully independent of the
            training data)
          type: kapicoder-homolog-v1-synthetic-ood
        metrics:
          - type: fim_accuracy
            name: FIM accuracy (normalized exact match), homologation
            value: 0.4461
            verified: false
          - type: exec_valid
            name: Exec-valid (completion parses and runs read-only), homologation
            value: 0.8426
            verified: false

KapiCoder-Mongo-Small (1.5B)

Status: PREVIEW / experimental, public preview. Published for inspection and feedback, not for production use. Internal evaluation on synthetic data; no blind human-written set. Read Limitations and risks before using it.

KapiCoder-Mongo-Small is a small code-completion model (about 1.5 B parameters) fine-tuned from Qwen/Qwen2.5-Coder-1.5B to complete MongoDB queries at the cursor (fill-in-the-middle, FIM), including with the cursor inside a find, update or aggregate call. It was built for the Kapibara Studio editor, which sends it the text before and after the cursor plus a context header (the editor-context-v2 contract described below). This is version v4b of the weights (repository tag v0.3.0-preview). It replaces v4 (same contract), which stays available at tag v0.2.0-preview; v3 (editor-context-v1) stays available at tag v0.1.0-preview.

Read this first

  • v4b expects the editor-context-v2 prompt (the v1 header plus a TARGET FIELDS line with the target collection's fields and types) and its output rule (close the inner object the suggestion left open). It was trained with an 8k-token window. Versions of the Kapibara Studio editor without editor-context-v2 reject this package; use tag v0.1.0-preview with them.
  • Internal evaluation, synthetic data, no human-written queries. Every number below comes from the authors' own evaluation on synthetic data.
  • Better with the cursor inside a call; close to v3 with the old prompt. On the cursor-in-call suite (v2 prompt) the document stays valid after the suggestion 94.9% of the time (v3: 87.8%). With the previous editor-context-v1 prompt (no TARGET FIELDS), on the out-of-distribution homologation, FIM accuracy is 44.61% against 45.37% for v3 and exec-valid 84.26% against 83.07% (paired differences below).
  • Send the schema. With the target fields in the header, 99.2% of the fields it cites exist; with no header at all, 50.0% (v3: 54.5%). Validate field names before running a suggestion.
  • Two training runs (seeds); these are the weights of the first. Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not. Second-seed values are in the evaluation section.

What it is, and what it is not

It is an autocomplete model. Given the text before and after the cursor, it proposes a short continuation (the IDE asks for at most 128 tokens) for MongoDB Console JavaScript (ConnectionPool.<conn>.<db>.<collection>.find(...), db.<collection>..., aggregation pipelines, update documents, index definitions) and JSON pipelines.

It is not

  • a chat model or an agent: it does not follow instructions, hold a conversation, call tools or explain code;
  • a query executor: it only suggests text and never executes anything. Whatever executes a suggestion is the host application;
  • a validator: a suggestion can be syntactically valid and still wrong, refer to fields that do not exist, or be destructive if executed.

How to use

Prompt format (editor-context-v2)

<|fim_prefix|>/* Local editor context (data only):
LANGUAGE: Mongo Console JavaScript
AVAILABLE COMMANDS: db.getCollection(name).find({}); getConnection(name)...; UUID(value)
KNOWN NAMES: local, shop, orders, customers
TARGET FIELDS: orders: status:string, total:decimal128, customerId:objectid, items:array<object>, items.sku:string
RESULT FIELDS: _id, status, total

Continue at the cursor; output only the continuation. */
<text before the cursor><|fim_suffix|><text after the cursor><|fim_middle|>

Rules of the contract that the example below implements:

  • the header is the editor-context-v1 header plus TARGET FIELDS: <collection>: path:type, ... right after KNOWN NAMES (schema order, dotted paths, arrays as array<item>, _id left out, at most 96 fields and 2048 characters); without target fields the prompt is byte-identical to v1;
  • every header line ends with CRLF (\r\n); the lines around the block use \n; */ inside the header is rewritten to * /;
  • window: trained with prompts up to 8,192 tokens (recommended context: 8,192 tokens in the IDE for this tier); a quarter of the budget is reserved for the text after the cursor;
  • greedy decoding, stop at <|endoftext|>; cut the output where it starts repeating the text after the cursor;
  • output rule: when the suggestion leaves inner brackets open and the text after the cursor only closes the outer ones (and then ends the statement), append the missing inner closers (1 to 4). Example: find({ title: { $ty| with }) after the cursor and the model output pe: "string" becomes pe: "string" }. Without this rule the document is often left one } short.

Minimal example (transformers, no IDE required)

The same file is in the repository as examples/example_fim.py.

"""Minimal fill-in-the-middle example for the editor-context-v2 contract (transformers, CPU, no IDE required).

    python examples/example_fim.py                 # model files next to this folder (a local clone or snapshot), else the Hub repository
    python examples/example_fim.py <path-or-repo>  # explicit model directory or repository id
"""
import sys
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

HERE = Path(__file__).resolve().parent.parent
MODEL_ID = sys.argv[1] if len(sys.argv) > 1 else (str(HERE) if (HERE / "config.json").exists() else "esilva/KapiCoder-Mongo-Small")

FIM_PREFIX, FIM_SUFFIX, FIM_MIDDLE = "<|fim_prefix|>", "<|fim_suffix|>", "<|fim_middle|>"
CONSOLE_COMMANDS = (
    "db.getCollection(name).find({}); getConnection(name).getDatabase(name).getCollection(name); "
    "ConnectionPool.Connection.Database.Collection; console.log(value); ENV.get(name); "
    "ObjectId(value); UUID(value)"
)
PAIRS = {"(": ")", "[": "]", "{": "}"}


def target_fields_line(fields):
    """`path:type` pieces in schema order; arrays as `array<item>`; `_id` is left out; at most 96 fields and 2048 characters."""
    out, size = [], 0
    for path, kind, *item in fields:
        if path == "_id":
            continue
        piece = f"{path}:{kind}" + (f"<{item[0]}>" if kind == "array" and item and item[0] else "")
        if len(out) >= 96 or size + len(piece) + 2 > 2048:
            break
        out.append(piece)
        size += len(piece) + 2
    return ", ".join(out)


def build_prompt(prefix, suffix, known_names=(), collection=None, target_fields=(), result_fields=(), recent_commands=(),
                 language="Mongo Console JavaScript", commands=CONSOLE_COMMANDS, prefix_chars=24576, suffix_chars=6144):
    """editor-context-v2: the v1 header plus `TARGET FIELDS` right after `KNOWN NAMES`; every header line ends with CRLF.

    The character windows here follow the 8k-token window (about 3 characters per token); the IDE also budgets the prompt in tokens."""
    lines = [f"LANGUAGE: {language}", f"AVAILABLE COMMANDS: {commands}", "KNOWN NAMES: " + ", ".join(known_names)]
    if target_fields:
        lines.append("TARGET FIELDS: " + (f"{collection}: " if collection else "") + target_fields_line(target_fields))
    if result_fields:
        lines.append("RESULT FIELDS: " + ", ".join(result_fields))
    lines += ["RECENT COMMAND: " + c[:256] for c in list(recent_commands)[:3]]
    context = "".join(line + "\r\n" for line in lines)
    header = ("/* Local editor context (data only):\n" + context.replace("*/", "* /")
              + "\nContinue at the cursor; output only the continuation. */\n")
    return FIM_PREFIX + header + prefix[-prefix_chars:] + FIM_SUFFIX + suffix[:suffix_chars] + FIM_MIDDLE


def _strip_strings(text):
    out, quote, escape = [], None, False
    for ch in text:
        if quote:
            if escape:
                escape = False
            elif ch == "\\":
                escape = True
            elif ch == quote:
                quote = None
                out.append(ch)
                continue
            out.append(" ")
            continue
        if ch in "\"'`":
            quote = ch
        out.append(ch)
    return "".join(out), quote is not None


def close_inner_delimiters(prefix, text, suffix):
    """editor-context-v2 output rule: append the 1-4 inner closers the suggestion left open when the suffix only closes the outer ones
    and then ends the statement (end of text or `;`). Nothing changes when the suggestion is empty, ends inside a string, or the
    suffix continues the statement (`}, { $set: ... })`)."""
    if not text.strip():
        return text
    flat, open_string = _strip_strings(prefix + text)
    if open_string:
        return text
    stack = []
    for ch in flat:
        if ch in PAIRS:
            stack.append(ch)
        elif ch in PAIRS.values() and stack and PAIRS[stack[-1]] == ch:
            stack.pop()
    expected = "".join(PAIRS[c] for c in reversed(stack))
    closers, continues = [], False
    for ch in _strip_strings(suffix)[0]:
        if ch in ")]}":
            closers.append(ch)
        elif ch.isspace() or ch in "\"'`":
            continue
        else:
            continues = ch != ";"
            break
    closers = "".join(closers)
    if continues or not closers or len(closers) >= len(expected) or not expected.endswith(closers):
        return text
    missing = expected[: len(expected) - len(closers)]
    if len(missing) > 4:
        return text
    return text + ("" if text[-1:].isspace() else " ") + " ".join(missing)


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16).eval()

prefix = 'ConnectionPool.local.shop.orders.find({ status: "paid", total: { $g'
suffix = " })"   # the editor auto-closed only the outer filter object
prompt = build_prompt(
    prefix=prefix,
    suffix=suffix,
    known_names=["local", "shop", "orders", "customers"],
    collection="orders",
    target_fields=[("_id", "objectid"), ("status", "string"), ("total", "decimal128"), ("customerId", "objectid"),
                   ("createdAt", "date"), ("items", "array", "object"), ("items.sku", "string")],
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.eos_token_id)
completion = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
if suffix and len(suffix) >= 2 and suffix in completion:   # the IDE cuts an echo of the text after the cursor
    completion = completion[: completion.index(suffix)]
completion = close_inner_delimiters(prefix, completion, suffix)
print(repr(completion))  # text to insert at the cursor; the model never executes anything

It runs on CPU. Use add_special_tokens=False: the FIM markers are already in the prompt text. Always review the suggestion.

Training

Base model Qwen/Qwen2.5-Coder-1.5B, revision df3ce67c0e24480f20468b6ef2894622d69eb73b (Apache-2.0). Training started from the base model, not from v3
Method LoRA (r=64, alpha=128, dropout 0.05 on the q, k, v, o, gate, up and down projections; 73,859,072 trainable parameters) on the bf16 base, merged into full bf16 weights afterwards. AdamW, peak LR 0.00015, 4,440 steps of 32,768 padded tokens (1.5 epochs)
Training data 205,238 examples, 82,960,670 tokens per epoch; 124,537,220 tokens seen in total
Cursor positions Cuts at typing points: right after {, ,, : or $, inside a string, in the middle of a field name or operator, after a dotted path, and with the closers the editor auto-inserts already in the text after the cursor
Labels 185,012 short cuts, 102,253 of them (55.3%) with targets written by a 7B teacher (KapiTeacher-Mongo-7B, a QLoRA fine-tune of Qwen2.5-Coder-7B) and kept only after a validator check (syntax, API, fields, a structural check of the completed document, read-only execution); the rest keep the generator's reference
LLM-written queries 44,104 of the short cuts (23.8%) come from queries written by Qwen3-14B and validated by execution; compared with v4, more cuts have no context header or an unrelated one
Long context 4,076 examples of 3k to 8k tokens (29.2% of the tokens): related definitions far from the cursor, and the full collection schema in the header with the needed field only there
Replay 12.0% of the tokens (16,150 examples) are permissive JavaScript/TypeScript from npm packages (MIT, Apache-2.0, BSD, ISC), to limit forgetting of generic code
Data origin Synthetic: fictitious schemas and queries generated from templates, a grammar and an LLM. No personal data by construction. No human review of the training samples
Hardware One AMD Radeon RX 7800 XT (16 GB), PyTorch ROCm, transformers 4.57.6
Training time 20.50 hours of training time
Runs Two training runs with the same data and settings: seed 20261104 (these weights) (the training run was started from a working tree with uncommitted changes) and seed 20261115 (not published)
Output Full model in bf16, safetensors

The training set is not published.

Evaluation

Internal evaluation on synthetic data; no blind human-written set. Greedy decoding, bf16 weights on the RX 7800 XT, a single run.

Cursor inside a call (cursor-in-call suite v1, editor-context-v2)

1,200 synthetic cases on held-out domains and an unseen schema: cursor right after (, {, ,, :, $ or ., inside a string, in the middle of a field name or operator, with empty, auto-closed or longer text after the cursor. Output limit 64 tokens, with the v2 output rule above.

  • Suffix-consistent: the document parses after inserting the suggestion (no duplicated or missing closer).
  • Operator valid: the document parses and every operator is legal where it is ($set only in an update document, $gt only on a field...).
  • Field-grounded: every field the suggestion cites exists in the target collection.
Model Suffix-consistent Operator valid Field-grounded (all) Field-grounded, schema in header Field-grounded, no header
v3 (v0.1.0-preview), same v2 prompt and rule 87.8% 85.7% 86.1% 96.1% 54.5%
v4 (v0.2.0-preview), 2k window 95.2% 95.1% 88.7% 99.2% 43.6%
KapiCoder-Mongo-Small (this model, v4b), 2k window 94.9% 94.8% 90.5% 99.2% 50.0%
KapiCoder-Mongo-Small (this model, v4b), 8k window 94.9% 94.8% 91.1%
Cursor position (suffix-consistent) v3 v4b
inside a string 41.3% 100.0%
right after an opening bracket 99.3% 99.3%
in the middle of an operator (`$g `) 83.3%
right after $ 81.3% 79.3%

Long context (8k window): with the target field defined about 2k tokens before the cursor, 97.3% of the cited fields exist (n=37); about 6k tokens before, 97.1% (n=34).

Previous benchmark (validation split, editor-context-v1, 2k context)

2,000 examples of the validation split of the previous synthetic dataset, v1 prompt (no TARGET FIELDS), up to 96 new tokens. v4b was not trained on this prompt format alone, so this table measures how it behaves with an editor that does not send the target fields.

Model FIM acc Exec-valid Exec ÷ ceiling Syntax-valid Nonexistent fields full / irrelevant / none (%) Generic-JS FIM acc
Qwen2.5-Coder-1.5B, zero-shot base 0.142 0.316 0.5052 0.667 3.76 / 10.61 / 12.26 0.406
v3 (v0.1.0-preview) 0.410 0.602 0.9616 0.998 1.21 / 5.56 / 7.10 0.606
v4 (v0.2.0-preview) 0.406 0.575 0.9185 0.990 2.00 / 12.63 / 10.97 0.628
KapiCoder-Mongo-Small (this model, v4b) 0.399 0.582 0.9297 0.991 1.82 / 9.60 / 12.26 0.626

Validator ceiling for exec-valid: 0.6255. Generic-JS: 500 cuts of permissive npm JavaScript (regression check).

Out-of-distribution homologation (homologation v1, editor-context-v1)

Internal evaluation, synthetic out-of-distribution homologation. The queries were not written by people. 6,030 fill-in-the-middle cuts from 28 schemas never used in training, written by Qwen3-14B and a grammar; not fully independent of the training data (same generator and grammar templates over other schemas). Same v1 prompt, greedy, up to 96 new tokens; exec-valid runs the completed text in mongosh behind a read-only proxy.

Model FIM acc (%) Exec-valid (%) Exec ÷ ceiling (%) Syntax-valid (%) Nonexistent fields full / none (%)
Qwen2.5-Coder-1.5B, zero-shot base 18.74 49.15 52.81 67.74 7.15 / 10.04
v3 (v0.1.0-preview) 45.37 83.07 89.26 99.72 7.50 / 8.74
v4 (v0.2.0-preview) 35.94 78.51 84.36 98.54 7.58 / 12.12
KapiCoder-Mongo-Small (this model, v4b) 44.61 84.26 90.53 99.62 7.34 / 7.01

Paired v4b − v3 on the same 6,030 items (95% CI): FIM accuracy -0.8 pp [-1.4, -0.1]; exec-valid +1.2 pp [+0.6, +1.8]. Paired v4b − v4: FIM accuracy +8.7 pp [+7.8, +9.5]; exec-valid +5.8 pp [+4.9, +6.6]. Validator ceiling for exec-valid: 93.07%.

Second training run (seed 20261115, not published)

Same data, settings and evaluation; only the seed changes. Between the two seeds, the cursor-in-call checks against v4 changed result: with the second seed, suffix-consistent and operator-valid fall about 1 point below v4 (beyond the 1-point non-inferiority margin); with the published seed they do not.

Check These weights (seed 20261104) Second seed
Homologation FIM accuracy 44.61% 44.73%
Cursor-in-call suffix-consistent (2k, v2) 94.9% 94.4%
Cursor-in-call operator valid (2k, v2) 94.8% 94.1%
Field-grounded with no header (n=135) 50.4% 43.7%
Homologation nonexistent fields, no or unrelated schema 5.71% 6.06%

Limits of this evaluation: the authors generated the data; exec-valid means "runs", not "does what was intended"; a single greedy evaluation run per model and seed.

Limitations and risks

  • Synthetic data only. Schemas are fictitious; expect lower scores on real collections, naming styles and query shapes.
  • Needs the target fields. With no header, fewer of the cited fields exist (50.0% on the cursor-in-call suite, v3: 54.5%); validate field names before running a suggestion.
  • Operators. About 5% of suggestions on the cursor-in-call suite still leave an operator where it is not legal or a document that does not parse; most remaining failures are in the middle of an operator ($g|) or right after $.
  • Long context. Trained up to 8k tokens; the farther the needed definition is from the cursor, the more it misses (see the long-context line above).
  • Two seeds only. Field-grounding without a header varies by several points between the seeds (table above); small differences against v3 or v4 are not established.
  • Dialects. Mongo Console JavaScript (ConnectionPool... and db.<collection> styles) and JSON pipelines only; chat is out of scope.
  • No safety guarantee. Destructive operations were neither trained nor tested for; during evaluation only read commands were executed. The host application must keep a human in the loop and must not run suggestions blindly.

Formats

Format Where Notes
safetensors, bf16 this repository Reference weights, loaded with transformers.
ONNX (onnxruntime-genai) esilva/KapiCoder-Mongo-Small-ONNX CPU builds for the Kapibara Studio editor, tag v0.3.0-preview. No DirectML build in this release.

No GGUF is provided. Neither the teacher model nor the training data are part of this repository.

License, modification notice and attribution

  • License: Apache License 2.0 (see LICENSE).
  • Modification notice: this model is a modified version of Qwen/Qwen2.5-Coder-1.5B (revision above). Changes: LoRA fine-tuning merged into the weights for MongoDB fill-in-the-middle autocomplete, with part of the labels distilled from a 7B teacher; export to safetensors bf16. It is not a product of Alibaba Cloud or the Qwen team and is not endorsed by them. See NOTICE.md.
  • Attribution: base model Qwen2.5-Coder by the Qwen team (Hui et al., Qwen2.5-Coder Technical Report, arXiv:2409.12186), Apache-2.0. The teacher was derived from Qwen2.5-Coder-7B (Apache-2.0); its weights are not part of this repository. Part of the training queries was written by Qwen3-14B (Apache-2.0).
  • The generic-code replay used npm packages under MIT, Apache-2.0, BSD-2/3-Clause and ISC licenses. The code itself is not redistributed; the package names, versions and licenses are listed in REPLAY_ATTRIBUTION.md.

Related model: KapiCoder-Mongo-Nano (0.5B), same task and contract.


Resumo em português (pt-BR)

Status: PREVIEW / experimental, preview público. KapiCoder-Mongo-Small é um modelo pequeno (cerca de 1,5 B de parâmetros) ajustado a partir do Qwen/Qwen2.5-Coder-1.5B para completar consultas MongoDB no cursor, inclusive com o cursor dentro de find, update e aggregate, no editor Kapibara Studio. Esta é a versão v4b dos pesos (tag v0.3.0-preview), que substitui a v4 (mesmo contrato, tag v0.2.0-preview); a v3 continua na tag v0.1.0-preview.

  • Contrato: editor-context-v2 (o cabeçalho v1 mais a linha TARGET FIELDS com os campos da coleção alvo) e a regra de saída que fecha o objeto interno deixado aberto. Versões do editor sem editor-context-v2 recusam este pacote; nelas, use a tag v0.1.0-preview.
  • Avaliação: avaliação interna sobre dados sintéticos; sem conjunto cego humano. Na suíte cursor-in-call o documento continua válido em 94,9% dos casos (v3: 87,8%) e, com o schema no cabeçalho, 99,2% dos campos citados existem. Com o prompt antigo (contrato v1, sem TARGET FIELDS), na homologação, FIM acc 44,61% contra 45,37% da v3 e exec-valid 84,26% contra 83,07%. Sem cabeçalho, 50,0% dos campos citados existem (v3: 54,5%).
  • Treino: 205.238 exemplos sintéticos, 124.537.220 tokens vistos, 20,50 horas em uma RX 7800 XT. Duas execuções de treino (duas sementes); estes são os pesos da primeira. Entre as duas sementes, as checagens de cursor contra a v4 mudaram de resultado: com a segunda semente, suffix-consistent e operador válido ficam cerca de 1 ponto abaixo da v4 (além da margem de 1 ponto); com a semente publicada, não. Sem dados pessoais por construção; não houve revisão humana das amostras.
  • Limites: só dado sintético; mande os campos da coleção no cabeçalho e valide os nomes antes de executar; o modelo só sugere texto e nunca executa.
  • Licença: Apache-2.0, com aviso de modificação: derivado e modificado do Qwen2.5-Coder-1.5B (equipe Qwen, Alibaba Cloud), sem endosso deles.