Remove evaluation benchmark IDs from YAML datasets header so HF does not label eval benchmarks as training data
12314e0 verified | license: cc-by-nc-4.0 | |
| library_name: transformers | |
| pipeline_tag: text-classification | |
| gated: manual | |
| extra_gated_prompt: "Access to ProKope-421M weights is provided for academic evaluation, benchmarking, and commercial licensing inquiries. Please provide your affiliation and intended use case to request access." | |
| extra_gated_fields: | |
| Company or Institution: text | |
| Corporate or Academic Email: text | |
| Intended Evaluation Use Case: text | |
| language: | |
| - en | |
| tags: | |
| - text-classification | |
| - typed-decisions | |
| - modernbert | |
| - prokope | |
| model_name: ProKope-421M | |
| author: Alan Canto | |
| creator: Alan Canto | |
| <p align="center"> | |
| <img src="banner.jpg" alt="ProKope-421M" width="100%" style="border-radius: 8px; margin-bottom: 16px;"> | |
| </p> | |
| # ProKope-421M | |
| ## Introduction | |
| **ProKope-421M** is a 421M-parameter non-autoregressive classification and scoring model developed by [ProKope AI](https://huggingface.co/ProKope-AI), based on `ModernBERT-large` with a 1,024-token context window. It evaluates structured state inputs in a single forward pass across three typed decision primitives: | |
| 1. **`noul`**: Binary boolean classification (`[0, 1]` probability). | |
| 2. **`choice`**: Multi-class categorical routing. | |
| 3. **`score`**: Ordinal severity and confidence estimation (`[0, 1]` scalar). | |
| For 8,192-token context workloads, see [ProKope-421M-8K](https://huggingface.co/ProKope-AI/ProKope-421M-8K). | |
| --- | |
| ## Evaluation | |
| Evaluated across 400 test cases (2,000 total decisions) on the [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark: | |
| | Model | Parameters | Context | Overall Accuracy | Decisions Correct | Brier Score | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | | |
| | **ProKope-421M-8K** | 421M | 8,192 | **71.85%** | 1,437 / 2,000 | 0.384 (raw) | | |
| | Featherless Simple Jev | 35B | 8,192 | 71.60% | 1,432 / 2,000 | 0.110 | | |
| | **ProKope-421M** | 421M | 1,024 | 71.40% | 1,428 / 2,000 | **0.071** (calibrated) | | |
| | prima-ratio | 12B | 8,192 | 70.20% | 1,404 / 2,000 | 0.125 | | |
| | mgoeckel/oscar-1-400m | 400M | 1,024 | 70.00% | 1,400 / 2,000 | 0.062 | | |
| | OpenDecider-small | 4B | 8,192 | 67.10% | 1,342 / 2,000 | 0.117 | | |
| | Bekko System One v0 | 400M | 1,024 | 66.80% | 1,336 / 2,000 | 0.113 | | |
| ### Domain Breakdown (ProKope-421M: 71.40% / 1,428 / 2,000) | |
| - **Invoice Processing**: 79.00% (395 / 500) | |
| - **Agent-Trace Observability**: 73.20% (366 / 500) | |
| - **Security Incidents**: 69.20% (346 / 500) | |
| - **Customer Service Routing**: 64.20% (321 / 500) | |
| ### Decision Primitive Breakdown | |
| - **`noul` (Binary Decision)**: 79.67% | |
| - **`choice` (Multi-Class Routing)**: 69.67% | |
| - **`score` (Ordinal Severity)**: 66.50% | |
| --- | |
| ## Model Details | |
| | Property | Value | | |
| | :--- | :--- | | |
| | **Backbone** | `ModernBERT-large` (28 bidirectional encoder layers, 1,024 hidden size) + 2-layer multi-head decision projection | | |
| | **Total Parameters** | 421,293,830 (`803.57 MB` in `bfloat16`) | | |
| | **Context Window** | 1,024 tokens (`head_max_len`: 256 tokens) | | |
| | **Throughput** | ~23.5 decisions/sec (~42 ms/decision on single GPU) | | |
| | **Recommended VRAM** | 2 GB+ for `bfloat16` / `float16` inference | | |
| | **Precision** | `torch.bfloat16` | | |
| | **License** | `CC-BY-NC-4.0` (Commercial licensing available via ProKope AI) | | |
| --- | |
| ## Quickstart | |
| ### 1. Python Inference (`RLAgent`) | |
| ```python | |
| from rl_agent_api import RLAgent | |
| agent = RLAgent("ProKope-AI/ProKope-421M") | |
| state = ( | |
| "Production API Gateway alert: 504 Gateway Timeout spiked to 14.8% over the last 3 minutes. " | |
| "Worker node 4 pod eviction reported due to memory pressure (96.2%)." | |
| ) | |
| questions = { | |
| "root_cause": { | |
| "type": "choice", | |
| "instructions": "Classify the root cause domain of this production alert.", | |
| "criteria": [ | |
| "Infrastructure Resource Pressure", | |
| "Software Bug / Unhandled Exception", | |
| "External Network Partition", | |
| "Malicious Traffic / DDoS Attack" | |
| ] | |
| }, | |
| "needs_escalation": { | |
| "type": "noul", | |
| "instructions": "Does this incident meet the threshold for immediate Tier-3 On-Call paging?", | |
| "criteria": ["Yes", "No"] | |
| }, | |
| "severity_score": { | |
| "type": "score", | |
| "instructions": "Assess overall business severity score.", | |
| "criteria": ["Low", "Medium", "High", "Critical"] | |
| } | |
| } | |
| results = agent.system_one(state, questions) | |
| for key, ans in results["answers"].items(): | |
| print(key, "->", ans) | |
| ``` | |
| ### 2. Reproducing the Benchmark Evaluation | |
| ```bash | |
| python eval_prokope.py | |
| ``` | |
| --- | |
| ## Limitations & Intended Use | |
| - **Intended Use**: Low-latency structured classification, incident triage, workflow routing, and calibrated probability estimation over bounded decision schemas (`noul`, `choice`, `score`). | |
| - **Limitations**: `ProKope-421M` is a non-autoregressive encoder-classifier and does not generate free-form text. Inputs exceeding the 1,024-token context window are truncated. | |
| --- | |
| ## Contact & Citation | |
| - **Organization**: [ProKope AI](https://huggingface.co/ProKope-AI) | |
| - **Authors**: Alan Canto, Wesley Foreman | |
| - **Inquiries & Commercial Licensing**: [prokope.model@gmail.com](mailto:prokope.model@gmail.com) | |
| ```bibtex | |
| @misc{canto2026prokope421m, | |
| author = {Alan Canto and Wesley Foreman}, | |
| title = {ProKope-421M: Non-Autoregressive Structured Typed Decision Model}, | |
| howpublished = {Hugging Face}, | |
| year = {2026}, | |
| url = {https://huggingface.co/ProKope-AI/ProKope-421M} | |
| } | |
| ``` | |