solintellegence/Sol-Lite-Base

Sol Lite Base

Sol Lite Base is a compact 14,995,843-parameter decoder-only language model trained from scratch on 8,153,333,760 tokens.

It combines grouped-query attention, selective recurrent depth, XSA value subtraction, loop conditioning, and a small learned n-gram memory.

This is a base model, not an instruction-tuned assistant.

Model summary

Property	Value
Parameters	14,995,843
Training tokens	8,153,333,760
Architecture	Decoder-only causal language model
Physical / effective blocks	10 / 14
Residual width	256
Vocabulary	4,096-token digit-aware byte-level BPE
Maximum context	2,048 tokens
Weight format	FP32 safetensors
License	CC BY 4.0

Quick start

Install the runtime dependencies:

pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub

Load the model and generate text:

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("solintellegence/Sol-Lite-Base")
sys.path.insert(0, model_dir)

from modeling_sol_lite import load_model, generate

model, tokenizer = load_model(model_dir, device="cpu")

text = generate(
    model,
    tokenizer,
    "The future of efficient language models is",
    max_new_tokens=64,
)

print(text)

Sol Lite Base uses the included custom PyTorch class SolForCausalLM.

The repository includes the complete standalone model implementation and does not require the original training code.

Results

Benchmark	Examples	Score
HellaSwag	10,042	27.72%
ARC-Easy	2,376	34.72%
ARC-Challenge	1,172	22.87%
PIQA	1,838	57.73%
ArithMark-3	1,000	34.20%
Intelligence Index	—	8.799

The four language-model tasks were evaluated zero-shot on their complete evaluation splits with lm-eval 0.4.12 using normalized accuracy.

ArithMark-3 used its independent-tokenization normalized-accuracy protocol.

Raw evaluation outputs are included under evals/.

These measurements are not independently verified, and no leaderboard position is claimed.

Architecture

Component	Configuration
Stored parameters	14,995,843
Physical / effective blocks	10 / 14
Recurrent layout	1 prelude + 4 middle blocks × 2 passes + 5 coda
Residual width	256
Attention	8 query heads, 2 key/value heads, head dimension 32
Attention modification	XSA value subtraction after causal attention
Attention normalization	Per-head Q/K RMSNorm with RoPE
Recurrent conditioning	Learned pass embeddings and channel-wise refresh gates
MLP	Gated SiLU MLP, width 1,465
Vocabulary	4,096-token digit-aware byte-level BPE
Learned memory	2,048-entry bigram/trigram EngramLite
Embeddings	Tied input/output table
Maximum context	2,048 tokens

Training data

Training used a gradual English curriculum built from:

• FineWeb-Edu
• DCLM
• English UltraFineWeb levels 1–3
• FineWeb-HQ
• FineMath
• FinePhrase

The tokenizer and training stream were frozen before training.

Public benchmark examples were not used for checkpoint selection.

Intended use

Sol Lite Base is intended for research on highly parameter-efficient language modeling, including:

• extremely small language models
• recurrent-depth architectures
• grouped-query attention at small scale
• compact learned memory
• distillation experiments
• efficient inference research

Because this is a base model, users should expect raw next-token completion behavior rather than instruction-following or assistant-style responses.

Limitations

At approximately 15M parameters, Sol Lite Base has significant capability limitations.

Outputs may be:

• inconsistent
• factually incorrect
• repetitive
• incoherent
• unsafe

The model is not instruction tuned and should not be treated as a conversational assistant.

Do not use it for high-stakes decisions.

Users are responsible for evaluating outputs and determining whether the model is suitable for their application.

Repository files

File	Purpose
model.safetensors	FP32 model weights
modeling_sol_lite.py	Standalone model, loader, and generator
config.json	Architecture configuration
tokenizer.json	Frozen tokenizer
tokenizer_config.json	Tokenizer metadata
training_state.json	Training and provenance summary
evals/	Raw benchmark results

Reproducibility and evaluation notes

The repository includes raw evaluation outputs under evals/.

Reported benchmark values should be interpreted as reference measurements for this release. They have not been independently verified.

No leaderboard position is claimed.

License

Sol Lite Base is released under the CC BY 4.0 license.

Dataset licenses and terms remain with their respective owners.