the parquet (what load_dataset serves) is canonical β 2,015 rows at 50.3%, the verified split that annotations/REPORT.md describes. The root JSONLs are an earlier, pre-verification generation; they're now moved to legacy/ to avoid confustion. The usage snippet tag is fixed too. Thanks for noticing :)
Adam Kovacs PRO
adaamko
AI & ML interests
Explainable AI, rule-based models, hallucination detection, fact-checking LLMs
Recent Activity
upvoted an article 4 days ago
A 0.6B model at #2 on MTEB(Law) liked a model 4 days ago
erikkaum/lattice-retrieval upvoted an article 5 days ago
Lattice: an 8 MB static retriever that embeds Wikipedia in 7 minutesOrganizations
replied to their post 6 days ago
posted an update 6 days ago
Post
104
π₯¬ LettuceDetect v2 β span-level hallucination detection for code, tool output, and structured documents.
Hallucination detectors are trained on document QA, but agents ground their answers in source code, tool output and markdown. On code-agent answers, existing detectors reach 0.17 span-F1 and even 550B zero-shot judges at most 0.22.
We built a unified span-level benchmark β 74,285 newly constructed examples (145K+ with RAGTruth and 14-language PsiloQA folded in), every span typed and character-labeled β and trained two detectors on it:
π€ KRLabsOrg/lettucedect-v2-qwen-2b β generative, typed spans + explanations in one pass, 32K context, **0.689 span-F1** (0.60 on code-agent)
β‘ KRLabsOrg/lettucedect-v2-mmbert-base β 307M multilingual encoder for high-throughput setups
π·οΈ KRLabsOrg/lettucedect-v2-taxonomy-head β types the spans of any binary detector
It also reaches the best reported English PsiloQA IoU (0.724) and 81.8 RAGTruth example-F1, so specializing on code didn't cost general RAG performance.
π Dataset: KRLabsOrg/lettucedetect-code-hallucination
π Paper: https://arxiv.org/abs/2607.00895
The models are now integrated natively into vLLM Semantic Router β joint blog post on how it works: https://vllm-sr.ai/blog/lettucedetect-v2-generative-hallucination-detection
Hallucination detectors are trained on document QA, but agents ground their answers in source code, tool output and markdown. On code-agent answers, existing detectors reach 0.17 span-F1 and even 550B zero-shot judges at most 0.22.
We built a unified span-level benchmark β 74,285 newly constructed examples (145K+ with RAGTruth and 14-language PsiloQA folded in), every span typed and character-labeled β and trained two detectors on it:
π€ KRLabsOrg/lettucedect-v2-qwen-2b β generative, typed spans + explanations in one pass, 32K context, **0.689 span-F1** (0.60 on code-agent)
β‘ KRLabsOrg/lettucedect-v2-mmbert-base β 307M multilingual encoder for high-throughput setups
π·οΈ KRLabsOrg/lettucedect-v2-taxonomy-head β types the spans of any binary detector
It also reaches the best reported English PsiloQA IoU (0.724) and 81.8 RAGTruth example-F1, so specializing on code didn't cost general RAG performance.
π Dataset: KRLabsOrg/lettucedetect-code-hallucination
π Paper: https://arxiv.org/abs/2607.00895
The models are now integrated natively into vLLM Semantic Router β joint blog post on how it works: https://vllm-sr.ai/blog/lettucedetect-v2-generative-hallucination-detection
- 3 replies
reacted to tomaarsen's post with β€οΈ 9 months ago
Post
4690
π€ Sentence Transformers is joining Hugging Face! π€ This formalizes the existing maintenance structure, as I've personally led the project for the past two years on behalf of Hugging Face! Details:
Today, the Ubiquitous Knowledge Processing (UKP) Lab is transferring the project to Hugging Face. Sentence Transformers will remain a community-driven, open-source project, with the same open-source license (Apache 2.0) as before. Contributions from researchers, developers, and enthusiasts are welcome and encouraged. The project will continue to prioritize transparency, collaboration, and broad accessibility.
Read our full announcement for more details and quotes from UKP and Hugging Face leadership: https://huggingface.co/blog/sentence-transformers-joins-hf
We see an increasing wish from companies to move from large LLM APIs to local models for better control and privacy, reflected in the library's growth: in just the last 30 days, Sentence Transformer models have been downloaded >270 million times, second only to transformers.
I would like to thank the UKP Lab, and especially Nils Reimers and Iryna Gurevych, both for their dedication to the project and for their trust in myself, both now and two years ago. Back then, neither of you knew me well, yet you trusted me to take the project to new heights. That choice ended up being very valuable for the embedding & Information Retrieval community, and I think this choice of granting Hugging Face stewardship will be similarly successful.
I'm very excited about the future of the project, and for the world of embeddings and retrieval at large!
Today, the Ubiquitous Knowledge Processing (UKP) Lab is transferring the project to Hugging Face. Sentence Transformers will remain a community-driven, open-source project, with the same open-source license (Apache 2.0) as before. Contributions from researchers, developers, and enthusiasts are welcome and encouraged. The project will continue to prioritize transparency, collaboration, and broad accessibility.
Read our full announcement for more details and quotes from UKP and Hugging Face leadership: https://huggingface.co/blog/sentence-transformers-joins-hf
We see an increasing wish from companies to move from large LLM APIs to local models for better control and privacy, reflected in the library's growth: in just the last 30 days, Sentence Transformer models have been downloaded >270 million times, second only to transformers.
I would like to thank the UKP Lab, and especially Nils Reimers and Iryna Gurevych, both for their dedication to the project and for their trust in myself, both now and two years ago. Back then, neither of you knew me well, yet you trusted me to take the project to new heights. That choice ended up being very valuable for the embedding & Information Retrieval community, and I think this choice of granting Hugging Face stewardship will be similarly successful.
I'm very excited about the future of the project, and for the world of embeddings and retrieval at large!
- 1 reply
posted an update 11 months ago
Post
372
We introduced π₯¬ TinyLettuce: lightweight hallucination detection with 17β68M encoders
Instead of relying on huge LLM judges that are slow and costly, we built tiny Ettin-based detectors that you can train in hours on a single GPU and run efficiently on CPU.
Hereβs whatβs inside:
1οΈβ£ Synthetic Data Generator
A toolkit to create hallucinations with controllable error typesβno manual annotation bottlenecks.
2οΈβ£ TinyLettuce Models (17β68M)
Compact classifiers built on Ettin encoders, designed for efficiency (8K context, modern transformer backbone).
3οΈβ£ Data & Training Utilities
Scripts and APIs to generate domain-specific labeled pairs at scale, plus ~3.6k examples we used for training.
4οΈβ£ Open and MIT-licensed
Code, data, and models are freely available for research and production.
5οΈβ£ Performance Highlights
- TinyLettuce-17M reaches 90.87% F1 (synthetic), outperforming GPT-OSS-120B (83.38%) and Qwen3-235B (79.84%)
- Runs in real-time on CPU: low latency, minimal memory, and pennies per million checks
- Shows competitive results on RAGTruth benchmarks
π Useful links:
Blog: https://huggingface.co/blog/adaamko/tinylettuce
GitHub: https://github.com/KRLabsOrg/LettuceDetect
PyPI: https://pypi.org/project/lettucedetect/
HF Collection: KRLabsOrg/tinylettuce-68b42a66b8b6aaa4bf287bf4
Notebook/Demo:
https://github.com/KRLabsOrg/LettuceDetect/blob/main/demo/tinylettuce.ipynb
β Stars and feedback are always appreciated!
Instead of relying on huge LLM judges that are slow and costly, we built tiny Ettin-based detectors that you can train in hours on a single GPU and run efficiently on CPU.
Hereβs whatβs inside:
1οΈβ£ Synthetic Data Generator
A toolkit to create hallucinations with controllable error typesβno manual annotation bottlenecks.
2οΈβ£ TinyLettuce Models (17β68M)
Compact classifiers built on Ettin encoders, designed for efficiency (8K context, modern transformer backbone).
3οΈβ£ Data & Training Utilities
Scripts and APIs to generate domain-specific labeled pairs at scale, plus ~3.6k examples we used for training.
4οΈβ£ Open and MIT-licensed
Code, data, and models are freely available for research and production.
5οΈβ£ Performance Highlights
- TinyLettuce-17M reaches 90.87% F1 (synthetic), outperforming GPT-OSS-120B (83.38%) and Qwen3-235B (79.84%)
- Runs in real-time on CPU: low latency, minimal memory, and pennies per million checks
- Shows competitive results on RAGTruth benchmarks
π Useful links:
Blog: https://huggingface.co/blog/adaamko/tinylettuce
GitHub: https://github.com/KRLabsOrg/LettuceDetect
PyPI: https://pypi.org/project/lettucedetect/
HF Collection: KRLabsOrg/tinylettuce-68b42a66b8b6aaa4bf287bf4
Notebook/Demo:
https://github.com/KRLabsOrg/LettuceDetect/blob/main/demo/tinylettuce.ipynb
β Stars and feedback are always appreciated!
reacted to tomaarsen's post with β€οΈ about 1 year ago
Post
4578
π I just published Sentence Transformers v5.1.0, and it's a big one. 2x-3x speedups of SparseEncoder models via ONNX and/or OpenVINO backends, easier distillation data preparation with hard negatives mining, and more:
1οΈβ£ Faster ONNX and OpenVINO backends for SparseEncoder models
Usage is as simple as
2οΈβ£ New
This new output format is immediately compatible with the MarginMSELoss and SparseMarginMSELoss for training SentenceTransformer, CrossEncoder, and SparseEncoder losses.
3οΈβ£ Gathering across devices
When doing multi-GPU training using a loss that has in-batch negatives (e.g. MultipleNegativesRankingLoss), you can now use
4οΈβ£ Trackio support
If you also upgrade
5οΈβ£ MTEB Documentation
We've added some documentation on evaluating SentenceTransformer models properly with MTEB. It's rudimentary as the documentation on the MTEB side is already great, but it should get you started.
Plus many more smaller features & fixes (crash fixes, compatibility with datasets v4, FIPS compatibility, etc.).
See the full release notes here: https://github.com/UKPLab/sentence-transformers/releases/tag/v5.1.0
Big thanks to all of the contributors for helping with the release, many of the features from this release were proposed by others. I have a big list of future potential features that I'd love to add, but I'm
1οΈβ£ Faster ONNX and OpenVINO backends for SparseEncoder models
Usage is as simple as
backend="onnx" or backend="openvino" when initializing a SparseEncoder to get started, but I also included utility functions for optimization, dynamic quantization, and static quantization, plus benchmarks.2οΈβ£ New
n-tuple-scores output format from mine_hard_negativesThis new output format is immediately compatible with the MarginMSELoss and SparseMarginMSELoss for training SentenceTransformer, CrossEncoder, and SparseEncoder losses.
3οΈβ£ Gathering across devices
When doing multi-GPU training using a loss that has in-batch negatives (e.g. MultipleNegativesRankingLoss), you can now use
gather_across_devices=True to load in-batch negatives from the other devices too! Essentially a free lunch, pretty big impact potential in my evals.4οΈβ£ Trackio support
If you also upgrade
transformers, and you install trackio with pip install trackio, then your experiments will also automatically be tracked locally with trackio. Just open up localhost and have a look at your losses/evals, no logins, no metric uploading.5οΈβ£ MTEB Documentation
We've added some documentation on evaluating SentenceTransformer models properly with MTEB. It's rudimentary as the documentation on the MTEB side is already great, but it should get you started.
Plus many more smaller features & fixes (crash fixes, compatibility with datasets v4, FIPS compatibility, etc.).
See the full release notes here: https://github.com/UKPLab/sentence-transformers/releases/tag/v5.1.0
Big thanks to all of the contributors for helping with the release, many of the features from this release were proposed by others. I have a big list of future potential features that I'd love to add, but I'm