End-to-end run with the extended glue rules (original run kept in end-to-end-20260930)
Browse files- .gitattributes +1 -0
- verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/README.md +6 -0
- verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/corrected-ext.jsonl +3 -0
- verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/predictions-threshold0.7.jsonl +0 -0
- verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/predictions-threshold0.8.jsonl +0 -0
- verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/scores.md +27 -0
.gitattributes
CHANGED
|
@@ -44,3 +44,4 @@ verifier-qwen3.5-2b/test_eval_entity/predictions.jsonl filter=lfs diff=lfs merge
|
|
| 44 |
verifier-qwen3.5-2b/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 45 |
verifier-qwen3.5-2b-v2/test_eval/predictions.jsonl filter=lfs diff=lfs merge=lfs -text
|
| 46 |
verifier-qwen3.5-2b-v2/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 44 |
verifier-qwen3.5-2b/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 45 |
verifier-qwen3.5-2b-v2/test_eval/predictions.jsonl filter=lfs diff=lfs merge=lfs -text
|
| 46 |
verifier-qwen3.5-2b-v2/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/corrected-ext.jsonl filter=lfs diff=lfs merge=lfs -text
|
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/README.md
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
End-to-end run of the edit pipeline with the extended glue rules (rules.py `extended=True`, the default in correct.py from this run on).
|
| 2 |
+
Test set ErfanRou/callcc-test-1k (1,978 transcripts); rewrite adapter qwen3.5-2b-100pct-masked-weighted-v2, verifier verifier-qwen3.5-2b-v2.
|
| 3 |
+
- corrected-ext.jsonl: correct.py --keep-intermediates output (after_rules, rewrite, every candidate with its verifier score)
|
| 4 |
+
- predictions-threshold0.7/0.8.jsonl: input/output/reference/edits_applied/premature_stop, evaluate.py-compatible
|
| 5 |
+
- scores.md: score_corrected.py table per slice and threshold
|
| 6 |
+
Replaces nothing; the earlier run (original rules) stays in ../end-to-end-20260930/.
|
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/corrected-ext.jsonl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d77db86724a967788861499464369f42a7cb709af8b5292228c9d8ec068df9a3
|
| 3 |
+
size 16639609
|
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/predictions-threshold0.7.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/predictions-threshold0.8.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
verifier-qwen3.5-2b-v2/end-to-end-20260930-extended-rules/scores.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# End-to-end corrector: outputs/edit/e2e/ext-upload/corrected-ext.jsonl
|
| 2 |
+
|
| 3 |
+
1978 transcripts scored; 9301 candidates; 74 rewrites skipped for stopping early.
|
| 4 |
+
|
| 5 |
+
| slice | method | WER | WER (clitic) | fixed | broken | net | broken/100 | entity fix | applied |
|
| 6 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 7 |
+
| main | whisper | 0.1929 | 0.1840 | 0 | 0 | 0 | 0.00 | 0.000 | 0 |
|
| 8 |
+
| main | rules only | 0.1851 | 0.1759 | 4050 | 130 | 3920 | 0.04 | 0.010 | 0 |
|
| 9 |
+
| main | accept all | 0.1827 | 0.1744 | 8501 | 3778 | 4723 | 1.26 | 0.120 | 9301 |
|
| 10 |
+
| main | verifier@0.3 | 0.1846 | 0.1755 | 5920 | 1818 | 4102 | 0.61 | 0.083 | 4339 |
|
| 11 |
+
| main | verifier@0.5 | 0.1848 | 0.1756 | 5098 | 1107 | 3991 | 0.37 | 0.062 | 2560 |
|
| 12 |
+
| main | verifier@0.7 | 0.1847 | 0.1755 | 4588 | 598 | 3990 | 0.20 | 0.043 | 1351 |
|
| 13 |
+
| main | verifier@0.8 | 0.1847 | 0.1755 | 4371 | 401 | 3970 | 0.13 | 0.028 | 811 |
|
| 14 |
+
| entity | whisper | 0.1866 | 0.1776 | 0 | 0 | 0 | 0.00 | 0.000 | 0 |
|
| 15 |
+
| entity | rules only | 0.1788 | 0.1694 | 2078 | 54 | 2024 | 0.03 | 0.010 | 0 |
|
| 16 |
+
| entity | accept all | 0.1762 | 0.1677 | 4383 | 1892 | 2491 | 1.21 | 0.120 | 4704 |
|
| 17 |
+
| entity | verifier@0.3 | 0.1783 | 0.1690 | 3032 | 899 | 2133 | 0.58 | 0.083 | 2194 |
|
| 18 |
+
| entity | verifier@0.5 | 0.1785 | 0.1692 | 2597 | 533 | 2064 | 0.34 | 0.062 | 1274 |
|
| 19 |
+
| entity | verifier@0.7 | 0.1785 | 0.1691 | 2345 | 289 | 2056 | 0.19 | 0.043 | 675 |
|
| 20 |
+
| entity | verifier@0.8 | 0.1787 | 0.1693 | 2224 | 201 | 2023 | 0.13 | 0.028 | 398 |
|
| 21 |
+
| typo | whisper | 0.1412 | 0.1316 | 0 | 0 | 0 | 0.00 | 0.000 | 0 |
|
| 22 |
+
| typo | rules only | 0.1333 | 0.1235 | 408 | 4 | 404 | 0.01 | 0.011 | 0 |
|
| 23 |
+
| typo | accept all | 0.1335 | 0.1245 | 807 | 383 | 424 | 1.12 | 0.189 | 871 |
|
| 24 |
+
| typo | verifier@0.3 | 0.1338 | 0.1242 | 554 | 153 | 401 | 0.45 | 0.156 | 369 |
|
| 25 |
+
| typo | verifier@0.5 | 0.1335 | 0.1238 | 489 | 89 | 400 | 0.26 | 0.122 | 208 |
|
| 26 |
+
| typo | verifier@0.7 | 0.1335 | 0.1238 | 448 | 52 | 396 | 0.15 | 0.067 | 112 |
|
| 27 |
+
| typo | verifier@0.8 | 0.1335 | 0.1237 | 425 | 32 | 393 | 0.09 | 0.033 | 65 |
|