Spaces:
Paused
Paused
Commit History
-
feat: add language detection to prepare_data.py, exclude non-English docs
- Uses langdetect to identify document language (samples pages 2-5) - 12 non-English docs excluded (8 Arabic, 4 French) - 99 English docs with has_revalidation=true - Added 'language' field to wbg_pdf_links.json entries85eb22c -
feat: prepare_data.py script + use has_revalidation filter in documents API
- prepare_data.py: scans local docs, uploads missing _direct_judged.jsonl to HF, generates updated wbg_pdf_links.json with src_docname and has_revalidation fields - documents API: only lists docs with has_revalidation=true - Fixed misleading error message in storage.jsfb404c5