update README (#1)

- update README (4e684e06f6055b6c095b1f9445b81e9744d92889)
- update overview image (aace1e6af2d57927d1fd05600f08a26dccd0dbb8)
- update README (e44c78b4432fe41a4fad34ee1edfa411cf601f18)


Co-authored-by: Chengying Tu <ChaseyT@users.noreply.huggingface.co>

README.md

@@ -32,10 +32,10 @@ For each language, we train and apply a dedicated L2 scorer to the corresponding
 
 ## 💡 Highlights
 
-> **Abstract:** Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present **UltraData-Code** as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of **UltraData-Code-L2**, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of **UltraData-Code-L3** in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by **7.80 points** and on MultiPL-E by **5.13 points**, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42 points** on EvalPlus and **8.07 points** on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.
+> **Abstract:** Code generation has become a core capability of large language models (LLMs), and code data is central to the pre-training process that develops it. As obtaining public code of sufficient quality becomes more difficult, further progress cannot rely on corpus expansion alone. Tiered data management instead applies increasingly targeted processing to smaller data pools, producing data states suited to different training objectives. Following this perspective, we present **UltraData-Code** as four connected data states spanning repository archival at L0, standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 preserves repository data and provenance for reprocessing, while L1 applies scalable filtering, normalization, and near-deduplication for broad pre-training. L2 selects high-quality algorithmically relevant files using language-adaptive relevance signals and quality constraints, yielding approximately 400B tokens of **UltraData-Code-L2** across 11 programming languages for focused continual pre-training. L3 converts selected implementations into structured programming exercises, yielding approximately 150B tokens of **UltraData-Code-L3** across the same 11 languages for task-oriented mid-training or supervised fine-tuning (SFT). Under controlled 10B-token continual pre-training of a 1B model, L2 improves pass@1 over L1 by **7.80** percentage points (pp) on EvalPlus and **5.13pp** on MultiPL-E. Replacing half of the L2 training tokens with L3 yields a further gain of **8.42pp** and **8.07pp** on the two benchmarks, respectively. Both gains increase when the training budget scales to 100B tokens. We have released our datasets and classifiers to support further research on LLMs.
 
-- **A connected code data construction pipeline.** UltraData-Code links repository archival and standardized natural code with selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages.
-- **Fine-grained curation of algorithmically relevant code.** We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection.
+- **A connected code data construction pipeline.** UltraData-Code links repository archival and standardized natural code with selection and structured synthesis, yielding approximately 400B tokens at L2 and 150B tokens at L3 across 11 programming languages. The datasets have been released to facilitate the development of LLMs.
+- **Fine-grained curation of algorithmically relevant code.** We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection.
 - **Implementation-grounded, task-oriented synthesis.** We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation.
 
 <div align="center">

README_ZH.md

@@ -18,10 +18,10 @@
 
 ## 💡 亮点
 
-> **摘要：** 代码生成逐渐成为大语言模型的核心能力，而代码数据则是培养这一能力的预训练过程中的关键组成部分。随着代码语料规模不断增长，其规模、多样性与质量日益影响模型在预训练阶段习得的能力。遵循分级数据治理视角，我们提出 **UltraData-Code**，将其构建为由四个相互衔接的数据状态组成的数据家族：从 L0 的代码仓库归档，到 L1 的标准化自然代码、L2 的算法导向筛选，再到 L3 的任务导向合成。L0 归档每个公开 GitHub 仓库默认分支上的最新版本，并保留其文件结构、关联关系与来源信息。L1 通过可扩展的过滤、规范化与近重复去重，获得标准化自然代码。随后，L2 基于由文件角色与启发式规则构成的语言自适应信号，并结合代码质量约束，从 L1 中筛选与算法相关的文件，最终得到覆盖 11 种编程语言、规模约为 400B tokens 的 **UltraData-Code-L2**。L3 对 L2 中的算法文件实施任务导向合成，将每个实现转化为结构化编程练习，生成覆盖相同 11 种语言、规模约为 150B tokens 的 **UltraData-Code-L3**。在相同的 10B-token 持续预训练设置下，对一个 1B 模型使用 L2 而非 L1 进行训练，可使其在 EvalPlus 和 MultiPL-E 上的 pass@1 分别提高 **7.80 个百分点** 和 **5.13 个百分点**，同时分别超过 Stack-Edu 4.37 个百分点和 3.05 个百分点。将一半的 L2 训练 tokens 替换为 L3 后，相较于仅使用 L2 训练，模型在 EvalPlus 和 MultiPL-E 上可进一步提升 **8.42 个百分点** 和 **8.07 个百分点**，同时分别超过最强的合成数据基线 5.57 个百分点和 7.80 个百分点。当训练预算增加至 100B tokens 时，L2 筛选与 L3 合成在两个基准上的增益均进一步扩大。
+> **摘要：** 代码生成已成为大语言模型（LLMs）的核心能力，而代码数据则是形成这一能力的预训练过程中的关键要素。随着获取高质量公开代码数据的难度日益增加，仅依赖语料规模扩张已难以推动模型能力持续提升。为此，分层数据管理通过对规模逐级缩减的数据池实施针对性不断增强的处理，构建适配不同训练目标的数据形态。基于这一理念，本文提出 **UltraData-Code**，其包含四个相互衔接的数据层级：L0 层的代码仓库归档、L1 层的标准化自然代码、L2 层的算法导向筛选，以及 L3 层的任务导向合成。其中，L0 层完整保留代码仓库数据及其来源信息，以支持后续再处理；L1 层通过可扩展的过滤、规范化与近重复数据消除，为通用预训练提供标准化代码数据；L2 层利用语言自适应的相关性信号与质量约束，筛选具有较高质量和算法相关性的代码文件，最终构建覆盖 11 种编程语言、规模约为 400B tokens 的 **UltraData-Code-L2**，用于面向特定能力的持续预训练；L3 层则将筛选后的代码实现转化为结构化编程练习，形成覆盖相同 11 种编程语言、规模约为 150B tokens 的 **UltraData-Code-L3**，适用于任务导向的中期训练或监督微调（SFT）。在受控实验中，我们使用 10B tokens 的数据对一个 1B 模型进行持续预训练。结果表明，与 L1 相比，L2 使模型在 EvalPlus 和 MultiPL-E 基准上的 pass@1 分别提升 **7.80** 和 **5.13** 个百分点。进一步以 L3 数据替换一半的 L2 训练 tokens 后，模型在上述两个基准上的性能分别再提升 **8.42** 和 **8.07** 个百分点。当训练预算扩展至 100B tokens 时，这两类性能增益均进一步扩大。为推动大语言模型相关研究，我们已公开所构建的数据集与分类器。
 
-- **完整的代码数据构建流水线：** 将仓库归档和规范化自然代码与细粒度精筛、结构化合成连接起来，在 11 种编程语言上分别形成约 400B tokens 的 L2 和约 150B tokens 的 L3。
-- **细粒度算法相关代码精筛：** 我们提出一种语言自适应筛选框架，将文件角色监督与语言特定启发式线索结合，学习超越显式 ALGO 文件的算法相关性。该框架在文件角色、算法相关性和代码质量模型之间复用预计算语义表征，最终结合角色与质量约束完成筛选。
+- **完整的代码数据构建流水线：** 将仓库归档和规范化自然代码与细粒度精筛、结构化合成连接起来，在 11 种编程语言上分别形成约 400B tokens 的 L2 和约 150B tokens 的 L3。相关数据集现已公开发布，旨在推动大语言模型的研发与能力提升。
+- **细粒度算法相关代码精筛：** 我们提出一种语言自适应筛选框架，将文件角色监督与语言特定启发式线索结合，学习代码文件的算法相关性。该框架在文件角色、算法相关性和代码质量模型之间复用预计算语义表征，最终结合角色与质量约束完成筛选。
 - **任务导向合成：** 我���提出一种结构化合成方案，将 L2 筛选出的每个算法相关实现转化为一道编程练习，并基于同一源实现联合生成独立完整的任务、分析、解答与候选测试用例。该转换既保留了源代码的计算意图，又为代码生成引入了显式的任务监督与解答监督。
 
 <div align="center">