10.4 GB
754 files
Updated 16 days ago
Name
Size
artifacts
configs
model
output
scripts
src
.python-version5 Bytes
xet
GUIDE.md4.97 kB
xet
README.md31.7 kB
xet
pyproject.toml655 Bytes
xet
uv.lock11.4 kB
xet
README.md

Specification Discovery Benchmark

Contents

1. Motivation

Requirements discovery is an important part of building software with coding agents. For example, Matt Pocock's grill-me skill [1] and Anthropic's Claude Code guidance [2] recommend a questioning process to resolve uncertainties before producing specifications and implementation plans. These approaches rely on large language models' ability to identify uncertainties in the task context and formulate questions that help resolve them.

In my personal workflow, I use a similar interviewing and requirements-discovery approach. In my experience, LLMs often require guidance to identify gaps in specifications. They usually fail to identify important unresolved decisions that would change the solution. Instead, they focus on small details that have little impact on the feature. So, I find myself guiding them during the discovery process and identifying unresolved issues myself.

The problem extends beyond coding. A capable model has to identify gaps in the users' requests and resolve them before answering. This could also happen over multiple turns, so the model must not eagerly commit to a solution before resolving uncertainties that could affect the solution. During this process, the model has to recognize that new answers could establish a new constraint, invalidate an assumption, or establish that further questioning is unnecessary.

The same problem appears when a model acts in an environment it doesn't know. For example, in our repair environment, the instructions explain how to prepare for measurements, but the model can only find out whether replacing a part invalidates calibration by replacing it. Work on world-model learning studies how agents learn how their environment behaves by interacting with it [10]. WorldCoder [11] and PoE-World [12] write the learned world model as a Python program and check it against what they observe.

In this study, we examine whether a language model takes the steps needed to understand a problem before committing to an answer. We study this using a text-based game with three environments. In each environment, the model can inspect records, which works like asking a question with a fixed answer, and perform operations, which works like running an experiment. The model investigates the environment and submits its expressed understanding as an executable specification written in Quint [3], alongside its final solution. The specification is the model's world model of the environment, written as code. This allows us to test whether the specification's behavior matches the environment or not. Using the investigation trace, specification, and final solution, we check whether the model is able to gather sufficient evidence and use the gathered information to correctly solve the problem while correctly representing the environment.

2. Related work

QuestBench [4] and MT-InfoSeek [5] examine whether models can identify and acquire the information needed to solve a problem. They formalize missing information using constraint satisfaction problems. QuestBench focuses on models' ability to choose a question that provides enough information to solve the problem. However, it supplies the relationships governing the problem as well as the questions to choose from, while leaving some variable values unknown. MT-InfoSeek extends QuestBench's evaluation to multi-turn conversations, where several missing values may be needed to solve the problem. It also tests whether the acquired information determines the answer, alongside the correctness of the model's final answer.

AR-Bench [6] investigates information gathering through detective cases, situation puzzles, and number guessing. In detective cases and situation puzzles, the model questions other agents that have access to hidden stories. The benchmark measures the final answer as well as progress during the interaction. For these tasks, an LLM judge assesses how many reference key questions the conversation resolves.

Our benchmark requires the model to discover parts of the environment's behavior and submit its expressed understanding as an executable specification in Quint. AR-Bench's [6] situation puzzles also require an explanation, but score it using character-level F1 against a reference explanation. We execute the submitted specification to assess its consistency with observations and its accuracy beyond observations.

Our benchmark is also related to work on world-model learning. WorldCoder [11] builds a world model as a Python program from its interactions with the environment, and PoE-World [12] represents the world model of Atari games as a combination of many small programs. AutumnBench [10] evaluates world-model learning in 43 grid-world environments. The agent first interacts with an environment without a reward, and it is then tested on masked-frame prediction, change detection, and planning. The authors report that humans outperformed all five reasoning models they evaluated, and they attribute the gap to differences in exploration and belief updating. Doing Experiments and Revising Rules [13] and ZendoWorld [14] study agents that propose experiments to discover a hidden rule. ZendoWorld reports that labeling accuracy is often dissociated from rule recovery. Our benchmark assesses understanding differently. The model submits its world model as an executable Quint specification, and we execute it against the environment, including behavior beyond the model's own interactions. We also assess the gathered evidence, the final solution, and the specification separately. In our benchmark, the model can ask questions by inspecting records and run experiments by performing operations within the same attempt, and in seven of the twelve configurations it cannot reach the answer without operations.

3. Benchmark design and evaluation

3.1 Environments and tasks

The benchmark contains three environments: workshop, allocation, and repair. In the workshop environment, the model must figure out how much cash is available after outstanding payments and the maximum closing balance it can achieve by a deadline. In the allocation environment, the model has to assign two jobs, Alpha and Beta, to the resources A, B and C, meet their resource and service requirements, and keep as many credits as possible. In the repair environment, the model must figure out which of the valve, sensor, and regulator is faulty and which changes return the device to service before the deadline, leaving as much energy as possible and, at equal energy, releasing the device earlier. Figure 1 shows one dependency from each environment.

Each environment has different configurations. These define its initial conditions and behavior, such as the repayment conditions in workshop or the faulty component in repair. Therefore, the model may need to establish different information to figure out the correct solution.

Some of the information needed to solve a task is available in records that the model can inspect, such as invoices, service terms, and job requirements. Other information can only be established by performing operations and observing their effects. We determined which configurations require operations by checking which requirements of the evidence assessment remain unresolved after inspecting every record without performing any operation. All four workshop configurations can be solved using inspected records alone. In allocation, three of the four configurations require operations, for example to establish whether releasing Alpha's reservation cancels its approval. All four repair configurations require operations, because the faulty component has to be identified from measurements and the model has to establish whether replacing it invalidates calibration.

Overview of the three environments

Figure 1. One dependency from each environment, and whether it can be established from inspected records or only by performing operations. The diagrams do not prescribe an investigation order.

3.2 Investigation and submission

In the beginning, the model receives the task objective and an initial observation of the environment. It also receives the names of accessible objects, available operations, and some information about how the environment behaves. The model then chooses what to inspect and which operations to attempt. The model is allowed a fixed number of interactions in each environment, which lets us assess whether it chooses the inspections and operations needed to solve the task. Inspecting an object reveals the information available about it. This could include instructions, requirements, or links to other objects.

Operations may change the state of the environment. When the model performs an operation, it observes whether the operation was accepted or rejected and the resulting observable state. When an operation is rejected, the state stays unchanged but it may provide evidence about the operation's prerequisites. The model must combine its observations with the information it has inspected to solve the task. For example, in the repair environment, instructions explain the preparations needed for measurements. However, pressure and flow readings help distinguish the possible faults, as illustrated in Figure 2. The model needs to identify the faulty component, but it also needs to establish whether replacement requires recalibration and whether the corrective changes meet the time and energy constraints.

A worked investigation in repair

Figure 2. An authored investigation in repair configuration 1. Preparation and measurements distinguish the fault, and replacing the valve shows that calibration is no longer valid.

The model is allowed to reset the environment to its initial state. This allows the model to repeat an experiment or try a different approach. After resetting, the information in the conversation stays preserved and the model can continue investigating using what it has already learned.

Using the information it has gathered, the model submits a structured final solution and a complete Quint specification. Investigation ends at the first submission attempt. It also ends when the model exhausts its allowance. During the submission phase the model has up to 10 attempts to correct compilation and execution errors related to Quint, as we are not testing for the model's Quint proficiency.

3.3 Expressing understanding in Quint

The final solution shows whether the model can use the gathered information to solve the task, but it does not establish that the model understands the environment. We therefore require the model to submit an executable specification written in Quint, representing its expressed understanding of the environment, alongside the final solution. In the specification, the model writes down the initial state and how each operation changes it, and it also defines what can be observed and when the objective is achieved.

Using Quint makes the specifications executable, allowing us to check whether the specification submitted is correct and matches what the model has observed. It also makes checking it against the actual behavior of the environment beyond those observations possible. The evaluation records disagreements between the specification's outputs and the recorded observations to identify particular errors rather than relying on the overall correctness only.

3.4 Evaluation criteria

All scoring is done automatically without an LLM judge or manual conversation grading. We check the model's recorded inspections and operations to determine whether it established the facts and dependencies needed to solve the task. However, gathering sufficient evidence doesn't mean that the model will answer correctly. Therefore, we assess the evidence separately from the final answer.

We assess whether the model has submitted a Quint specification that accurately models the environment. We execute it and compare its behavior with the actual environment, including behavior beyond the interactions the model tried. We also record disagreements to identify what the specification represents incorrectly. Finally, the final solution is checked against an accepted reference answer.

3.5 Experimental setup and reporting

The evaluation is done using K2-Horizon-3.7B [7]. It was chosen as it is a small dense model that has features that modern LLMs have, like reasoning, native tool-calling, and long-context support. We use the model with the following configurations: a temperature of 1.0, top-p of 0.95, and the high reasoning setting, with a context limit of 262,144 tokens and a maximum of 32,768 tokens per response.

The study contains 3 environments, each with 4 configurations. We run 10 attempts for each configuration, each in a separate conversation, totaling 120 attempts. We report the results by configuration, environment, and overall, giving each configuration equal weight. We calculate 95% confidence intervals where measurements are complete. For binary outcomes, we use Wilson intervals within each configuration [8]. For aggregate results and continuous measurements, we calculate bootstrap intervals [9] by resampling whole attempts within each configuration 2,000 times. We omit the bootstrap interval when resampling produces no variation.

4. Results and analysis

The model was able to gather sufficient evidence in 64 attempts and submitted a correct final solution in 30 attempts overall. Evidence sufficiency was 53.3%, with a 95% confidence interval of 45.8–60.8%, while final-solution correctness was 25.0%, with an interval of 18.3–31.7%. The results differed across environments, as shown in Table 1.

Environment Sufficient evidence Correct final solution Spec: initial state Spec: operations Spec: observations Spec: calculations Spec: overall
Workshop 30/40 14/40 37/37 0/37 37/37 6/37 0/37
Allocation 18/40 16/40 36/38 0/38 38/38 30/38 0/38
Repair 16/40 0/40 6/30 0/30 9/30 28/30 0/30
Overall 64/120 30/120 79/105 0/105 84/105 64/105 0/105

Table 1. Results by environment and overall. The specification columns show how many submissions correctly described each aspect of the environment. Overall correctness requires passing all specification checks. The 15 attempts where the model didn't submit compilable specifications are excluded from the specification columns.

4.1. Investigation and stopping

The model decided to submit its answer before exhausting the allowance in 45 attempts. However, in 18 of these attempts, it hadn't gathered sufficient evidence to figure out the correct answer. In the other 75 attempts, the investigation ended because the model exhausted its allowance. Of these, 38 ended without sufficient evidence.

One of the attempts that ended in voluntary submission without sufficient evidence was allocation configuration 3, attempt 8. In this configuration, the Alpha job had a resource assignment and approval for the required service. The model decided to move it to another resource and reported that 14 credits would remain. However, it didn’t check whether removing the existing assignment would cancel the approval. Renewing that approval would cost two credits, leaving 12 instead of 14. The environment preserved approval in this configuration, so the answer was correct. However, the investigation had not established whether the extra payment was necessary.

A similar problem happened in the repair environment, configuration 1, attempt 5. The model was able to identify the faulty valve. However, it didn't replace it during investigation to check whether replacement would require calibration. It then submitted its answer without including calibration in the corrective changes. On the other hand, in configuration 3, attempt 2, the model used pressure and flow readings to identify the faulty regulator. After replacing it, the model observed that calibration was no longer valid and established that recalibration was needed.

4.2. Using the gathered information

As shown in Figure 3, gathering sufficient evidence did not always lead to the correct final answer. This is particularly apparent in the workshop and repair environments. Out of 64 attempts with sufficient evidence, only 24 produced correct solutions. However, in the allocation environment, six attempts produced correct solutions without establishing all the evidence required to figure out a correct answer.

Evidence sufficiency and final-answer correctness across the three environments.

Figure 3. Attempts grouped by evidence sufficiency and answer correctness. Each bar represents the 10 attempts of one configuration, and the configurations are grouped as in Section 4.4.

In the workshop environment, configuration 1, attempt 3, the model gathered sufficient evidence to figure out the correct answer. The model was able to establish the initial balance of 100,000 pence with obligations of 62,000 pence. This left 38,000 pence available after those obligations. It also figured out that returning and inspecting the equipment would increase the amount by 8,000 pence after the inspection fee. Therefore, the maximum closing balance was 46,000 pence. But the model reported 46,000 pence for both the available cash and the maximum closing balance. The available cash should have been 38,000 pence.

A similar problem happened in the repair environment, configuration 1, attempt 4. The model gathered sufficient evidence to figure out the correct answer. However, its final answer omitted the required calibration and incorrectly stated that the device could not be returned to service. More generally, none of the 16 attempts with sufficient evidence in this environment produced a correct final solution.

4.3. Expressed understanding of the environment

The model didn't produce specifications that matched the environment across all behavior checks. All 105 evaluated specifications contained an error in how they represented at least one operation. Figure 4 shows where these errors occurred. For instance, all 37 submitted workshop specifications represented returning equipment incorrectly. Furthermore, all 30 repair specifications contained errors in how several operations worked, including calibration, isolation, and pressure measurement.

Specification errors by operation and other aspects of the environment.

Figure 4. The percentage of specifications with at least one mismatch in each operation and in the initial state, observations, and calculations. Recovered record values were correct in every specification and are not shown.

These errors also occurred even when the model submitted the correct answer. Only 20 of the 105 specifications matched all checked observations from the investigation. Of the 29 correct final solutions submitted with an evaluated specification, 16 came with a specification that contradicted at least one of these observations. For example, in the workshop environment, configuration 4, attempt 2, the model reported the correct answer of 38,000 for both the available cash and maximum closing balance. During investigation, the environment accepted a payment of 12,000 pence for materials. However, the model submitted a specification that required the amount to be zero, which meant it rejected the same payment. Therefore, the model submitted a correct answer along with a specification that contradicted its observations.

In the workshop environment, configuration 1, attempt 3, the opposite occurred. The model answered incorrectly but submitted a specification that matched all checked observations from its investigation. Its accuracy on additional cases was 66.6%, and its final answer was also incorrect.

4.4. Information established by acting

The configurations differ in whether they can be solved from inspected records alone, as described in Section 3.1. In the five configurations that can be solved this way, the model gathered sufficient evidence in 38 of 50 attempts and submitted a correct final solution in 21. In the seven configurations that require operations, it gathered sufficient evidence in 26 of 70 attempts and submitted a correct final solution in 9. Table 2 shows the results for each group, and Figure 3 shows each configuration separately.

Configurations Attempts Sufficient evidence Correct final solution Specification matches all checked observations
Solvable from inspected records (workshop 1–4, allocation 2) 50 38 21 13/46
Require operations (allocation 1, 3, 4; repair 1–4) 70 26 9 7/59

Table 2. Results grouped by whether a configuration can be solved from inspected records alone. The last column excludes attempts without a compilable specification.

In allocation configuration 4, the model had to perform a dispatch to establish that dispatch requires neither approval nor an adapter. It did not establish this in any of the 10 attempts, and none of them produced a correct final solution.

The two groups are not a controlled comparison, because repair is the only environment where the fault has to be diagnosed from measurements. The closest comparison is within allocation, where configuration 2 can be solved from records alone, and the model gathered sufficient evidence in 8 of its 10 attempts there compared with 10 of 30 in the other three configurations.

5. Limitations

The environments simplify requirements discovery. They have predefined objectives and behavior. Furthermore, they do not capture changing intentions during conversations, conflicting requirements, or negotiation over what the problem should be. Every answer in our environments is fixed and available, so the benchmark tests capabilities that requirements discovery depends on rather than requirements discovery itself.

Assessment of expressed understanding depends on Quint proficiency. The model may gather sufficient evidence but struggle to write its understanding as executable Quint specifications.

We wrote the evidence requirements for each configuration by hand, five for each workshop configuration, seven for each repair configuration, and between four and seven for allocation, and the assessment only checks these requirements. We generated the behavior checks from the states that the reference implementation reaches from the initial state, which range from 140 states in workshop configuration 2 to 55,347 in repair configuration 1, and we do not check specifications in any other state.

We ran all 120 attempts with K2-Horizon-3.7B, using the same sampling settings and interaction allowance throughout, so the results describe this model only. At the start of each attempt, the model receives the available operations and the first records, and it finds the remaining records through links in the responses, as with the gauge record in repair, which is linked from the service record.

The model writes its specification once, at the end, so we only see its final description of the environment. In workshop configuration 4, attempt 2, for example, the submitted specification rejected a payment that the environment had accepted earlier in the same conversation.

6. Proposed path forward

Our next step is to repeat the 120 attempts with larger models. We also plan to ask the model for an updated Quint specification after each inspection or operation, and to replay the recorded commands against each of these specifications in the same way as we do for the final one. This will show at which step a specification first contradicts an observation and whether the model's next operation tests that part of it. In repair configuration 1, attempt 5, the model replaced no part at all, so no operation in that conversation tested how a replacement affects calibration. We also considered fine-tuning the model on investigations that establish missing facts, and on pairs of configurations that differ in one condition, such as allocation configurations 1 and 3.

7. Reproducing the evaluation

We designed three interactive environments for this evaluation. Their source code and evaluation configurations are included in this repository.

The running guide provides instructions for installing the dependencies, preparing the study, and running the conversations on Colab. It also explains how to score the resulting conversations locally and generate the evaluation report. Figures 2 to 4 are generated from the saved scores by scripts/make_figures.py.

The accompanying artifacts include an interactive benchmark guide with worked examples and a practice mode, and an evaluation report covering all 120 attempts. The saved attempt folders have one folder per attempt with the conversation and its scores, and the Quint specification and final solution for the 105 attempts with a valid submission. The submission guide lists what is in the bucket. We also uploaded the exact model checkpoint we used to the model folder, and source.json records its original revision and file checksums.

References

  1. Matt Pocock. (2026). The /grill-me Skill. AIHero. Updated 24 August 2026.

  2. Anthropic. Best practices for Claude Code: Let Claude interview you. Claude Code documentation. Accessed 21 October 2026.

  3. Quint contributors. The Quint specification language. Software and documentation. Accessed 21 October 2026.

  4. Belinda Z. Li, Been Kim, and Zi Wang. (2025). QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?. NeurIPS 2025, Datasets and Benchmarks Track. arXiv:2503.22674v2.

  5. Yepeng Huang, Jiawen Zhang, Michelle Dai, Xiaorui Su, Shanghua Gao, Zi Wang, and Marinka Zitnik. (2026). Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking. arXiv:2608.14808v1.

  6. Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. (2025). From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?. ICML 2025. arXiv:2506.08295v1.

  7. IFM. K2-Horizon-3.7B. Model checkpoint and model card. Hugging Face, revision 09568bab0d7ee4152a12ee13c21902ca910d15b2.

  8. Edwin B. Wilson. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 209–212.

  9. B. Efron. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1–26.

  10. Archana Warrier, Dat Nguyen, Michelangelo Naim, Moksh Jain, Yichao Liang, Karen Schroeder, Cambridge Yang, Joshua B. Tenenbaum, Sebastian Vollmer, Kevin Ellis, and Zenna Tavares. (2026). Benchmarking World-Model Learning with Environment-Level Queries. ICML 2026. arXiv:2510.19788.

  11. Hao Tang, Darren Key, and Kevin Ellis. (2024). WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment. NeurIPS 2024. arXiv:2402.12275.

  12. Wasu Top Piriyakulkij, Yichao Liang, Hao Tang, Adrian Weller, Marta Kryven, and Kevin Ellis. (2025). PoE-World: Compositional World Modeling with Products of Programmatic Experts. NeurIPS 2025. arXiv:2505.10819.

  13. Wasu Top Piriyakulkij, Cassidy Langenfeld, Tuan Anh Le, and Kevin Ellis. (2024). Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning. NeurIPS 2024. arXiv:2402.06025.

  14. Sophia Koehler, Antonia Wüst, Inga Ibs, Wasu Top Piriyakulkij, Wolfgang Stammer, Constantin Rothkopf, Kevin Ellis, and Kristian Kersting. (2026). Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction. arXiv:2607.08233.

Total size
10.4 GB
Files
754
Last updated
Nov 4
Pre-warmed CDN
US EU US EU

Contributors