DialFork

Benchmarks for procedural hallucination in conversational agents

Paper · GitHub · Dataset · 🏆 Leaderboard

LLM agents are increasingly asked to carry out procedures: book the appointment, verify the identity, file the report. Only a few steps of a procedure are tool calls; most are things the agent says: asking, confirming, telling you the result. An agent can stay fluent, polite and factually correct, get every tool call right, and still quietly drift from its procedure in the conversation itself. We call this procedural hallucination. DialFork is the first benchmark for this overlooked and complementary side of agents, the one users see and interact with: it measures procedural hallucination from the dialogue text alone, with labels that need no manual annotation.

The code to evaluate your methods, the paper's methods and the generation pipeline are on GitHub.

_{Made with 💛 at Idiap Research Institute · Happy forking! 🔱}