Paper Type
Complete
Paper Number
PACIS2026-2052
Description
Large Language Models (LLMs) are increasingly used in iterative workflows, yet common benchmarks mainly evaluate single-turn tasks and fail to measure performance in extended interactions. This article addresses this gap by introducing a benchmark based on programmatically verifiable Python task chains. In this framework, models iteratively extend and refactor a growing codebase across sequential steps, while unit tests evaluate correctness and trigger a single opportunity for self-correction after failures. Ten state-of-the-art models, including GPT-5 and Gemini 2.5 Pro, are evaluated. Results reveal substantial performance differences and show that sequential reasoning depth and error recovery are only partially correlated. GPT-5 achieves the deepest task chains and highest recovery rate, while some smaller models show strong recovery despite moderate reasoning depth. This work provides a reproducible method for evaluating long-horizon reasoning and shows that contextual consistency and error recovery should be explicit objectives in future model development.
Recommended Citation
Schweim, Dirk and Ahmad, Shawez, "Evaluating Long-Horizon Reasoning and Self-Correction in Large Language Models with Verifiable Task Chains" (2026). PACIS 2026 Proceedings. 24.
https://aisel.aisnet.org/pacis2026/ai_ml/ai_ml/24
Evaluating Long-Horizon Reasoning and Self-Correction in Large Language Models with Verifiable Task Chains
Large Language Models (LLMs) are increasingly used in iterative workflows, yet common benchmarks mainly evaluate single-turn tasks and fail to measure performance in extended interactions. This article addresses this gap by introducing a benchmark based on programmatically verifiable Python task chains. In this framework, models iteratively extend and refactor a growing codebase across sequential steps, while unit tests evaluate correctness and trigger a single opportunity for self-correction after failures. Ten state-of-the-art models, including GPT-5 and Gemini 2.5 Pro, are evaluated. Results reveal substantial performance differences and show that sequential reasoning depth and error recovery are only partially correlated. GPT-5 achieves the deepest task chains and highest recovery rate, while some smaller models show strong recovery despite moderate reasoning depth. This work provides a reproducible method for evaluating long-horizon reasoning and shows that contextual consistency and error recovery should be explicit objectives in future model development.
Comments
01-AIML