Paper Type
Complete
Paper Number
PACIS2026-1974
Description
LLM-based agents are reshaping software engineering, yet empirical evidence on their practical effects remains limited. This study provides a controlled comparison between an IDE-integrated assistant, GitHub Copilot Ask, and an LLM-based agent, GitHub Copilot Agent, during brownfield onboarding tasks. In an experiment with 24 developers, we measured productivity, perceived workload, interaction patterns, and prompt behaviour using the SPACE framework. Copilot Agent reduced mean task completion time by 61.7% and NASA-TLX workload by 57.4%, while code correctness did not significantly improve. However, interaction data show a shift from active collaboration to passive supervision, raising concerns about over-reliance and skill erosion.
Recommended Citation
Appelt, Carmen Laura Janina and Glauben, Adrian, "From Assistants to Agents: Exploring Efficiency and Human Agency in AI-Supported Programming" (2026). PACIS 2026 Proceedings. 13.
https://aisel.aisnet.org/pacis2026/ai_fow/ai_fow/13
From Assistants to Agents: Exploring Efficiency and Human Agency in AI-Supported Programming
LLM-based agents are reshaping software engineering, yet empirical evidence on their practical effects remains limited. This study provides a controlled comparison between an IDE-integrated assistant, GitHub Copilot Ask, and an LLM-based agent, GitHub Copilot Agent, during brownfield onboarding tasks. In an experiment with 24 developers, we measured productivity, perceived workload, interaction patterns, and prompt behaviour using the SPACE framework. Copilot Agent reduced mean task completion time by 61.7% and NASA-TLX workload by 57.4%, while code correctness did not significantly improve. However, interaction data show a shift from active collaboration to passive supervision, raising concerns about over-reliance and skill erosion.
Comments
02-FutureofWork