Paper Type

Complete

Abstract

Human-in-the-loop has become the near-universal response to generative AI risk management. Yet organizations struggle to operationalize it. Drawing on qualitative analysis of GenAI implementations across eight organizations spanning regulated and non-regulated settings, we conceptualize HITL as a governance control architecture that allocates decision rights, accountability, and intervention points across a human-AI work system. We identify three recurring oversight configurations: (1) AI Recommendation, where end users verify outputs; (2) AI-Supported Work, where humans perform the core work while AI contributes within structured checkpoints; and (3) AI-Led Workflows, where AI executes sequential steps and humans supervise at gates and handle exceptions. We develop a use-case contingency framework linking six task characteristics to the oversight configuration organizations view as workable. The framework explains why uniform oversight fails and describes common failure modes when oversight is misfit to task properties.

Paper Number

1728

Comments

SIG OSRA

Download

Share

COinS
 
Aug 15th, 12:00 AM

Beyond the Principle: How Organizations Implement Human-in-the-Loop Oversight for Generative AI

Human-in-the-loop has become the near-universal response to generative AI risk management. Yet organizations struggle to operationalize it. Drawing on qualitative analysis of GenAI implementations across eight organizations spanning regulated and non-regulated settings, we conceptualize HITL as a governance control architecture that allocates decision rights, accountability, and intervention points across a human-AI work system. We identify three recurring oversight configurations: (1) AI Recommendation, where end users verify outputs; (2) AI-Supported Work, where humans perform the core work while AI contributes within structured checkpoints; and (3) AI-Led Workflows, where AI executes sequential steps and humans supervise at gates and handle exceptions. We develop a use-case contingency framework linking six task characteristics to the oversight configuration organizations view as workable. The framework explains why uniform oversight fails and describes common failure modes when oversight is misfit to task properties.