Paper Type
ERF
Abstract
Artificial intelligence (AI) retrieval systems increasingly support high-stakes decision-making in domains such as law enforcement, healthcare, and finance. Governance frameworks often view bias as a system-level metric. However, AI retrieval systems operate as staged infrastructures, where stages jointly shape the outcomes. This study examines how demographic bias manifests across three technical stages of AI retrieval systems: representation, retrieval, and decision ranking. Using an empirical study in a law enforcement context, we measure racial bias at each stage and demonstrate that bias metrics vary depending on where in the pipeline they are assessed. We argue that system-level auditing alone is insufficient for governance of AI-driven decision systems in high-stakes domains. Instead, we propose stage-aligned governance mechanisms for each stage of the AI retrieval system. Our findings highlight the need for stage-level accountability in high-stakes AI deployments.
Paper Number
1260
Recommended Citation
Dutta, Archan and Kanungo, Vyanktesh, "Governing Demographic Bias Across AI Retrieval Systems" (2026). AMCIS 2026 Proceedings. 5.
https://aisel.aisnet.org/amcis2026/sig_dsa/sig_dsa/5
Governing Demographic Bias Across AI Retrieval Systems
Artificial intelligence (AI) retrieval systems increasingly support high-stakes decision-making in domains such as law enforcement, healthcare, and finance. Governance frameworks often view bias as a system-level metric. However, AI retrieval systems operate as staged infrastructures, where stages jointly shape the outcomes. This study examines how demographic bias manifests across three technical stages of AI retrieval systems: representation, retrieval, and decision ranking. Using an empirical study in a law enforcement context, we measure racial bias at each stage and demonstrate that bias metrics vary depending on where in the pipeline they are assessed. We argue that system-level auditing alone is insufficient for governance of AI-driven decision systems in high-stakes domains. Instead, we propose stage-aligned governance mechanisms for each stage of the AI retrieval system. Our findings highlight the need for stage-level accountability in high-stakes AI deployments.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG DSA