Paper Number
ICIS2026-3333
Paper Type
Completed
Abstract
Organizations increasingly use large language models (LLMs) for decisions that depend on organizational knowledge that is partly non-codified or tacit. This creates a knowledge gap between the LLM and the organization, because textual context conveys only a subset of what past organizational decisions actually drew on. We propose the Contextualized Alignment Lens Model (CALM) to provide a quantitative estimate of the alignment to the application context. CALM captures decision policies, localizes misalignment at the cue level, supports targeted prompt-based interventions, and bounds the contribution of tacit knowledge to a given decision. We demonstrate these affordances for ten LLMs making 30,000 decisions on European Court of Human Rights Article 6 cases. CALM addresses AI alignment as an organization-specific measurement problem and gives IS researchers and developers a tool to evaluate and improve the fit between LLM decisions and their sociotechnical context.
Recommended Citation
Weller, Niklas and Blohm, Ivo, "Measuring Contextualized AI Alignment in Organizational Decision-Making" (2026). ICIS 2026 Proceedings. 6.
https://aisel.aisnet.org/icis2026/design_pm/design_pm/6
Measuring Contextualized AI Alignment in Organizational Decision-Making
Organizations increasingly use large language models (LLMs) for decisions that depend on organizational knowledge that is partly non-codified or tacit. This creates a knowledge gap between the LLM and the organization, because textual context conveys only a subset of what past organizational decisions actually drew on. We propose the Contextualized Alignment Lens Model (CALM) to provide a quantitative estimate of the alignment to the application context. CALM captures decision policies, localizes misalignment at the cue level, supports targeted prompt-based interventions, and bounds the contribution of tacit knowledge to a given decision. We demonstrate these affordances for ten LLMs making 30,000 decisions on European Court of Human Rights Article 6 cases. CALM addresses AI alignment as an organization-specific measurement problem and gives IS researchers and developers a tool to evaluate and improve the fit between LLM decisions and their sociotechnical context.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
11_Design_development_PM