Paper Number
ICIS2026-3333
Paper Type
Completed
Abstract
Organizations increasingly use large language models (LLMs) for decisions that depend on organizational knowledge that is partly non-codified or tacit. This creates a knowledge gap between the LLM and the organization, because textual context conveys only a subset of what past organizational decisions actually drew on. We propose the Contextualized Alignment Lens Model (CALM) to provide a quantitative estimate of the alignment to the application context. CALM captures decision policies, localizes misalignment at the cue level, supports targeted prompt-based interventions, and bounds the contribution of tacit knowledge to a given decision. We demonstrate these affordances for ten LLMs making 30,000 decisions on European Court of Human Rights Article 6 cases. CALM addresses AI alignment as an organization-specific measurement problem and gives IS researchers and developers a tool to evaluate and improve the fit between LLM decisions and their sociotechnical context.
Recommended Citation
Weller, Niklas and Blohm, Ivo, "Measuring Contextualized AI Alignment in Organizational Decision-Making" (2026). ICIS 2026 Proceedings. 6.
https://aisel.aisnet.org/icis2026/design_pm/design_pm/6
Measuring Contextualized AI Alignment in Organizational Decision-Making
Organizations increasingly use large language models (LLMs) for decisions that depend on organizational knowledge that is partly non-codified or tacit. This creates a knowledge gap between the LLM and the organization, because textual context conveys only a subset of what past organizational decisions actually drew on. We propose the Contextualized Alignment Lens Model (CALM) to provide a quantitative estimate of the alignment to the application context. CALM captures decision policies, localizes misalignment at the cue level, supports targeted prompt-based interventions, and bounds the contribution of tacit knowledge to a given decision. We demonstrate these affordances for ten LLMs making 30,000 decisions on European Court of Human Rights Article 6 cases. CALM addresses AI alignment as an organization-specific measurement problem and gives IS researchers and developers a tool to evaluate and improve the fit between LLM decisions and their sociotechnical context.
Comments
11_Design_development_PM