Paper Type
ERF
Abstract
AI-based conversational agents are increasingly deployed to support mental health. However, despite their rapid adoption, there is no systematic, theory-grounded way to evaluate the quality of human–AI interaction. Most existing approaches rely on user satisfaction, sentiment analysis, or engagement metrics, which fail to capture the relational dynamics central to effective support (Mohr et al., 2017). Drawing on therapeutic alliance theory from psychotherapy research (Bordin, 1979; Horvath et al., 2011), this study proposes a novel interaction-level construct, alliance variance, to evaluate AI-mediated mental health conversations. Alliance variance models moment-to-moment fluctuations in emotional attunement, goal alignment, and interactional coordination across conversational turns. Using de-identified psychiatrist–patient session transcripts, we outline a computational framework for operationalizing alliance dynamics and describe an early-stage agentic AI prototype that monitors these signals during interaction. This work contributes to HCI research by introducing a theory-informed, dynamic measure of interaction quality for conversational systems in sensitive domains.
Paper Number
1466
Recommended Citation
Zhu, Bin; Rupani, Yash Vijaykumar; and Chen, Cyrus, "Toward Alliance-Aware Evaluation of AI-Mediated Mental Health Interactions" (2026). AMCIS 2026 Proceedings. 10.
https://aisel.aisnet.org/amcis2026/sig_hci/sig_hci/10
Toward Alliance-Aware Evaluation of AI-Mediated Mental Health Interactions
AI-based conversational agents are increasingly deployed to support mental health. However, despite their rapid adoption, there is no systematic, theory-grounded way to evaluate the quality of human–AI interaction. Most existing approaches rely on user satisfaction, sentiment analysis, or engagement metrics, which fail to capture the relational dynamics central to effective support (Mohr et al., 2017). Drawing on therapeutic alliance theory from psychotherapy research (Bordin, 1979; Horvath et al., 2011), this study proposes a novel interaction-level construct, alliance variance, to evaluate AI-mediated mental health conversations. Alliance variance models moment-to-moment fluctuations in emotional attunement, goal alignment, and interactional coordination across conversational turns. Using de-identified psychiatrist–patient session transcripts, we outline a computational framework for operationalizing alliance dynamics and describe an early-stage agentic AI prototype that monitors these signals during interaction. This work contributes to HCI research by introducing a theory-informed, dynamic measure of interaction quality for conversational systems in sensitive domains.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG HCI