Paper Type
ERF
Abstract
Conversational agents play a crucial role in automating customer service interactions, assisting users in handling inquiries, processing requests, and managing various tasks. While large language models like GPT-4 and Llama 2 have improved conversational capabilities, challenges remain in tracking context and maintaining coherence in complex, multi-turn dialogues, especially in specialized technical domains. Their effectiveness in non-English, domain-specific customer service contexts also remains underexplored. This study investigates Llama 2’s ability to manage multi-turn dialogues in the French-language automobile insurance sector. Using the task-technology fit framework, we assess its alignment with customer service requirements, focusing on context tracking, coherence, and task completion. The model is fine-tuned on real French-language interactions and benchmarked against human agents. Contributions include insights into Llama 2’s strengths and limitations in complex dialogues, strategies for improving context-aware conversational AI, and refinement of the task-technology fit framework for conversational systems.
Paper Number
1747
Recommended Citation
Essaied, Khadija; Mellouli, Sehl; and Khoury, Richard, "Context Tracking with Llama 2: A Task-Technology Fit Analysis for Enhancing Customer Service conversations" (2025). AMCIS 2025 Proceedings. 17.
https://aisel.aisnet.org/amcis2025/sig_aiaa/sig_aiaa/17
Context Tracking with Llama 2: A Task-Technology Fit Analysis for Enhancing Customer Service conversations
Conversational agents play a crucial role in automating customer service interactions, assisting users in handling inquiries, processing requests, and managing various tasks. While large language models like GPT-4 and Llama 2 have improved conversational capabilities, challenges remain in tracking context and maintaining coherence in complex, multi-turn dialogues, especially in specialized technical domains. Their effectiveness in non-English, domain-specific customer service contexts also remains underexplored. This study investigates Llama 2’s ability to manage multi-turn dialogues in the French-language automobile insurance sector. Using the task-technology fit framework, we assess its alignment with customer service requirements, focusing on context tracking, coherence, and task completion. The model is fine-tuned on real French-language interactions and benchmarked against human agents. Contributions include insights into Llama 2’s strengths and limitations in complex dialogues, strategies for improving context-aware conversational AI, and refinement of the task-technology fit framework for conversational systems.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIGAIAA