Generative AI for Intelligence Augmentation: Categorization and Evaluation Frameworks for Large Language Model Adaptation

Authors

Jie Tao, Fairfield UniversityFollow
Lina Zhou, University of North Carolina at CharlotteFollow
Xing Fang, College of Applied Science & Technology, Illinois State UniversityFollow

Abstract

Generative AI (GenAI) has transformed how businesses operate and innovate and how individuals learn, live, and work. Large language models (LLMs), a specific type of GenAI, focus on generating human-like text based on user instructions. Like other types of GenAI, LLMs have received wide recognition for their potential to augment human intelligence, but several challenges hinder efforts to realize their full potential in practice. Some notable challenges include not adequately exploring LLM applications beyond chatbots and/or text generation, the difficulty in categorizing various LLM adaptation strategies (particularly regarding human interactions), and the lack of a reference framework for evaluating and selecting LLM adaptation strategies from a human-centered perspective. To address these challenges, we propose a categorization framework for LLM adaptation that features two human-centered dimensions and stage LLM adaptation with respect to when it interacts with human intelligence. Additionally, we introduce an evaluation framework that incorporates a human-centered perspective that goes beyond the common machine-centered measures. Our empirical investigations, in which we use text classification as use cases, not only demonstrate the application of these frameworks but also compare various adaptation strategies. These artifacts and findings provide fresh insights and practical recommendations for selecting effective adaptation strategies to improve the efficacy of LLMs for intelligence augmentation. We further identify future research issues to address current limitations and suggest improvements for the proposed frameworks.

DOI

10.17705/1thci.00210

Recommended Citation

Tao, J., Zhou, L., & Fang, X. (2024). Generative AI for Intelligence Augmentation: Categorization and Evaluation Frameworks for Large Language Model Adaptation. AIS Transactions on Human-Computer Interaction, 16(3), 364-387. https://doi.org/10.17705/1thci.00210
DOI: 10.17705/1thci.00210