Paper Type
Complete
Paper Number
PACIS2025-1523
Description
Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their performance remains limited when addressing complex real-world problems that demand diverse expertise. Existing approaches to leveraging multiple LLMs face critical limitations, including restricted generalizability, ineffective collaboration, or underutilization of iterative refinement capabilities. To address these challenges, this paper proposes a novel framework, Wisdom Integration for LLM Crowds (WILC), which enables fine-grained query-level collaboration between LLMs through multi-round reflective dialogues. WILC introduces two key innovations: a multi-round reflective dialogue mechanism that iteratively refines solutions, and a contextual bandits approach for dynamically matching the most suitable LLM to each query based on evolving capability requirements. By combining these innovations, WILC achieves adaptive capability complementarity across LLMs without task-specific supervised learning. Extensive experiments on three challenging benchmarks demonstrate that WILC significantly outperforms individual LLMs and existing methods, establishing a new paradigm for effectively harnessing the collective wisdom of LLM crowds.
Recommended Citation
Fang, Yanbin; Wei, Xuan; and Chen, Wei, "WILC: A Wisdom Integration Framework for LLM Crowds" (2025). PACIS 2025 Proceedings. 12.
https://aisel.aisnet.org/pacis2025/aiandml/aiandml/12
WILC: A Wisdom Integration Framework for LLM Crowds
Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their performance remains limited when addressing complex real-world problems that demand diverse expertise. Existing approaches to leveraging multiple LLMs face critical limitations, including restricted generalizability, ineffective collaboration, or underutilization of iterative refinement capabilities. To address these challenges, this paper proposes a novel framework, Wisdom Integration for LLM Crowds (WILC), which enables fine-grained query-level collaboration between LLMs through multi-round reflective dialogues. WILC introduces two key innovations: a multi-round reflective dialogue mechanism that iteratively refines solutions, and a contextual bandits approach for dynamically matching the most suitable LLM to each query based on evolving capability requirements. By combining these innovations, WILC achieves adaptive capability complementarity across LLMs without task-specific supervised learning. Extensive experiments on three challenging benchmarks demonstrate that WILC significantly outperforms individual LLMs and existing methods, establishing a new paradigm for effectively harnessing the collective wisdom of LLM crowds.
Comments
AI ML