Paper Type
Complete
Paper Number
PACIS2026-2119
Description
Modern LLM platforms promise that analysts can "just chat" with diverse business documents. In practice, however, they often struggle to reliably interpret tables in complex files, combine information across sources, and carry out accurate calculations. Existing solutions typically support either document retrieval or mathematical processing, but rarely both in an integrated way. To address this gap, we follow a Design Science Research approach to develop and evaluate a modular LLM-based artifact for analytical question answering. The artifact processes complex document formats, including PDFs, presentations, and spreadsheets, and separates textual retrieval from table-based computation for higher reliability. We evaluate the system against a commercial LLM baseline and a single-index dense-vector RAG configuration using 100 real-world economic queries. The results show substantial accuracy gains for complex information retrieval and arithmetic tasks. This study contributes prescriptive design knowledge for building structure-aware AI artifacts that support more reliable automation of financial and analytical work.
Recommended Citation
Yu, Joe and Zhang, Yurui, "Designing an LLM Agent for Enterprise Question Answering over Heterogeneous Documents" (2026). PACIS 2026 Proceedings. 10.
https://aisel.aisnet.org/pacis2026/practioner/practioner/10
Designing an LLM Agent for Enterprise Question Answering over Heterogeneous Documents
Modern LLM platforms promise that analysts can "just chat" with diverse business documents. In practice, however, they often struggle to reliably interpret tables in complex files, combine information across sources, and carry out accurate calculations. Existing solutions typically support either document retrieval or mathematical processing, but rarely both in an integrated way. To address this gap, we follow a Design Science Research approach to develop and evaluate a modular LLM-based artifact for analytical question answering. The artifact processes complex document formats, including PDFs, presentations, and spreadsheets, and separates textual retrieval from table-based computation for higher reliability. We evaluate the system against a commercial LLM baseline and a single-index dense-vector RAG configuration using 100 real-world economic queries. The results show substantial accuracy gains for complex information retrieval and arithmetic tasks. This study contributes prescriptive design knowledge for building structure-aware AI artifacts that support more reliable automation of financial and analytical work.
Comments
16-Practitioner