Abstract
Multi-criteria decision support systems (DSS) rely on criteria weight elicitation, yet the available methods differ substantially in the input they require and the cognitive effort it imposes. The Analytic Hierarchy Process (AHP) requires a full pairwise comparison matrix, the Best-Worst Method (BWM) reduces this to two reference vectors, while Ranking Comparison (RANCOM) requires only an ordinal ranking. These formats differ in how much of a decision-maker's preference they can represent, raising a design question: how much weight-recovery accuracy is sacrificed by adopting a lighter input format? We address this with a large-scale Monte Carlo study in which reference weight vectors are sampled from a Dirichlet distribution, encoded into each method's input, and recovered with its solver. Because the simulated decision-maker is perfectly consistent, the experiment isolates the representational loss of each method rather than the error real experts introduce. We evaluate recovery with four complementary cardinal and ordinal metrics. AHP yields the lowest cardinal error and BWM the second lowest across the whole criteria range, while RANCOM reproduces the input ranking exactly, since it receives that ranking as input. The accuracy gap between methods narrows as the number of criteria grows, while the judgement count and input complexity required increase.
Paper Type
Short Paper
DOI
10.62036/ISD.2026.35
A Monte Carlo Study on the Trade-Off Between Cognitive Effort and Weight Recovery in AHP, BWM, and RANCOM: Implications for Decision Support Systems Design
Multi-criteria decision support systems (DSS) rely on criteria weight elicitation, yet the available methods differ substantially in the input they require and the cognitive effort it imposes. The Analytic Hierarchy Process (AHP) requires a full pairwise comparison matrix, the Best-Worst Method (BWM) reduces this to two reference vectors, while Ranking Comparison (RANCOM) requires only an ordinal ranking. These formats differ in how much of a decision-maker's preference they can represent, raising a design question: how much weight-recovery accuracy is sacrificed by adopting a lighter input format? We address this with a large-scale Monte Carlo study in which reference weight vectors are sampled from a Dirichlet distribution, encoded into each method's input, and recovered with its solver. Because the simulated decision-maker is perfectly consistent, the experiment isolates the representational loss of each method rather than the error real experts introduce. We evaluate recovery with four complementary cardinal and ordinal metrics. AHP yields the lowest cardinal error and BWM the second lowest across the whole criteria range, while RANCOM reproduces the input ranking exactly, since it receives that ranking as input. The accuracy gap between methods narrows as the number of criteria grows, while the judgement count and input complexity required increase.
Recommended Citation
Shekhovtsov, A. & Sałabun, W.(2026). A Monte Carlo Study on the Trade-Off Between Cognitive Effort and Weight Recovery in AHP, BWM, and RANCOM: Implications for Decision Support Systems Design. In M. Valenta, B. Mannová, R. Pergl, A. Przybylek, M. Lang, H. Linger, C. Schneider, N. Iivari, & E. Insfran (Eds.), Making ISD Sustainable: Reloaded with AI and Automation (ISD2026 Proceedings). Prague, Czech Republic: Czech Technical University in Prague. ISBN: 978-80-01-07585-2. https://doi.org/10.62036/ISD.2026.35