Paper Type
Complete
Abstract
Decision support systems (DSS) often generate multi-criteria rankings recalculated from current data each cycle. Because internal reference points are reconstructed every period, ranking changes may reflect shifts in the measurement scale rather than real performance differences. We define this phenomenon as benchmark drift and examine how incorporating historical memory affects the stability–responsiveness trade-off in temporal evaluation systems. Following a design science research approach, we propose a history-aware benchmark mechanism that smooths ideal and anti-ideal reference vectors using exponentially weighted moving averages (EWMA) governed by a single parameter α. Implemented within a TOPSIS framework and evaluated on twelve years of Eurostat energy indicators for 27 EU countries, the mechanism shows that about 27% of ranking variation in the classical configuration is drift-driven. Moderate smoothing reduces this share by 25–35% while preserving sensitivity to genuine performance change, providing a reusable design component for longitudinal DSS evaluation.
Paper Number
1547
Recommended Citation
Kizielewicz, Bartłomiej; Wątróbski, Jarosław; Bączkiewicz, Aleksandra; and Sałabun, Wojciech, "Benchmark Drift in Temporal Decision Support Systems: A Design Science Approach to Stability and Responsiveness" (2026). AMCIS 2026 Proceedings. 15.
https://aisel.aisnet.org/amcis2026/sig_dsa/sig_dsa/15
Benchmark Drift in Temporal Decision Support Systems: A Design Science Approach to Stability and Responsiveness
Decision support systems (DSS) often generate multi-criteria rankings recalculated from current data each cycle. Because internal reference points are reconstructed every period, ranking changes may reflect shifts in the measurement scale rather than real performance differences. We define this phenomenon as benchmark drift and examine how incorporating historical memory affects the stability–responsiveness trade-off in temporal evaluation systems. Following a design science research approach, we propose a history-aware benchmark mechanism that smooths ideal and anti-ideal reference vectors using exponentially weighted moving averages (EWMA) governed by a single parameter α. Implemented within a TOPSIS framework and evaluated on twelve years of Eurostat energy indicators for 27 EU countries, the mechanism shows that about 27% of ranking variation in the classical configuration is drift-driven. Moderate smoothing reduces this share by 25–35% while preserving sensitivity to genuine performance change, providing a reusable design component for longitudinal DSS evaluation.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.
Comments
SIG DSA