Abstract

A controlled laboratory environment is proposed for evaluating classical and machine-learning-based forecasting algorithms. The paper argues that conclusions drawn from real-world data may be systematically biased because the true data-generating structure is unknown. The study therefore constructs 15,768 reference time series with controlled length, noise, trend, seasonality and integer-valued structure. Eight forecasting methods are compared using RMSE and within-series rankings. Linear regression and ETS achieve the best overall accuracy, while Random Forest and XGBoost are competitive only in specific structural settings. The results show that algorithm selection should depend on identified demand characteristics rather than on aggregate benchmark accuracy alone.

Recommended Citation

Bartkowiak, M., Cyplik, P., Karolewski, A. & Adamczak, M.(2026). Reference Time Series as a Controlled Laboratory Environment for Testing Classical and Machine Learning Forecasting Algorithms. In M. Valenta, B. Mannová, R. Pergl, A. Przybylek, M. Lang, H. Linger, C. Schneider, N. Iivari, & E. Insfran (Eds.), Making ISD Sustainable: Reloaded with AI and Automation (ISD2026 Proceedings). Prague, Czech Republic: Czech Technical University in Prague. ISBN: 978-80-01-07585-2. https://doi.org/10.62036/ISD.2026.91

Paper Type

Poster

DOI

10.62036/ISD.2026.91

Download

Share

COinS
 

Reference Time Series as a Controlled Laboratory Environment for Testing Classical and Machine Learning Forecasting Algorithms

A controlled laboratory environment is proposed for evaluating classical and machine-learning-based forecasting algorithms. The paper argues that conclusions drawn from real-world data may be systematically biased because the true data-generating structure is unknown. The study therefore constructs 15,768 reference time series with controlled length, noise, trend, seasonality and integer-valued structure. Eight forecasting methods are compared using RMSE and within-series rankings. Linear regression and ETS achieve the best overall accuracy, while Random Forest and XGBoost are competitive only in specific structural settings. The results show that algorithm selection should depend on identified demand characteristics rather than on aggregate benchmark accuracy alone.