Paper Number
1620
Paper Type
Complete Research Paper
Abstract
High quality data is essential to the success of machine learning projects, especially for training, but also after deployment. Even slight differences between training and runtime data may degrade performance. Based on the application case of truck driver stress prediction, we collected physiological, activity, and driving data using an Apple Watch 7, heart rate data using an ECG and weather data from a web service. We experimentally evaluated the prediction performance of increasing time-shifts applied to our data sources. Such problems are known as Out-of-Distribution situations. In this paper, we showcase how developers can approach such problems and perform analyses to identify features highly prone to Out-of-Distribution issues. These results are central to quality assurance for successful Machine Learning projects. We also propose Data Robustness Stories to document Out-of-Distribution issues.
Recommended Citation
Severin, Benedikt; Keil, Maria; Meiser, Arnd; Straub, Sarah M.; Ruiner, Caroline; Hagemann, Vera; Klumpp, Matthias; and Hesenius, Marc, "Time-Shift Robustness Evaluation for Applications Using Artificial Intelligence" (2024). ECIS 2024 Proceedings. 2.
https://aisel.aisnet.org/ecis2024/track03_ai/track03_ai/2
Time-Shift Robustness Evaluation for Applications Using Artificial Intelligence
High quality data is essential to the success of machine learning projects, especially for training, but also after deployment. Even slight differences between training and runtime data may degrade performance. Based on the application case of truck driver stress prediction, we collected physiological, activity, and driving data using an Apple Watch 7, heart rate data using an ECG and weather data from a web service. We experimentally evaluated the prediction performance of increasing time-shifts applied to our data sources. Such problems are known as Out-of-Distribution situations. In this paper, we showcase how developers can approach such problems and perform analyses to identify features highly prone to Out-of-Distribution issues. These results are central to quality assurance for successful Machine Learning projects. We also propose Data Robustness Stories to document Out-of-Distribution issues.
When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.