Paper Number

1620

Paper Type

CRP

Abstract

High quality data is essential to the success of machine learning projects, especially for training, but also after deployment. Even slight differences between training and runtime data may degrade performance. Based on the application case of truck driver stress prediction, we collected physiological, activity, and driving data using an Apple Watch 7, heart rate data using an ECG and weather data from a web service. We experimentally evaluated the prediction performance of increasing time-shifts applied to our data sources. Such problems are known as Out-of-Distribution situations. In this paper, we showcase how developers can approach such problems and perform analyses to identify features highly prone to Out-of-Distribution issues. These results are central to quality assurance for successful Machine Learning projects. We also propose Data Robustness Stories to document Out-of-Distribution issues.

Share

COinS
 
Jun 14th, 12:00 AM

Time-Shift Robustness Evaluation for Applications using Artificial Intelligence

High quality data is essential to the success of machine learning projects, especially for training, but also after deployment. Even slight differences between training and runtime data may degrade performance. Based on the application case of truck driver stress prediction, we collected physiological, activity, and driving data using an Apple Watch 7, heart rate data using an ECG and weather data from a web service. We experimentally evaluated the prediction performance of increasing time-shifts applied to our data sources. Such problems are known as Out-of-Distribution situations. In this paper, we showcase how developers can approach such problems and perform analyses to identify features highly prone to Out-of-Distribution issues. These results are central to quality assurance for successful Machine Learning projects. We also propose Data Robustness Stories to document Out-of-Distribution issues.

When commenting on articles, please be friendly, welcoming, respectful and abide by the AIS eLibrary Discussion Thread Code of Conduct posted here.