Location
Hilton Hawaiian Village, Honolulu, Hawaii
Event Website
https://hicss.hawaii.edu/
Start Date
3-1-2024 12:00 AM
End Date
6-1-2024 12:00 AM
Description
As deep neural networks (DNNs) are increasingly used in practical settings, developers often adopt pre-existing DNN architectures from a vast collection of well-established models. However, in industrial environments, factors beyond simply achieving high accuracy are becoming important. The runtime performance is critical, as delays can lead to conveyor stops or equipment damage. Currently, determining the runtime performance of a DNN requires multiple iterations and testing specific configurations, whereas existing methods for benchmarking DNNs mainly compare different hardware or runtime parameters. We present tritonPerf, an approach to obtain and compare the runtime (latency and throughput) for a broad range of settings and existing DNN architectures in the final application environment. It allows data scientists to evaluate the performance and compare the results for a wide range of models before time and resource-intensive hyperparameter tuning is performed. We demonstrate the gain of tritonPerf in an extensive field study using an industrial setting, where we benchmark and compare the runtime of 57 different models.
Recommended Citation
Völter, Constantin; Koppe, Timo; and Rieger, Phillip, "Don't Buy the Pig in a Poke: Benchmarking DNNs Inference Performance before Development" (2024). Hawaii International Conference on System Sciences 2024 (HICSS-57). 11.
https://aisel.aisnet.org/hicss-57/st/sw_development/11
Don't Buy the Pig in a Poke: Benchmarking DNNs Inference Performance before Development
Hilton Hawaiian Village, Honolulu, Hawaii
As deep neural networks (DNNs) are increasingly used in practical settings, developers often adopt pre-existing DNN architectures from a vast collection of well-established models. However, in industrial environments, factors beyond simply achieving high accuracy are becoming important. The runtime performance is critical, as delays can lead to conveyor stops or equipment damage. Currently, determining the runtime performance of a DNN requires multiple iterations and testing specific configurations, whereas existing methods for benchmarking DNNs mainly compare different hardware or runtime parameters. We present tritonPerf, an approach to obtain and compare the runtime (latency and throughput) for a broad range of settings and existing DNN architectures in the final application environment. It allows data scientists to evaluate the performance and compare the results for a wide range of models before time and resource-intensive hyperparameter tuning is performed. We demonstrate the gain of tritonPerf in an extensive field study using an industrial setting, where we benchmark and compare the runtime of 57 different models.
https://aisel.aisnet.org/hicss-57/st/sw_development/11