Against the backdrop of ubiquitous computing, companies from various industries are building up everincreasing amounts of business process data. Seeking to salvage these hidden “data treasures,” the need for analytical information systems is ever-growing to guide corporate decision-making. However, information systems research is still very much focused on static, explanatory modeling provided by business intelligence suites instead of embracing the opportunities offered by predictive analytics. Describing insights from a real-world manufacturing scenario, we seek to enhance the understanding of predictive modeling. In particular, we highlight that simply dumping data into “smart” algorithms is not a silver bullet. Rather, successful analytics projects require constant refinement and consolidation. To this end, we provide guidelines and best practices for modeling, feature engineering and interpretation leveraging tools from business information systems as well as machine learning.