Proactively analyzing patient pathways can help healthcare providers to anticipate treatment-related risks, detect undesired outcomes, and allocate resources quickly. For this purpose, modern methods from the field of predictive business process monitoring can be applied to create data-driven models that capture patterns from past behavior to provide predictions about running process instances. Recent methods increasingly focus on deep neural networks (DNN) due to their superior prediction performances and their independence from process knowledge. However, DNNs generally have the disadvantage of showing black-box characteristics, which hampers the dissemination in critical environments such as healthcare. To this end, we propose the design of HIXPred, a novel artifact combining predictive power with explainable results for patient pathway predictions. We instantiate HIXPred and apply it to a real-life healthcare use case for evaluation and demonstration purposes and conduct interviews with medical experts. Our results confirm high predictive performance while ensuring sufficient interpretability and explainability to provide comprehensible decision support.