More and more people are reading health-related news stories from the internet due to ease of access. However, not all health-related news on the internet is reliable and making decisions based on unreliable health-related news can have detrimental effects on people’s health. Evaluating health-related news stories from the internet is difficult for two reasons: (1) there is an abundance of news stories, which is exacerbated by social media, and (2) evaluating health-related information requires expertise in the medical field, which most people do not have. In this study, we combine text analysis and machine learning to develop a model that evaluates health-related news stories based on evaluation criteria developed by healthcare professionals. The results of this study can be used in the implementation of reliability badges on health-related news stories, which could be applicable in the context of social media and may help people identify unreliable health-related news more easily.