Open Data: Evaluating Topic Modeling Interpretability Using Topic Labeled Gold Standard Sets

Authors

Biagio Palese, Northern Illinois UniversityFollow
Gabriele Piccoli, Louisiana State University at Baton RougeFollow

Abstract

This paper provides the gold standard sets described and used by Palese and Piccoli (2020) which advance a scalable method, Human Interpretable Topics (HIT), for assessing the interpretability of topic modeling results. Gold standard set are a small collection of documents extracted from a dataset and manually labeled by humans. The role of such a gold standard dataset is to establish a “ground truth” against which researchers can benchmark the results generated by algorithmic topic models. We first provide a detailed description of the classification procedure used to create the labeled sets. Then, we make available descriptive statistics of the three different gold standard sets named respectively: inclusive, full agreement and partial agreement. These gold standard sets can be used to benchmark different/new models built in research analyzing online customers’ reviews in the context of the lodging industry. We hope a larger number of researchers will follow our example and use the AIS Transactions on Replication Research journal to share open access gold standard sets in different areas of interest.

Recommended Citation

Palese, Biagio and Piccoli, Gabriele (2020) "Open Data: Evaluating Topic Modeling Interpretability Using Topic Labeled Gold Standard Sets," AIS Transactions on Replication Research: Vol. 6, Article 9.
DOI: 10.17705/1atrr.00052
Available at: https://aisel.aisnet.org/trr/vol6/iss1/9

DOI

10.17705/1atrr.00052