Abstract

Online teaching evaluations have replaced in-class teaching evaluations as the new norm for assessing teaching performance among institutions of higher education. Faculty are reluctant to adopt online teaching evaluations and raise concerns regarding lower response rates compared to traditional in-class evaluations. A low response rate implies an underrepresentation of the target student population and threatens the validity of the evaluation. The literature comparing online with in-class evaluations focuses on the average evaluation ratings and concludes, with few exceptions, that the two methods produce statistically indifferent results. The reliability of an evaluation, however, is not addressed. This study employs Monte Carlo simulation to investigate the effects of various evaluation conditions, including response rates, class sizes, and observed evaluation scores, on evaluation accuracy. The complex distribution of evaluation ratings are carefully simulated based on a real dataset, and the effects of the investigated factors on evaluation accuracy are calculated and analyzed. Implications for assessing teaching performance are discussed.

Share

COinS
 

Can We Trust Teaching Evaluations When Response Rates are not High? Implications from a Monte Carlo Simulation

Online teaching evaluations have replaced in-class teaching evaluations as the new norm for assessing teaching performance among institutions of higher education. Faculty are reluctant to adopt online teaching evaluations and raise concerns regarding lower response rates compared to traditional in-class evaluations. A low response rate implies an underrepresentation of the target student population and threatens the validity of the evaluation. The literature comparing online with in-class evaluations focuses on the average evaluation ratings and concludes, with few exceptions, that the two methods produce statistically indifferent results. The reliability of an evaluation, however, is not addressed. This study employs Monte Carlo simulation to investigate the effects of various evaluation conditions, including response rates, class sizes, and observed evaluation scores, on evaluation accuracy. The complex distribution of evaluation ratings are carefully simulated based on a real dataset, and the effects of the investigated factors on evaluation accuracy are calculated and analyzed. Implications for assessing teaching performance are discussed.