Smart retailing technologies are transforming retailers’ operations, consumer shopping experiences, and perceptions about retailers. These technologies are becoming increasingly numerous and may have different interfaces and ways of implementation. Based on the literature review, this study develops a framework to measure the impacts of implementing non-store-based and store-based smart retailing technology pilot projects, with appropriate metrics based on brand leadership, market performance and word-of-mouth. The proposed method is based on a survey distributed to a sample of customers and control groups, using an experimental Latin square design that controls the effects of extraneous variables. With this methodology, different interfaces can be evaluated in the non-store-based pilot projects, and different locations in the store-based pilot projects implementation, controlling extraneous variables such as seasonality, customer profiles and store-specific characteristics. Finally, the academic and managerial contributions to retailers, consumers and technology companies are highlighted, as well as limitations and avenues for future research.