Incorporating Domain-Specific Neural Machine Translation into Augmented Reality Systems

Authors

Krzysztof Wołk, Polish-Japanese Academy of Information TechnologyFollow

Abstract

Neural machine translation (NMT) systems can improve augmented reality (AR) systems. For example, a smartphone display can translate and text-augment information for travelers, assist with translation of signs, restaurant menus, medical documents, etc., and facilitate real-time translation of foreign-language lectures for students. However, NMT has been neglected in AR research because of its quality and performance requirements. General-purpose translation services, e.g., Google Translate, are used in some AR systems, but they depend on Internet connections and are not well-tuned to providing high-quality translation in specific domains. We propose a laboratory tool prototype that classifies phone camera images, recognizes and translates image text, and augments camera images with translated text in real time. Text from various domains is translated via NMT based on recurrent and convolutional neural networks—with and without subword units. According to bilingual evaluation understudy metrics, the results are very promising, which encourages future applications.

Recommended Citation

Wołk, Krzysztof, "Incorporating Domain-Specific Neural Machine Translation into Augmented Reality Systems" (2020). PACIS 2020 Proceedings. 14.
https://aisel.aisnet.org/pacis2020/14