Augmented Reality to Navigate Foreign Language Environments Thomas Richard Schalow University of Marketing and Distribution Sciences, Kobe Abstract Augmented reality is perhaps less well known than virtual reality, but in the near- term it promises to transform our world, and the tourist industry, in more and more useful ways than the artiicial digital environments that have become known to people through movies such as Avatar. In the context of tourism, augmented reality technologies allow real-time translation of signs and other printed, text-based information from the language of the foreign country to the language of the traveler, without the need to actually change the environment, or enhance the knowledge base of the traveler. Merely viewing the world through something known as an augmented reality browser, most likely through a device such as a smartphone, will magically transform the sign and its text, the world, and theen- tire tourist experience. Keywords Augmented reality; foreign languages; translation; smartphones 1. Introduction: the problem of foreign languages for the tourist industry The tremendous expansion in tourism in the past two decades, in line with declining costs for travel, has allowed more people to travel who lack the advanced linguistic skills necessary to optimally navigate foreign environments. This is par- ticularly true when travelers have only basic levels of education, and especially so when Asians travel to Europe and Europeans travel to Asia. Travelers from China, for example, though capable of reading signs using what many in the West perceive to be dificult Chinese ideograms, could be entirely bafled by signs using common Western languages such as French or English. In the same manner, travelers from places such as Europe and America are now commonly visiting tourist destinations in Asia and elsewhere, where their linguistic skills are severely challenged by written languages that are quite different in form from the languages they are familiar with. Although English has become a lingua franca for much of the tourist industry, English and other Western European languag- es are still not yet a part of the everyday environment for more than perhaps a quarter of the worlds population. Consequently, attempts by travelers to achieve an authen- tic experience in a radically foreign environment will inevitably be constrained by the limited linguistic access afforded to non-natives. This dificulty is especially felt when alphabets are non-Latin in their form, or in environments where pictographs 962 are used in place of alphabets. The linguistic cues found on signs and other textual information sources in these foreign environments are hidden from the traveler as a result of language differences, and threaten to make the tourism experience dificult or even potentially dangerous. Technology has stepped in to help with these problems, offering translation devices, electronic dictionaries, and the like, but problems still persist. Electronic dictionaries, for example, are of little use when there is no alphabet to facilitate navigation, as is the case for logographic/pictographic languages such as Chinese or Japanese, or in the case where alphabets are non-Latin in form, such as for Thai or Korean. What is needed is a tool to magically transform the environment into one that is linguistically familiar to the traveler, without the need to actually alter the environment from the perspective of the native. In this paper we will look at how augmented reality (AR) is making the whole world accessible to any traveler, regard- less of native language or foreign language experience. For practical reasons, this paper will focus only on the problems faced by travel- ers dealing with text or pictographic-based information sources. It is recognized that a great deal of information is available only through verbal communication, and this information will remain unavailable to the tourist as long as we lack real-time universal language translation software. Nonetheless, AR applications focusing only on text or pictographs will be of great utility to travelers. Since these AR applica- tions are either available today or will be available in the near future, they will have a more immediate impact on the travel experience than real-time verbal language translation, which yet needs to overcome formidable technological challenges before it is achieved. 2. An introduction to augmented reality Augmented reality is perhaps less well known than virtual reality (VR), but in the near term it promises to transform our world in more and more useful ways than the artiicial digital and virtual reality environments that have become known to peo- ple through movies such as Avatar. Augmented reality, a term irst used in the early 1990s by two engineers at Boeing named Caudell and Mizell, does not create a to- tally new and artiicial reality. Instead, it merely enhances the reality we experience in our everyday lives, overlaying descriptions, instructions, or explanations on top of the environment we perceive through our normal senses, such as sight. This is most often achieved by viewing the augmented world through the video display of a smartphone. In the most technologically advanced forms of AR, the actual environ- ment itself is not changed. Only our perception of that environment is altered, in a way that makes it more understandable or richer in context. In order to better understand AR technology, I would ask you to consider a bot- tle of medicine sitting on a shelf in a Japanese pharmacy. A traveler visiting Japan on a ten-day vacation has had some problems with the food and developed an upset Thomas Richard Schalow Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign Language Environments 963 stomach in a remote area of the country. While not life threatening or requiring a visit to a doctor, the visitor would appreciate some immediate relief and is now looking at medicines in the small towns only pharmacy. The traveler speaks no Japanese and the only available sales clerk speaks no English. The traveler has noticed some packages in the pharmacy with pictures of what appear to be a human stomach, but there are no products with familiar names to choose from, and the available Japa- nese products have only Japanese characters printed on the packages. Fortunately, the traveler is carrying a smartphone and has an Internet connection through the telephone network. Illustration 1: This is a Japanese medicine for an upset stomach The traveler points a smartphone camera at the products on the pharmacy shelves and AR technology begins to work. The AR technology irst establishes that there is a barcode on the packages. It has found a marker to work with. The AR tech- nology then also establishes the location of the pharmacy, as provided by a gravi- metric device or GPS signal and the smartphones ability to make use of this signal. Finally, the smartphone camera notices there are objects, the Japanese text, on the package, and using OCR software is able to read the information. As a result, due to the marker, the visitor knows the name of the product and any other informa- tion encoded in the barcode. With smarter barcodes, even information such as the intended use of the product could be provided. The visitor also learns, as a result of information provided by querying a databases based on the location of the phar- macy, that a trained pharmacist named Nakano Tomoko, who is able to speak some English and thus might be able to help our traveler, works in the pharmacy from noon to 5 p.m. everyday. Finally, the OCR software within the camera has translated the objects, or Japanese text, on the package into Italian, the native language of the traveler, and our traveler is now able to identify a Japanese versionof the medicine being sought. All of this AR magic is made possible as a result of something known as an augmented reality browser, similar to the browsers used on personal computers, to navigate the world through the lens of the smartphone camera.The video feed pro- 964 vided by the smartphone, combined with a network Internet connection to an online database of information, transforms the environment seen by the traveler through the browser into a world illed with relevant information, in the native language of the traveler. Using the markers, location, and/or objects we have just mentioned, the AR browser is able to generate the appropriate information to overlay on the environ- ment the traveler sees through the smartphone camera. Illustration 2: This is an augmented reality browser window seen through a smart- phone camera (Source: Wikipedia) 3. Aumented reality technology with markers, location ad objects Markers represent the least technologically advanced means for implementing AR. A marker is a special type of object, such as a barcode, that is consciously added to the environment in order to provide information. In most cases it requires that the environment be altered by actual physical human intervention, since barcodes and other common forms of markers are not usually embedded on anything other than products. One might imagine, for example, a sign on which a barcode has been pasted by a tourist organization, in cooperation with the local government. The barcode or marker can be scanned by the camera video feed from a smart- phone, and that code will be deciphered by software communicating with an online database to overlay the appropriate textual information on the browser window to the augmented reality world. The barcode or marker might, in our sign example, in- dicate we are in a tsunami zone and need to exercise caution. Perhaps we also noted the wave on the sign, but were not certain if this indicated this was an ideal location for swimming or suring, or if danger was indicated. The amount of information em- Thomas Richard Schalow Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign Language Environments 965 bedded in a barcode can be large, so we might also learn the sign was put in place last year, and therefore the information is fairly current and probably worth considering. Illustration 3: This is a Japanese sign indicating an earthquake and tsunami zone Augmented reality applications based on location, as provided by gravimetric or GPS technology, do not require the physical environment be changed in order to overlay information on the environment. These applications are more lexible than marker-based AR, but require greater technological sophistication from the AR de- vice than marker-based systems. They of course require that the AR device, such as the smartphone, be equipped with a means to compute relative location on the basis of GPS signals. At present this is one of the limiting factors in the adoption of AR, as commercial GPS devices do not yet have the accuracy to locate objects as small as a sign with pinpoint accuracy. Although the military is able to blow up ships, buildings, and other objects with missiles coordinated by GPS signals, commercial users can presently expect nothing more than to be able to identify buildings on the basis of location, and even in these cases errors of up to hundreds of meters is not uncommon. Therefore, the type of information that can be reliably overlaid on an en- vironment using location-based AR is still limited by errors in GPS signal accuracy. Nonetheless, once a location is established, AR technology can begin to provide the user with a wide variety of information about the area. That information can come from databases used by Twitter, Facebook, or other online sources of social informa- tion. The inal form of AR we are looking at involves object recognition, and the subset of OCR-enabled translation. Although we do not need to know the location of the object to provide this type of information, the processing requirements for translating information using object recognition place a substantial strain on todays smartphones, and robust network connections are also a necessity in most cases. In 966 our sign example, no barcode or marker would be required if we could rely on our smartphone and its OCR software to translate the Japanese characters on the sign. There would be no need to physically alter the environment in order for our visi- tor to have the same information, as a result of the OCR-based translation, native Japanese visitors to the area would have. Fortunately, a great deal of effort has gone into developing OCR and object recognition software, and a number of smartphone applications have already been introduced. 4. Smartphone translation applications and limited augmented reality The potential beneits of augmented reality are so great because browsers will be able to combine marker, location, and object recognition technologies to construct an entire world of information unavailable to tourists who cannot read the language used in the country where they are traveling. It will even be able to augment the information available to native citizens and language users, overlaying additional contextual information taken from online databases. Today tourists already have access to more limited applications that use just a small spectrum of AR technology. Instead of viewing the world through an aug- mented reality browser, for example, we are able to use our smartphone cameras and software such as Word Lens, Camera Translator, CamDictionary, Photo Translator, or similar applications to provide limited OCR-based translation of signs. Even these more limited applications work remarkably well, with a great number of language choices, and can be of tremendous assistance to travelers. It is now possible to travel in Japan, China, or other exotic environments and be able to read at least some of the signs without a working knowledge of the foreign language. Lesser used languages, such as Lao or Cambodian, at present lack the population base to make the develop- ment of OCR translation software commercially viable, but it is only a matter of time before we will also have applications to make reading signs in Vientiane or Phnom Penh a trivial matter. Some countries, of course, already translate some of their signs into English, particularly in large cities, and this translation greatly facilitates ease of travel. Trav- el to more remote and less densely populated locations, however, still presents dif- iculties, as governments lack the economic resources to provide translations for each and every sign that is put up. Moreover, while English translations are certainly welcome, they are not a solution to the navigation problems of every foreign visi- tor. As we pointed out at the beginning of this paper, due to declining transportation costs and other factors, more people who lack basic foreign language skills are now traveling abroad. English translations might be useful to more educated tourists, but are of limited use in their raw form to these travelers. Thomas Richard Schalow Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign Language Environments 967 Illustration 4: This is a Cambodian road sign with an English translation Nonetheless, there is still a case to be made for providing English or some other Western language translations on signs whenever possible, and particularly in countries where text is not alphabet-based, or not Latin in its form. English itself is an object that can easily be decoded by OCR software available today. The Italian visitor to Cambodia, for example, might not be able to read a sign in the native Cam- bodian/Khmer alphabet, but if that sign included an English translation an educated traveler would probably be able to understand the information. With the assistance of an OCR translation application, the English itself could even be translated into its Italian equivalent to make the information available to anyone able to read Ital- ian. Eventually, when Khmer-Italian translation software is available the English language translations would become unnecessary, but until that time they provide a method to make the information accessible to the widest possible group of travelers. 5. Future challenges for augmented reality - hardware issues Although it is most likely that augmented reality will initially become a part of the travelers world through a device such as the smartphone, Arth and Schmalstieg (2011) believe that smartphones are not yet truly ready to deliver full augmented reality to the traveling masses. One basic problem smartphones have is they are not meant to provide continuous camera operations for long periods of times due to limitations in battery technology. I do not believe this is a serious problem, how- ever, as I expect augmented reality will be used only when needed, rather than on a continuous basis. We will continue to view the world through our own eyes, unaided by AR technologies,to gather most of the information we need, and only access an augmented reality version of the world when confronted by needs such as language translation. Another serious issue relates to network speeds, and network availability. Bar- codes and markers can be decoded locally, without the need for an Internet con- nection, if the set of phrases or languages required is limited. For true AR imple- mentations, however, a broadband Internet connection to enable access to databases where larger volumes of information are stored will be required. For more sophisti- 968 cated AR environments a variety of databases will be queried, to provide the maxi- mum amount of available information. In fact, the amount of information available through AR browsers, even today, has the potential to overwhelm the user, and many slower networks, with essentially irrelevant information. AR technology will require sophisticated iltering algorithms to discard information we do not need, while at the same time gathering as much relevant information as possible. The ilter will also be required to authenticate information, guaranteeing it is accurate, unbiased, and current. Unfortunately, due to the fact that augmented reality development for smart- phones is at the mercy of hardware and network service providers, who make their marketing and development decisions based on factors that may not take the needs of augmented reality developers into account, hardware constraints on augmented real- ity will not be easily overcome. We will certainly eventually begin to see independ- ent augmented reality devices brought to the market, perhaps in the form of goggles, for example, that are developed speciically with the power and network needs of augmented reality applications in mind. 6. Augmented reality: software and social issues The most dificult issues confronting AR and other forms of pervasive comput- ing will present themselves as software issues and, as Kang and Cuff (2005) have observed, social issues. This is in part due to the fact that individuals will create most of the information needed to construct a useful augmented reality environment, probably using one of the major AR platforms available today, such as Layar, junaio, or Wikitude. When I, for example, post a remark on a social information network, in my native English language, about the sign I have seen on some store near my home in Japan, that remark becomes part of the database of information available to all subsequent travelers to the area. If this remark was posted to Twitter, for example, that remark now becomes part of the Twitter database. If I have noted that this par- ticular store is open for business from 9 a.m. to 5 p.m., that information could now also become available to any augmented reality environment that is created using the Twitter database. In this example I have not relied on OCR translation software to interpret the sign, but merely on my own linguistic ability to produce a translation of the informa- tion posted on the sign in Japanese. However, my linguistic ability may be incom- plete, or even so deicient as to produce errors. Perhaps the sign actually indicated that the store was open during these times, except on Monday, when it is closed for business. As a result of my own incomplete knowledge, or misunderstanding, I have introduced a piece of data that is only partially correct. Users of the information I have provided will view an augmented reality world that is created with at least some information that is not entirely true. Thomas Richard Schalow Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign Language Environments 969 This error may have been introduced as a result of carelessness on my part. Per- haps I failed to read the entire sign due to a lack of time. Perhaps this is the irst and only error I have made in a post to the Twitter database. Although my translations are normally reliable, in this particular instance the translation was deicient.However, there is also the possibility that I consistently post incorrect information. Perhaps my linguistic skills are so poor that I should not be subjecting others to what amounts to little more than slightly educated guesses about my environment. Who, though, knows that the information I provide should not be trusted? It is also possible that I am purposefully posting incorrect information. Perhaps I have a grudge against the owner of this storeand am consciously seeking to destroy the stores business by providing incorrect information. Again, who is to know what my motives are when I post the information, and whether or not that information can be trusted? These issues are so important because augmented reality is a constructed real- ity, constructed by information that is subject to concerns about credibility, timeli- ness, and relevance. To the extent that any of these are compromised, the environ- ment itself becomes compromised, and perhaps seriously so. These same issues are present on the World Wide Web, of course, and have not been resolved there. It is as possible to ind incorrect, biased information as it is to ind accurate, relevant infor- mation on web pages.The most glaring biases or inaccuracies that could potentially become a part of AR environments will be eliminated by ilter and algorithmsbased on large number and data sets. If there are 99 positive reviews for a hotel, for exam- ple, from a source such as Tripadvisor, and 1 negative review, I can probably assume the 1 negative response was an aberration, and it could be iltered out. Augmented reality browsers will also certainly, as Yovchevaet. al. (2012) note, ilter the information they provide to the traveler on the basis of some sort of con- textual awareness software. This would help to overcome the information overload that threatens to make augmented reality impractical. Viewing too much information crowded onto a small smartphone screen would hide truly important information. As Google and other search engines have discovered, the way in which all this iltering is performed will determine how useful the information will be. Filtering, however, also represents a form of control over expression, and over the value assigned to information. Tourist organizations should be proactive in their embrace of AR technology, working with developers for the major AR platforms to guarantee information is accurate, unbiased, and current. There are also important privacy concerns that are related to augmented reality environments, and these need to be considered by anyone involved with this new technology. In one sense we are all now consumers as well as products in a new augmented reality. This idea goes beyond the term prosumer, used by Alvin and Heidi Tofler (2007) and coined to express our dual function as producer and con- sumer. As human social products, our actions, our location, our very existence, be- 970 comes the source for data consumed by others. To what degree do I have the right to deny others knowledge about my location, about who I am, and when I surrender this right by checking in with a service such as FourSquare, do I lose all control over those rights? These are some of the concerns I hope the tourist industry will address as it works with AR developers to set standards and establish methods by which user-created information can be evaluated for credibility, timeliness, and relevance. 7. Conclusions A successful tourism experience presents the exotic in a form that is under- standable and in some way familiar to the traveler. It does not overwhelm the cogni- tive or sensory inputs of the traveler. The widespread adoption of augmented reality browsers, and development of the software and information needed to enable them in any foreign environment, will make the linguistic dificulties and associated cog- nitive overload of past travel a forgotten inconvenience. Merely viewing the world through one of these augmented reality browsers, most likely through a device such as a smartphone, will transform the world and the tourist experience for the traveler. Thomas Richard Schalow Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign Language Environments 971 References Arth, Clemens and Dieter Schmalstieg. Challenges of Large-Scale Augmented Reality on Smartphones.ISMAR 2011 Workshop: Enabling Large-Scale Outdoor Mixed Reality and Augmented Reality. 2011. Internet access. http://www.navteq.com/outdoor_mar2011/challenges.pdf Cunha, Carlos, Raul Morais, MaximinoBessa, and Manuel Cabral Reis. Contextualized Ubiquity: A new opportunity of rendering information and services. Journal of Theo- retical and Applied Electronic Commerce Research. Vol. 5, 3, 2010, pp. 55-64. ISSN 0718-1876. Curran, Kevin, Denis McFadden, and Ryan Devlin. The Role of Augmented Reality within Ambient Intelligence.International Journal of Ambient Computing and Intelligence. Vol. 3, 2, 2011, pp. 16-33. ISSN 1941-6237. Fragoso, Victor, Steffen Gauglitz, Jim Kleban, Matthew Turk and Shane Zamora. Trans- latAR: A Mobile Augmented Reality Translator. IEEE Workshop on Applications of Computer Vision. 2011. Internet access. http://cs.ucsb.edu/~vfragoso/publications/workshops/131.pdf Furht, Borko, ed. Handbook of Augmented Reality. New York: Springer Science and Media, 2011. ISBN 978-1461400639. Julier, Simon, YohanBaillot, Dennis Brown and Marco Lanzagorta. Information Filtering for Mobile Augmented Reality.IEEE computer graphics and applications. Vol. 22, 5, 2002, pp. 12-15. ISSN 0272-1716. Kang, Jerry and Dana Cuff. Pervasive Computing: Embedding the Public Sphere. Washing- ton & Lee Law Review. Vol. 62, 1, 2005, pp. 93-147. ISSN 0043-0463. Kent, James. The Augmented Reality Handbook - Everything you need to know about Aug- mented Reality.2011. ISBN 978-1743040942. Madden, Lester. Professional Augmented Reality Browsers for Smartphones: Programming for juniao, Layar and Wikitude. West Sussex, UK: John Wiley & Sons, 2011. ISBN 978-1119992813. Tofler, Alvin and Heidi Tofler. Revolutionary Wealth: How It Will Be Created and How It Will Change Our Lives. New York: Doubleday, 2007. ISBN 978-0385522076. Yovcheva, Zornitza, DimitrioBuhalis, and Christos Gatzidis. Overview of Smartphone Aug- mented Reality Applications for Tourism.e-Review of Tourism Research. Vol. 10, 2, 2012, pp. 63-66. ISSN 1941-5842.