How Augmented Reality Will Change The Way We Interact, Work, Travel, and Play

September 24, 2019 – In this next installment of Peter Diamandis’ thoughts on where augmented reality (AR) is taking us, he talks about its impact in a number of industry sectors. In a previous posting, Diamandis described the AR-AI convergence in manufacturing and how it is revolutionizing the industry. In this next description of AR in the present and near future, he goes on to describe how it will impact healthcare, retail, advertising, education, travel, training, transportation, and more. So come along for the AR-AI ride and please feel free to share your thoughts with other readers by commenting here. 

Augmented Reality (AR) today has led to the appearance of more than 2,000 apps and over 1.4 billion active iOS devices. Even at this early stage of its evolution, AR is now permeating the consumer products space. In just the next four years, the International Data Corporation (IDC) forecasts AR headset production will grow by 141% annually, reaching 32 million units by 2023. Where will all these AR headsets be used?

Healthcare 

(1) Surgeons and physicians

Whether through detailed and dynamic anatomical annotations or visualized patient-specific guidance, AR will soon augment the work of every medical practitioner. Today it is already being used as a diagnostic tool. Magic Leap recently hired SyncThink, a company that uses AR for eye-tracking to diagnose concussions and balance disorders. Another startup, XRHealth, has launched ARHealth on Magic Leap, a tool to aid in rehabilitation, pain distraction, and psychological assessment.

Surgeons at the Imperial College London are using Microsoft’s HoloLens 1 in pre-operative reconstructive and plastic surgery where they would normally use CT scans to map blood vessels that supply vital nutrients during procedures. Dr. Philip Pratt, the Imperial College project’s senior researcher, states, “With the HoloLens, we’re now doing the same kind of [scan] and then processing the data captured to make it suitable to look at. That means we end up with a silhouette of a limb, the location of the injury, and the course of the vessels through the area, as opposed to this grayscale image of a scan and a bit more guesswork.” This can dramatically lower surgical risks by helping surgeons visualize the depth of vessels and then choose the optimal incision location. And while HoloLens 1 has only been used for pre-op visualizations, the HoloLens 2 is on track to become part of the operating theatre. The Philips’ Azurion image-guided therapy platform is built specifically for HoloLens 2, providing surgeons with real-time patient data and dynamic 3D imagery as they operate.

AR headsets with their virtual overlays can also be used to improve the sharing of expertise across hospitals and medical practices. In this way, medical specialists with unique skills can be used to offer advice or direct surgeons through complex procedures remotely from anywhere on the planet. Today, Brainlab, a German company, is collaborating with Magic Leap to create a 3D spatial viewer to allow clinicians to work together in surgical procedures across disciplines.

(Editor’s Note: When my brother, a family physician working for the Moravian Mission in Nicaragua many years ago needed to consult the expertise of a surgeon to deal with a complicated procedure, he had to be hooked up through a ham radio operator to an expert in the United States. The remote consult worked but it was fraught with loss of signal from time to time, and the many garbled transmissions that impeded the communication. You’ll be happy to note that the patient lived. AR and the Internet, as my brother would attest, represents a dimensional leap for telemedicine.)

Beyond democratizing medical expertise, AR can even provide instantaneous patient histories, gearing doctors with AI-processed information for more accurate diagnoses in a fraction of the time. By saving time, AR frees doctors to spend a greater percentage of the day engaging in face-to-face contact with patients, establishing trust, compassion, and an opportunity to educate the healthcare consumer rather than merely treating him or her. And when it comes to digital record-keeping, doctors can use the voice control interface to transcribe entire interactions with patients vastly increasing the amount of healthcare delivered daily.

(2) Assistance for those with disabilities

Today, over 3.4 million visually impaired individuals live in the U.S. alone. But with the development of AI-integrated smart glasses the severity of visual impairment may be dramatically eased. As new companies enter the market such as NavCog, Horus, AIServe, and MyEye, they are joining Microsoft and other name-recognized vendors in changing the world for those who are blind. Microsoft has begun development of a “Seeing AI” app which translates the visual world into audio descriptions for the blind using a smartphone camera lens as eyes.

During the Reality Virtual Hackathon in January, hosted by Magic Leap at MIT, two of the top three winners had developed applications for those with disabilities. CleARsite was developing environment reconstruction, haptic feedback, and Soundfield Audio was using sound to enhance a visually impaired individual’s interaction with the world. HeAR, another developer, is using Magic Leap 1 headsets to translate vocalizations and sign language into readable text in speech bubbles that appear in the user’s field of view.

(3) Biometric displays

In biometrics, cyclist sunglasses and swimmer goggles have evolved into the perfect medium for AR health metric displays. Smart glasses like the Solos ($499 USD) and Everysight Raptors ($599 USD) provide cyclists with data on speed, power, and heart rate, along with navigation instructions. Meanwhile, Form goggles ($199 USD) display a swimmer’s pace, calories burned, distance, and stroke count in real-time to a depth of 9.75 meters (32 feet) underwater. With AR applications like these, it will shift accessible health data from our wrists into our field of view.

Retail & Advertising

(1) Virtual shopping

In the year 2030 when we walk into any store it will be retrofitted with AI and sensors to interact with us. Every mannequin will wear a digital design customized to our preference. Forget digging through racks of garments or hunting down your size. With cross-referencing to our personal purchase history, gaze patterns, and current closet inventory, AR-AI displays will be tailor-made for our wardrobe and adjusted to our individual measurements.

Why can I say this? Because Google Lens, an app available on most Android smartphones is already leaping into this marketplace, allowing us to scan QR codes and objects through the smartphone camera. With the Style Match feature, it already can give us some of the capability described in the previous paragraph by identifying pieces of clothing or furniture and viewing similar designs available online and through e-commerce platforms.

(2) Advertising

AR features are quickly encroaching into ads as well. In July, the New York Times debuted an AR ad for Netflix’s “Stranger Things,” guiding smartphone users to scan the page with their Google Lens app and experience the show’s fictional Starcourt Mall come to life.

Immersive AR advertising in the future won’t all be unsolicited and obtrusive. Many will prove helpful. For example, as you walk down a grocery store aisle, discounts and special deals on your favorite items will populate your AR smart glasses. Or if you find yourself admiring an expensive pair of pants, your headset might suggest similar items at a lower cost, or cheaper distributors with the same product. Passing a stadium on the way to work, you may learn about next weekend’s best concert ticket deals and your AR-AI may remind you that a friend with an upcoming birthday might like to join you at this event. You won’t be bombarded with these messages if you choose not to be by simply toggling them off.

Education & Travel

(1) Customized, continuous learning

The convergence of today’s AI and AR will give us the ability to create individual learning environments. Students will be empowered to learn as sensors track neural and physiological data and provide instant feedback to nurture the process. It is this convergence of AR with biometric sensors and AI that has the potential to alter education making it delocalized, customizable to every individual, responsive, and a means to accelerated learning achievement.

Already today within classrooms, Magic Leap One’s Lumin operating system allows multiple headset wearers to share a common digital experience, such as viewing a dissection or studying a historical map. The Magic Leap CAD application can assist students in collaborating on design projects.

Learning in the classroom, however, may become a thing of the past. Already, numerous AR mobile apps can be used to identify objects in a user’s visual field, instantaneously presenting relevant information. And as user interface hardware undergoes its own product evolution so will the software which will make gazing out a window at a cloud an interactive information experience in which the wearer will learn about the water cycle and climate science, while walking past an old building, will mean you learn about its history. It is this innovation that will lead to explosive knowledge abundance.

(2) Training

AR will enable on-the-job training at far lower costs in almost any environment, from factories to hospitals. Smart glasses are already beginning to guide manufacturing plant employees to learn to assemble new equipment. Smart glasses in retail will mean new employees will be able to take an AR tour to learn their jobs.

Today, at Jaguar Land Rover, automotive technicians can better understand the internal components of a vehicle without dismantling it using Bosch’s Re’flekt One AR solution giving them X-ray vision to visualize the internal components of a vehicle without removing a dashboard.

In healthcare, AR solutions will train medical students using hyper-realistic artificial cadavers which will facilitate rapid learning for honing for surgical skills in numerous specialties.

And in sports training, simulators with AR headsets will vastly improve athletic performance, while those who practice chess or play the piano will be given a virtual interface to perfect their skills anywhere they can find a flat surface.

(3) Travel

The convergence of AR glasses combined with AI will make decision-making on trips much easier. For example what hotel or what restaurant to choose. But perhaps even more exciting is one of AR’s more sophisticated uses, instantaneous translation. Whether you need to decode a menu or access subtitles while conversing across a language barrier, instant translation is about to improve exponentially with the rise of AI-powered AR glasses. Even today, Google Translate in your smartphone is already helping to convert menu text and street signs in real-time. AR will mean you will no longer have to deal with the latency of waiting to figure out what is written in a language you don’t speak.

Transportation & Navigation

(1) Autonomous vehicles

To start, Nvidia’s Drive platform for Level 2+ autonomous vehicles is already combining sensor fusion and perception with AR dashboard displays to alert drivers to road hazards, to highlight points of interest, and to provide navigation assistance.

In the current transition phase to autonomy, AR integration will allow drivers to better monitor road conditions. That why Volkswagen is already partnering with Nvidia to produce I.D. Buzz electric cars, set to run on the Drive OS by 2020. Nvidia is also partnering with Toyota, Uber, and Mercedes-Benz to make AR displays commonplace in their vehicles in the near future.

(2) Navigation

We’ve all seen (or been) that someone spinning around with their smartphone to decipher the first few steps of a digital map’s commands. But AR is already making everyday navigation intuitive and efficient. With the Google Maps AR feature already being demonstrated on Pixel phones, instead of staring at a map from a bird’s eye view, you will be able to direct the smartphone camera at the street superimposing directions virtually layered on top of the image. Not only that but as the AI identifies what you are seeing, it can instantaneously communicate with your GPS to pinpoint location and orientation. Although a mainstream rollout date has not yet been announced, this Android feature will likely make it on many smartphone models in the very near future.

Entertainment

(1) Gaming & Sports

We got our first taste of AR’s real-world gamification in 2016, when Nintendo released Pokémon Go. And today, the gaming app has now surpassed 1 billion downloads. But by contrast to VR, AR will increasingly be seen as the medium for bringing gamers together in the physical world, encouraging outdoor exploration, activity, and group collaboration.

AR is expected to make eSports a growing collaborative experience, turning player’s visual displays into live-action stadiums. This year, the market for eSports will exceed $1.1 billion USD.

(2) Art

Many of today’s most popular AR apps allow users to throw dinosaurs into their surroundings, i.e., Monster Park, learn how to dance, i.e., Dance Reality, or try virtual tattoos, i.e., InkHunter. As high-definition rendering becomes more commonplace, art, too, will grow more accessible. Magic Leap aims to construct an entire “Magicverse” of digital layers superimposed on our physical reality. Location-based AR displays, ranging from art installations to gaming hubs, will be viewable in a shared experience across hundreds of headsets. Individuals will simply toggle between modes to access whichever version of the universe they desire. Endless opportunities to design our surroundings will arise.

Apple recently announced the [AR]T initiative consisting of floating digital installations viewable through [AR]T Viewer apps in Apple stores. These installations can also be found in [AR]T City Walks guiding users through popular cities, and [AR]T Labs, which teach participants how to use Swift Playgrounds, an iPad app, for creating AR experiences.

(3) Shows

At the recent Siggraph Conference in Los Angeles, Magic Leap introduced an AR-theater hybrid called Mary and the Monster, wherein viewers watched a barren “diorama-like stage” come to life in AR.

While audience members shared the common experience like a traditional play, individuals could also zoom in on specific actors to observe their expressions more closely. So say goodbye to opera glasses and hello to a future of AR headsets when you go to a Broadway show, opera, orchestra or rock concert.

Final Thoughts

While AR headset manufacturers and mixed reality developers are racing to build enterprise solutions in numerous business sectors, consumer use is following close behind. Magic Leap is leading the way with its “Magicverse” of localized AR displays in shared physical spaces. And as AR-supportive hardware gets integrated into our smartphones, businesses will have an invaluable opportunity to gamify products and immerse millions of consumers in service-related AR experiences. Even beyond the most obvious first-order AR business cases, new industries to support the augmented world of 2030 will soon create a market surge in headset hardware, data storage solutions, sensors, holography and projection technologies.