Virtually a Reality: VR & AR Enter the Safety Space

The goal of the VR headset is to immerse you in your virtual experience. This device is worn over your eyes like a pair of goggles to block out all external light and show you an image on high-definition screens in front of your eyes.

Oh yes, there’s even The Void—a VR game played in laser-tag like facilities—which allows patrons to immerse themselves in the game. In one of their simulations, participants become Ghostbusters and get to blast and trap Slimer and his gang of ghastly friends or frenemies perhaps. Dan Aykroyd even appears as Dr. Stantz in the promo video; it’s a “total protonic” reprisal for sure.

And there’s so much more out there, too, with the emergence of products like the Oculus Rift VR headset; DAQRI smart glasses; Strivr’s VR training platform; Microsoft’s HoloLens MR head-mounted display; and even your smartphone. With all that’s emerging in the space, it’s probably not too surprising that Charles Fink, a consumer tech expert covering VR and AR, argued in a recent Forbes article “Why 2019 Was the Best Year for AR and VR at CES,” referring to the Consumer Electronics Show, a massive tech event held every January in Las Vegas.

While VR and AR are hot tech topics, “The biggest barrier to wide adoption of immersive technologies is the lack of good user experience design,” said Tuong Huy Nguyen, senior principal analyst at Gartner, in an article. “Businesses already experiment with VR, but hesitate to fully commit. On the other hand, customers are fascinated by the new entertainment possibilities, but do not want to invest in head-mounted displays (HMDs) as long as the offering is so small. This is going to change during the next five years.”*

To be clear, we are not saying VR, AR or MR is the panacea for safety training and keeping people safe in the field and that these technologies will eliminate the need for more traditional safety training and protocols, but, as you’ll see in this article, there are myriad tantalizing safety-related applications to keep an eye on.

So what’s the difference between these emerging technologies and how can they help keep people safe? We’ll get to that in a bit.

My first experience with what is now considered a forebear to VR technology was primitive by today’s standards but fun nonetheless. Remember the View-Master? Patented in 1939, the classic red toy allows children—adults, too; admit it—to view scenes through a stereoscopic twin-mirror system that produces a mesmerizing amount of visual clarity. At least baby boomer and Gen X children thought so. Today’s kids have VR View-Masters to play with that blow away the vintage slide reels from my youth. Except in the nostalgia department, of course.

Like many people, my first experience with AR technology occurred in 2016 when Niantic released Pokemon Go. For a couple of weeks, my kids and I would go around trying to catch the odd creatures of the Poke-verse that would appear in our universe through the app, but alas we couldn’t find that little ball of energy, Pikachu, hanging around our neighborhood. Perhaps he was too busy filming “Detective Pikachu”; some movie stars just don’t do cameos. While some people love Pokemon Go—or Niantic wouldn’t be valued as a $4 billion company by some estimates—we soon became bored throwing virtual Pokeballs at these imaginary animals invading (loitering in?) our real world. We decided to give the software the virtual heave-ho. At least we stopped before we could walk off a cliff like two California Go players reportedly did in 2016.

More recently, I was able to interact with my bottle of 19 Crimes Cabernet Sauvignon from Treasury Wine Estates. With a unique marketing strategy that some purist oenophiles might consider, ahem, criminal, the winery’s AR app allows you to “hear” from some of the British hooligans who were banished to Australia in the 18th century. These wrongdoers were charged with committing one of 19 crimes ranging from grand larceny (a theft over 1 shilling) to impersonating an Egyptian. As punishment, the offenders were sent to Australia and given the opportunity to start a new life rather than face the gallows. Of course, the trip to get there was often fraught with danger.

If you point your phone’s camera toward the label of a bottle of 19 Crimes, the convict from yesteryear “comes to life” through the AR app and tells a tale of how that ne’er-do-well wound up Down Under. The marketing strategy seems to be dead-on as it has helped make the brand seem hip while helping it realize 60 percent growth in sales volume and 70 percent in value in 2017 alone, according to BetterRetailing.com.

Of course, many people still see VR and AR as child’s play, but please don’t read that as a pejorative.

The reality is we’re entering a new era where computer software will help bring about advancements in so many areas, and that includes training for workplace safety as well as many other potential safety-related applications. So repeat after me: There’s nothing wrong with having a little fun. As Judy Willis, who practiced neurology for 20 years and is an expert on brain research regarding learning, wrote in an article posted on Psychology Today’s website: “When classroom activities are pleasurable, the brain releases dopamine, a neurotransmitter that stimulates the memory centers and promotes the release of [acetylcholine], which increases focused attention.”

In an email, Willis told me: “I have no specific research or insights about VR although I am a proponent of its educational potential for engaging learners beyond the ‘video game model’ of which I write.”

The model she is referring to relates to her research into how video games can help with education.

In a separate article posted on Edutopia.com, Willis wrote: “The popularity of video games is not the enemy of education, but rather a model for best teaching strategies. Games insert players at their achievable challenge level and reward player effort and practice with acknowledgement of incremental goal progress, not just final product. The fuel for this process is the pleasure experience related to the release of dopamine.”

There’s that dopamine term again, so releasing some while using VR seems like a no-brainer, right?

“A lot of people still haven’t had VR experiences of any kind. It’s still very low,” said Grant Winters, director of business development at Industrial Training International, which offers a host of VR training courses for cranes, lifts and other types of heavy equipment. “It’s a fun thing to show to people, and say, ‘Look, I know this is used for gaming and I know it can look graphically like a video game, but this is a very real industrial training tool.’ ”

*Smarter With Gartner, 3 Reasons Why VR and AR Are Slow to Take Off, September 6, 2018: https://gtnr.it/2CtmQKk

So what exactly is the difference between the technologies?

In 1994, ATR Communication Systems Research Laboratories’ researchers Paul Milgram and Fumio Kishino sought to answer that very question in a research paper explaining the “virtuality continuum.” They wrote, “The conventionally held view of a Virtual Reality (VR) environment is one in which the participant observer is totally immersed in, and able to interact with, a completely synthetic world.”

In other words, virtual reality immerses people in a place that only exists in the minds of its creators and the eyes of its users; with apologies to novelist Gertrude Stein: There is no there there, so don’t try punching a location into your Uber app.

Augmented Reality, on the other hand, according to the researchers, “refers to all cases in which the display of an otherwise real environment is augmented by means of virtual (computer graphic) objects.”

Basically, you can add information to real-world objects through AR software that allows graphics and information to be overlaid on a real scene for amusement, training and other purposes. So when you look at your phone through the Pokemon Go app, you see a little critter bopping around in front of, but, of course, no creature has magically materialized.

The third option, Milgram and Kishino explained, is a “merging of real and virtual worlds, which we refer to generically as mixed reality.”

A Microsoft Corp. spokesperson told me in an email that “mixed reality encompasses the spectrum from augmented reality to virtual reality, blending the physical and digital worlds to produce new environments where physical and digital objects coexist and interact in real time.”

In a video explaining Microsoft’s HoloLens MR headset, you see a holographic model of the human body from the inside out. It’s almost like a cross between the traditional plastic human body you commonly find in a high school science lab and something you’d see in a Body Worlds exhibit. After the person in the video examines the holographic body, you see a closeup of a beating heart inside. A narrator says, “Using transparent lenses, spatial sound and understanding of your environment, holograms look and sound like they’re actually part of the world around you.”

Holograms have come a long way since National Geographic featured one of a skull on its November 1985 cover above the headline, “The Search for Early Man.”

Businesses and consumers alike are beginning to accept this new reality about the future of these versions of faux reality as well as their potential.

Spending on VR and AR is projected to rise almost 72 percent between 2017 and 2022. In 2018, spending on VR and AR was set to hit $27 billion, according to International Data Corp. By 2022, IDC predicts that spending will reach $53 billion alone in the consumer marketplace and $56 billion in retail, discrete manufacturing and transportation. A separate report from Allied Market Research predicts the market for mixed reality will have a 72 percent compound annual growth rate through 2024 with a global valuation of $5.4 billion in five years. As we all know, big revenue opportunity is a big catalyst for big-time innovation.

“Commercial interest in both augmented and virtual reality continues to accelerate as new hardware ships, improved software appears and more use cases evolve,” said Tom Mainelli, IDC’s vice president of devices and AR/VR, in a news release.

And since things are moving fast, researchers are looking for ways to minimize or eliminate the common complaints about the technology, including how it can cause motion sickness, vertigo and general discomfort from having to wear a headset.

Indeed, one of the most well-documented problems with VR and AR headsets—besides their potentially prohibitive costs—is that they can make some people feel nauseated. It’s commonly known as “cybersickness.” But researchers at the University of Waterloo in Canada, for instance, are working on a solution. “Despite decreased costs and significant benefits offered by VR, a large number of users are unable to use the technology for more than a brief period because it can make them feel sick,” said Séamas Weech, a postdoctoral research fellow at the university and the lead author of the paper, in an article published on ScienceDaily.com. “Our results show that this is partly due to differences in how individuals use vision to control their balance. By refining our predictive model, we will be able to rapidly assess an individual’s tolerance for virtual reality and tailor their experience accordingly.”

In an email update, Weech told Spark: “Our subsequent research uses electroencephalography (EEG) to decode the neural signatures that correspond to motion sickness in VR. With this new way of characterizing cybersickness, we hope to boost the accuracy of our predictive model.” EEG is the measurement of electrical activity in different parts of the brain.

“If the device doesn’t work at around 12 milliseconds, you’ll start to feel sick,” added Eric Klein, the design manager of Teague, an almost-century-old design company. The company has designed the interior of every Boeing commercial plane going back to 1946; more recently Teague has used VR to help design the new 777x commercial airplanes, which will be able to hold up to 425 passengers. The 777x is expected to take to the skies this year and be delivered to clients next year. “We shoot for like 90 frames a second in order to keep the quality of the imagery high enough where you don’t feel ill,” Klein explained.

Especially since the speed and quality of the technology has improved, it’s not surprising that one of the key areas we’re seeing VR for in the business world is workplace safety training.

In a recent blog, Southern California Edison wrote: “When it comes to safety, the learning challenge is to re-create a potentially high-hazard situation in an immersive way while minimizing risk to the learner.”

The 130-year-old Golden State utility company, which provides electricity to some 15 million residents over a 50,000-square-mile area, worked with STRIVR, a 3-year-old virtual reality company that got its start by creating immersive VR training videos designed to help the Stanford University football team excel on the gridiron. The two companies collaborated to create three virtual safety experiences related to three risk-based scenarios. In one, there’s an electrical wire down. In another, a customer is found to be in serious medical distress from heat exposure because an air conditioner is not working. In a third, a worker performs a “Circle of Safety” vehicle check designed to help employees learn what to look for so they can take a truck on the road safely. “In this safe space,” according to the blog, “everything matters, even rare encounters like the desert tortoise hiding under the tire.”

Especially since the speed and quality of the technology has improved, it’s not surprising that one of the key areas we’re seeing VR for in the business world is workplace safety training.

In a recent blog, Southern California Edison wrote: “When it comes to safety, the learning challenge is to re-create a potentially high-hazard situation in an immersive way while minimizing risk to the learner.”

The 130-year-old Golden State utility company, which provides electricity to some 15 million residents over a 50,000-square-mile area, worked with STRIVR, a 3-year-old virtual reality company that got its start by creating immersive VR training videos designed to help the Stanford University football team excel on the gridiron. The two companies collaborated to create three virtual safety experiences related to three risk-based scenarios. In one, there’s an electrical wire down. In another, a customer is found to be in serious medical distress from heat exposure because an air conditioner is not working. In a third, a worker performs a “Circle of Safety” vehicle check designed to help employees learn what to look for so they can take a truck on the road safely. “In this safe space,” according to the blog, “everything matters, even rare encounters like the desert tortoise hiding under the tire.”

Kristi Rawlinson is head of marketing communications at STRIVR. In an interview, she told me that about 70 percent of the VR experiences STRIVR has developed revolve around safety training. It’s important “to train your employees in a way that they retain the information,” she said, “like checking the landing gear on an airplane. If they don’t know exactly what they’re looking for or haven’t been trained in the right way, I don’t know about you, but I would not want to get on that airplane.”

That makes two of us.

But the aviation industry has a long history of using flight simulators going back to the late 1920s with the Link Trainer. In a recent edition of “Ask the Captain” in USA Today, John Cox, a retired pilot who runs an aviation consulting company was asked, “What do you think about flight training that is augmented by virtual reality?” Cox’s response to the interestingly worded question: “Flight training technology is expanding constantly. Simulators today are very realistic and are a form of virtual reality already. This trend is likely to continue.”

In a separate example of how safety and VR can be intertwined, Rawlinson referred to a case study on STRIVR’s website for Tyson Foods, the Springdale, Arkansas-based Fortune 100 company. According to the article, “After deploying VR as part of employee training, Tyson experienced more than a 20 percent reduction in injuries and illnesses compared to the year prior. This beat their goal of 15 percent. Initial survey results have been overwhelmingly positive; 89 percent of learners said they felt more prepared after VR training.”

That might have something to do with retention advantages inherent in VR training.

In an article from Training magazine’s website titled “The Impact and Potential of Virtual Reality Training in High-Consequence Industries,” Dr. Narendra Kini, the president and CEO of Miami Children’s Health System, told the publication that the retention level of information after a year via VR learning can be as high as 80 percent vs. 20 percent for traditional training. “The level of understanding through VR is great because humans are primarily visual and VR is a visual format,” he said.

The author of the January 2018 article, David Wentworth, a principal learning analyst at the Brandon Hall Group, added that VR offers training-related advantages, such as improved safety in on-the-job training, increased productivity, low-cost training for new employees, and it’s enjoyable, too. He also said that interest in the technology is growing among highly regulated industries, such as aerospace, chemicals, manufacturing and energy.

You can add retail to that list as well.

As CBS News reported in December 2018, Verizon Wireless is using VR training to help its associates prepare for a dangerous, albeit unlikely, situation: a robbery and hostage situation. As CBS reported, “In one digital scenario, two gunmen strike as the store opens, taking one employee hostage and going straight for the safe. It was only a simulation, but as CBS News correspondent Tony Dokoupil learned firsthand at a Verizon training site outside Washington, D.C., the fear was all too real.”

Throughout the simulation, employees are asked to answer questions on how they would respond if such a crime took place. According to the story, Verizon sees fewer than 50 such robberies a year at its 1,600-plus stores. The U.S. in general saw almost 119,000 robberies involving a firearm in 2017 alone, according to Statista. This is no small problem. Part of the allure of training with VR is the ability to immerse people in uncommon situations and offer that training to many employees.

“We’ve found that virtual reality is the best way to effectively replicate learning experiences that were previously challenging or impossible such as our ‘Retail Store Robbery’ VR and ‘Customer De-escalations’ training,” said Lou Tedrick, Verizon’s vice president of global learning and development, in a written statement emailed to Spark. “The feedback we’ve received from our participants in these difficult learning scenario trainings justify the additional investment involved in VR over a more traditional instructional method.”

In 2012, a Seattle-based woman named Becky Anderson was on the operating table to have polyps removed from her vocal chords. It was supposed to be a routine procedure, but the doctor’s laser touched the endotracheal tube inside Anderson’s airway, according to the law firm representing her. The tube caught fire, and she had to have multiple surgeries to repair the damage. A year later, a jury awarded her $18 million in damages to add to the $12 million the hospital had already paid out in an initial settlement.

As you probably guessed, surgical fires are quite rare. According to a final report from the National Quality Forum released in 2017, there were 51.4 million inpatient procedures alone in the United States in 2010, the most recent year listed. So when the ECRI Institute says there are an estimated 90 to 100 surgical fires in the United States every year, it’s alarming but those types of medical mistakes are barely a blip on the radar statistically speaking.

It’s also hard to prepare people for such an occurrence because they happen so rarely and are hard to simulate.

“It’s a really challenging thing for people to train on because it’s difficult to replicate,” said Cole Sandau, CEO for Health Scholars, a medical learning software company that offers VR training. “Just reading about it isn’t really necessarily good enough. You need to really have some ability to practice with that.”

That’s where VR comes in; after all, time is of the essence for the scrub nurse or scrub tech in the operating room, for whom the training is designed, to put the fire out quickly.

“They’ve got about a minute,” Sandau said. During the VR training, “The way it works is the fire will go off, and we actually put a timer in front of them. Most people are so focused on the fire they never see the timer, which is really interesting. The fire progresses, and the more it progresses, the more severe the burn itself becomes. If they were able to get it out within a few seconds, which they should be able to do if they’re properly set up and prepared, the burn is really no worse than a sunburn. If they let it go and 15, 20 seconds lapses because they just either don’t know what to do or they weren’t set up to deal with it, the burns get really severe.”

According to Sandau, some vendors offer non-VR training courses that include a real fire, which, for obvious reasons, adds a degree of risk to the trainee as well. “Here we can very much simulate it to the point that I would say probably seven to eight out of 10 people who do our VR experience, when the fire actually erupts on the patient, take physical steps away from it. They’re really feeling the fact that there’s a fire there. It scares them, and they physically respond away from it.”

In other words, with VR training, you won’t get burned but you will be engulfed by the safety training simulation.

As ITI’s Winter explained to me, the company’s VR software for heavy machinery training offers coaching moments for its users. “For example, if we introduce a failure of some type, maybe a sling failure, a wench failure, an outrigger failure,” he said, “different things that can affect the crane, having a catastrophic event. It’ll even pause the simulation for a moment and give some verbal instructions to the operator about what’s going to happen and the best way to get out of that situation.”

And if you add too much weight to a crane during a simulation, you might be in for a surprise. “We can tip the crane over and simulate an accident. And a lot of people feel that’s very real, especially in our motion-based unit that has actuation, and will actually slant or tilt when the crane begins to tip over. It can be a very engaging experience—and sometimes scary.”

That feeling of being lost in the moment is not limited to in-person VR simulations either.

When I was researching the Titanic for my interview with Dr. Robert Ballard, the man who led the expedition to find the ship in 1985, I watched a History Channel documentary from a few years ago. In the video, three men walk into a hangar as a narrator proclaims, “Investigating an airliner or shuttle crash, the pieces are collected, brought to a hangar and studied to learn how it broke apart. What went wrong and why? It’s the same with Titanic. The pieces hold the answers.” At that moment, a massive holographic image of the ship based on underwater photos of the wreckage is projected in the room, and with the software the ship can be manipulated and put back together like a colossal jigsaw puzzle.

I asked Teague’s Klein if the company produced a similar experience when it designed the inside of airplanes.

“The biggest one that we’ve done is around 60 by 40 feet,” he said. “You can physically walk around inside the airplane.”

He says the Oculus Rift really opened the door to a new generation of VR technology. Just a few years ago, Oculus was nothing more than a startup in search of funding as part of a Kickstarter campaign. Shortly thereafter, Facebook Inc. bought Oculus for a reported $3 billion. Oculus did not respond to numerous requests for an interview for this story.

Klein explained that Teague worked on one airline interior for a client where, with VR headsets in hand, they went to different locations around the United States to get feedback. “That insight fed directly into the next revision of all of that architecture.”

Additionally, according to a 2017 Harvard Business Review article titled “Why Every Organization Needs an Augmented Reality Strategy,” a Boeing study found that by using AR for training, workers were able to assemble aircraft wings 35 percent faster. Also, people who had little or no experience assembling wings were able to complete the task correctly with a 90 percent better success rate over people who trained on 2-D drawings and documents.

While Teague is focused on VR at the moment, Klein said the company has begun dabbling in AR for prototyping. Klein said he’s sees a lot of potential for using AR for fieldwork safety. “Anytime you’ve got data that you’re matching with the physical world or it’s not just overlaying things but also diagnostics, sensing, that kind of thing,” he said, “AR feels like the thing.”

When I reached out to Microsoft for information on the HoloLens, one of the case studies they provided me about mixed reality, which as we mentioned incorporates AR, too, was on how Japan Airlines Co. uses the HoloLens to train engine mechanics as well as flight crew members who are interested in becoming co-pilots. “We believe that HoloLens can contribute to the safety of our business, which is the most important criteria for airlines,” an executive from the airline is quoted as saying in the 2016 case study.

A complementary video on the topic shows a hologram of a plane engine. While showing off the technology, it is noted that the engine can be inspected component by component with a voice-over narrator explaining its use and significance. Later on in the video, we see a yellow hazard area light up on the floor in front of and behind the virtual engine as a warning signal is sounded, and an area of the engine itself contains a yellow warning explaining that part of the engine is “hot.” The narrator then says, “Please be aware of the areas marked as hazardous. In a real-life scenario, these areas must be clear for safety when you start an engine.”

If you don’t see that as a lift for safe best practices, then perhaps this will.

Thyssenkrupp is a German elevator company. In a separate video from Microsoft, we see how the company is using MR to elevate its safety practices.

In the 2016 video, a Thyssenkrupp executive explains how a technician “goes to the job site prepared, probably better prepared than ever before because he has the relevant data available to him in the best format possible. And in a variety of ways, he can tailor the data to what is needed.” The executive adds that: “One of the most important capabilities we see with HoloLens is we can trigger remote calls to a subject-matter expert” to get a second set of eyes.

In the video, the technician calls a senior tech to look over a circuit board via a camera in the device before proceeding. After reviewing the circuit, the expert says everything looks OK and “you’re good to go.” (Note: They could have also used “telepresence” technology or “holoportation” as Microsoft calls it, to beam the person’s image into the room, but that would be more for show than purpose.)

ITI’s Winters also explained that the company has experimented with adding AR applications for inspecting cranes and other forms of heavy equipment “where we can put a crane into a real environment and walk around and see different aspects of the equipment” as well as workflow processes. Not surprisingly, Winters also mentioned how cloud technology allows ITI to “grow, upgrade and continue to service our customers with advanced updates or content.”

Workers using AR headsets also are able to access and collect data as needed. Some AR headsets can record a video to capture your surroundings to document an incident or near-miss, some allow you to tag equipment parts or do research on them for maintenance purposes, and some allow you to access schematics or other safety information while keeping your hands free. And as haptic technology improves from research being done from organizations like the Hasso-Plattner Institut in Germany, people will be able to “feel” and manipulate objects in the virtual world by using “forced feedback” through self-adhesive electrodes for electric muscle stimulation rather than wearing typical VR gloves. According to the website VR Focus: “The HPI team note that one of the challenges of creating a haptic system for MR and AR is that users will encounter physical objects in addition to virtual ones, so keeping the hands free of encumbrances is important.”

Separately, a recent research paper titled “A Critical Review of the Use of Virtual Reality in Construction Engineering Education and Training” published in the International Journal of Environmental Research and Public Health, states: “One major benefit of AR is the provision of engaging, motivating and immersive contents.”

The authors explained that one area primed to take advantage of AR, VR and is for construction safety. “The construction industry is a high-risk industry where the accident rate remains high,” they wrote. “Some of the reasons leading to the high risk include limited safety knowledge of on-site employees and lack of safety awareness and training of these employees. Traditionally, construction safety training is provided in a classroom setting with slide presentations or videos. The safety information provided in the presentations and videos often do not represent real construction site conditions.”

The authors also discuss the potential merits of a safety training and visualization system that integrates Building Information Modeling (BIM)—a tool for designing and constructing buildings and infrastructure—location tracking, AR and gaming technologies to provide construction workers with safety information they can access through their mobile devices. This can improve workers’ “safety awareness effectively” and “enhance workers’ real-time communication ability in an unsafe environment,” they noted.

Another area that could be primed to implement futuristic technology for safety is first responders. The National Institute of Standards and Technology (NIST), for one, is creating virtual environments for training first responders, such as firefighters and medical technicians. “Such interfaces could be embedded in firefighters’ masks or smart glasses worn by emergency medical technicians, for example,” an article on NIST.gov says. “A visual display might show the temperature or audio might warn that oxygen is low in a backpack tank. The idea is to present helpful data in an intuitive and nonintrusive manner.”

There’s nothing virtual about an actual emergency; it’s all too real, so anything to help first responders handle those types of difficult conditions could not only be a game-changer but a life-saver as well.

As the fictional character Ferris Bueller famously said during his hedonistic “Day Off,” “Life moves pretty fast. If you don’t stop and look around once in a while, you could miss it.” The same holds true for AR, VR and MR. Teague’s Klein told me: “It’s amazing how fast the transformation is happening. Literally three years ago when we started we went from purchasing things to leasing them because things were just changing so fast that it wasn’t worth actually holding them.”

It’s safe to say that industry will see more and more safety-related innovations in VR, AR and MR, especially as more companies realize the potential of these technologies to help keep people, processes and property safe and sound; in fact, we’d call it a virtual lock.