“Someone can grab a HoloLens, start a Guides session, and literally have a trainer in their head,” he said. “If they do need help, they can call an expert right from the app.”
Toyota feedback also helped improve other aspects of the Guides experience. For example, technicians used to put a QR code on the hood of a vehicle, scan it, and then follow the holographic instructions.
But those holograms could appear to drift as a worker moved around the car. Toyota worked with Microsoft to develop “object understanding.” That meant Toyota technicians could scan the entire vehicle, which helped lock a hologram in place and eliminate the parallax problem. A hologram pointing toward a bolt would always point to the exact location, no matter the viewer’s angle.
Over the years, the authoring experience in Guides has improved to where creating holographic instructions is as easy as creating a PowerPoint, Kleiner added.
“I don’t have to hire an army of consultants to build this. I don’t have to have a bunch of people with computer science degrees. I can give it to the experts on the frontlines, and they can generate their own content to train other folks or share,” he said.
Toyota found immediate value in Guides as a training tool, Kleiner said. Instead of working one-on-one with trainees, Toyota trainers can let trainees work independently and supervise multiple trainees at the same time – increasing their efficiency many times over.
During a pilot project at their San Antonio plant, Toyota used HoloLens 2 and Guides to train employees on how to assemble a new version of the Toyota Tundra. The data showed it was a success, Kleiner said. Defects were cut in half. Depending on the individual, training time fell between 20% and 50%, he said.
Partners in mixed reality
While HoloLens 2 devices have helped define what’s possible in the industrial metaverse, Microsoft is a platform company, Taylor said. That’s why Microsoft is committed to making its mixed reality software available wherever its customers are – whether that’s on a HoloLens 2 or another company’s device. And while Dynamics 365 Mixed Reality Apps provide enterprise-grade software so customers can get to work immediately, Microsoft has also built a Mixed Reality partner network of ISVs who can extend solutions to meet unique needs in different industries, from construction and education to healthcare and pharmaceutical.
The U.S. Department of Labor’s Occupational Safety and Health Administration (OSHA) turned to one of these partners to improve worker safety during trench excavation. On construction sites, trenching and excavation are risky but essential jobs. One cubic yard of soil can weigh as much as a car, according to OSHA, and that delicate balance with gravity can be upset in an instant. That is why trench collapses are the leading cause of death in the construction industry.
To help mitigate this danger, OSHA instructors are using a custom mixed reality app to give trench safety inspectors immersive, hands-on hazard training without real-world risk. The app, created by the software studio Clirio, consists of six different scenes using realistic and immersive graphics to show variables like ground conditions, hazards, safety equipment and best practices for mitigating risk. Sophisticated sound design and animation add to the lifelike experience.
“Whenever you teach somebody something, it’s one thing to tell them, it’s one thing to show them, and it’s another thing to let them do it,” said Anthony Towey, Director of the OSHA Training Institute. “We can actually go into a trenching and excavation environment that’s as close as they’re going to get in the field.”
Thousands of construction contractors could use this training, he added. “And now they’ll be able to do it in a safe environment, where they can practice and make mistakes.”
Hokkaido Electric Power Company is using HoloLens 2, Dynamics 365 Remote Assist and a custom app to help staff inspect critical equipment at a thermal power plant. On every inspection patrol, workers navigate a vast labyrinth of boilers, turbines and generators and examine thousands of pieces of equipment for often subtle changes that can help them avoid larger problems, said Takaharu Umemoto, who works in the company’s Information Technology Section, Thermal Power Department. That requires extensive know-how and experience, he said. It would often take new hires one year of shadowing experienced technicians before they could handle everything.
Today, new hires get up to speed much quicker, he said, and the app has improved the efficiency of patrol inspections. The experience has made the company enthusiastic supporters of mixed reality technology, Umemoto said.
“I had an image of (mixed reality) as a technology for games, but it was a revelation to find that it can be used as an intuitive and easy-to-understand solution from the perspective of transferring patrol inspection skills,” Umemoto said.
‘Everyone has a voice now’
As early customer pilots with HoloLens 2 have led to larger deployments, Microsoft has invested heavily in making the cloud-connected technology easier to manage at scale. That means IT departments can manage a HoloLens 2 headset just like any other laptop, phone or tablet, Evans said.
Those improvements helped Toyota move HoloLens 2 from the lab to the enterprise seamlessly, Kleiner said. “Usually when we leave the lab, there’s a huge learning curve: how do we maintain and operate these at the enterprise level?” Kleiner said. “All that was avoided because when we took the device to the IT department they said ‘Oh, this is just another Windows machine. We know how to handle it.’”
Microsoft will continue to work with customers to solve unique and tough challenges, Evans noted.
Customers like the U.S. Army are helping improve both software and hardware, while others are helping drive industry-specific improvements that may eventually have broader application, Evans said. It’s akin to how technologies like ceramic brakes and variable valve timing first appeared in Formula 1 but ultimately went from the racetrack to everyday streets.
“The military program has its own set of requirements that are tuned to the needs of the soldier. So, it’s helping push the whole platform forward. It’s great to have early adopters that are driving requirements because you end up with this trickle-down effect,” Evans said.
One thing Evans has said Microsoft hears from customers is that, unlike consumers who expect a constant crop of new gadgets, businesses don’t want to have to replace their devices every two years. That causes too much churn. “No one wants to be obsoleted for 10% better capabilities. They don’t need a successor yet, but they want to know it will be there at the right time,” he said.
Evans said Microsoft is pushing forward on all core hardware technologies: display, tracking, sensors, battery life. “We’re just looking for the right design point to make it a meaningful update. They want a successor device that’s going to enable an even higher return on investment,” Evans said.
As the HoloLens hardware and software continue to evolve, Toyota’s Kleiner expects Guides to remain “the killer mixed reality app” for frontline workers. One day, it could be like Word, available on any device. But for now, Toyota will keep rolling out HoloLens 2 headsets across the company, giving frontline workers the tools to work and collaborate in new ways.
“We now have a device we can deploy to every person,” Kleiner said. “It’s easy to maintain, and it allows our workers to participate in the larger conversation, regardless of rank or team structure. Everyone has a voice now.”
Related Links:
Top image: Desktop collaborators can annotate in 3D space and augment what a frontline worker wearing a HoloLens 2 is seeing with the latest update to Dynamics 365 Guides. (Image courtesy of Microsoft)
Jake Siegel writes about Microsoft research and innovation.
