Posted on Leave a comment

The making of the HoloLens 2: How advanced AI built Microsoft’s vision for ubiquitous computing

Cloud collaboration

A key advantage of intelligent cloud-powered holographic computing is the ability to share information with others who have a HoloLens or another device with similar capabilities, said Marc Pollefeys, the director of Microsoft’s Mixed Reality and AI Zurich Lab in Switzerland.

Pollefeys is leading a team that develops core computer vision algorithms for a mixed reality cloud service called Azure Spatial Anchors that allows holograms to persist, locked in the real world, for anyone with the appropriate level of access to view.

For example, spatial anchor technology allows a manager in a factory to place holograms next to equipment on an assembly line that contain vital, real-time operating and maintenance information that any credentialed worker with a mixed reality capable device can access.

“If I can only place information that I will see back on my device, it’s probably never worth placing holograms in the world, but if I can annotate the world and afterward anyone else in the company that has the right access can see all of the information, it is suddenly much more valuable,” Pollefeys said.

To create this capability, Pollefeys and his team developed AI computer vision algorithms that process data from sensors to extract 3D geometric information about the environment and piece it together in the cloud to create a digital twin, or map, of the area of interest.

HoloLens has always built up a 3D or spatial understanding of its environment to function. Azure Spatial Anchors creates, refines and shares these maps across devices, Pollefeys noted. That’s why the maps from individual devices are pieced together and stored in the cloud.

“It doesn’t make sense to have that data only on an individual device,” he said. “It is one of those things where I have a little piece of the puzzle, and somebody else has a little piece of the puzzle, and all of the devices together have covered the whole space of interest.”

These maps get denser, more precise and robust over time as different mixed reality capable devices – HoloLenses as well as properly equipped phones, tablets and laptops – map their environment and share the data with the cloud.

For example, the map of the factory floor where the manager left holograms floating over pieces of equipment on the assembly line is steadily refined as more and more credentialed workers view the holograms with their devices.

This capability also enables scenarios such as a meeting between architects and clients to view and interact with a holographic 3D blueprint of a building, each of them with mixed reality capable devices looking at the blueprint from their own point of view as they sit around a table.

Azure contains pre-built services to write applications for these types of experiences on HoloLens and any other mixed reality device, including smartphones and tablets running the iOS and Android operating systems, noted White.

“That collaboration experience isn’t just locked to HoloLens,” she said. “And, the cost and complexity and skillset required to make an application that does something amazing is far down.”

The cross-device and platform capability, for example, enables experiences such as Minecraft Earth, which merges the popular video game with mixed reality in a way that players can build and place virtual structures in the real world that persist so that other players can interact with them on their devices.

“We all get to participate because it is based on using cloud technology that can be understood and interpreted by all different devices,” said White.

Technology that is designed for people

For HoloLens to work as envisioned, the technology that underpins the experience needs to understand the world in ways that are similar to the way people do, Kipman noted.

That’s why he and his collaborators across Microsoft have developed, deployed and leveraged AI solutions throughout the ubiquitous computing fabric, from the silicon in the headset of HoloLens 2 to Azure AI and mixed reality services.

Back at his digital whiteboard, Kipman has now sketched out a vision for ubiquitous computing that is rife with words, boxes, arrows – and a stick-figure picture of two people locked in conversation next to an intelligent device.

That, he says, is the ultimate goal of ubiquitous computing – to get people to interact with other people in natural ways.

To drive home the point, he establishes a moment of intense, deliberate eye contact and says, “Hopefully, you are getting more out of this conversation because you are physically present with me.”

“We could have done this over the phone,” he continues. “We could have done it over Skype. I could have recorded it and sent you a tape. You didn’t choose to do that. You chose to be physically present with me. Why? Because that’s how we do human things.”

“The con is you have to be here at the same time I am here, and we have to be in the same location. The power of this technology is it gives us the ability to displace space and time.”

Top image: Microsoft Technical Fellow Alex Kipman models the HoloLens 2, a sensor-packed holographic computing headset. Photo by Microsoft.


John Roach writes about Microsoft research and innovation. Follow him on Twitter.

Leave a Reply