The clever tricks of technology and science behind the holograms
Earlier this year, Microsoft introduced a crazy-sounding, ambitious holodeck for your face. The HoloLens is a computer on your head, but it’s not a virtual reality headset and there are no wires nor a bulky PC to tether yourself to. Instead, HoloLens puts its floating windows and holographic dogs into the real world. Into your living room.
It sounds awesome. It looks awesome. But how exactly does it work? Microsoft shared some of the broader details when it announced the device in January. Now, after the technology giant’s big BUILD developer conference, we know even more about the ins and outs of this wonderfully weirdo piece of tech.
There are a couple of key ingredients to taking a head-mounted computer-hat and turning it into an effective hologram machine, the first of which is getting those images in front of your eyes. To do that, HoloLens has a pair of translucent screens for its eye-pieces, ones that allow for the injection of floating windows into your line of sight without completely blocking you off to the world in the way big headsets like the Oculus Rift do.
But it’s about more than just putting a screen in front of your eyes. The real magic is turning that visual information into something that appears to be an actual 3D-object that you can walk around and look at from all angles, as opposed to flat images that just sit in front of your eyes *coughcoughgoogleglasscough*.
This is where the rest of the HoloLens’ sensors come into play. In the top part of the HoloLens visor there are a number of cameras facing out that scan the area and collect as much data as possible. HoloLens then turns that information into a digital model where the holograms can live in 3D space. Microsoft hasn’t explained the nitty gritty of how these work, but it’s reasonable to assume they’re basically like a head-mounted version of the Xbox’s Kinect camera, and that they use a combination of standard video and infrared depth-sensing vision to draw a digital picture of your surroundings. These digital eyes look out at the world with a 120-degree gaze, HoloLens inventor Alex Kipman told Wired shortly after the gadget’s announcement, and they suck in everything they can. They get a view not only of the room but also of your limbs, so you can reach out and “click” on non-existent holograms with your hands.
But just seeing things isn’t enough. To make these holograms convincing as you move, the HoloLens needs to know exactly where your head is. To that end, it also contains an accelerometer (to measure the speed your head is moving), a gyroscope (to measure the tilt and orientation of your head), and a magnetometer (to function as a compass). With all those sensors, HoloLens will be able to collect enough data to know what your room looks like and where your head is. That’s enough data to decide where the holograms should appear to be and what they should look like.
Does that sound like a lot of data? It’s a lot of data. At Microsoft’s BUILD conference, Kipman said this info-lanche is on the scale of terabytes, way more than any mobile device would be able to deal with. But the HoloLens has a chip in it that Microsoft calls the HPU, or Holographic Processing Unit. Microsoft won’t give away many details about what’s happening in there, but in short, this processor digs through all the sensor data and translates it into a smaller and much more manageable chunks of data, which the HoloLens’ GPU and CPU can use. And remember, HoloLens doesn’t need to be connected to a computer, a mobile phone, or anything at all. Everything goes down inside the headset.
Finally, there’s the element of sound. HoloLens will have microphones for keeping tabs on ambient sound and taking commands via voice control. It will also have speakers, which serve two purposes. First, they don’t block out the sound of the real world like headphones would. But more importantly they’ll also use binaural sound to fool your ears into thinking they’re hearing things in 3D space. It’s like stereo sound but without the headphones. So if you create, say, a holographic TV and then turn your back on it, these speakers will play TV noise in a way that makes it sound like the TV is actually behind you, even though nothing is there.
These illusions—images that change like real objects as you move your head, and noise that sounds like its anchored to a real place—add up to a holographic illusion that is apparently pretty eerily convincing. It’ll be at least a few months before you can have one on your head to see for yourself, though.
And while the idea of having holographic dogs in your living room is cool, the most important applications for HoloLens are likely to be niche and industrial. But the mechanics behind the magic are truly impressive, and HoloLens is just the first we’re seeing of them. It’s far from the last.