TeleHuman' taps Kinect for 3D holographic videoconferencing

By Chris Jablonski | May 6, 2012, 11:22pm PDT Summary: A Queens University researcher has created a Star Trek-like human-scale 3-D videoconferencing pod that allows people in different locations to video conference as if they were standing in front of each other.

Credit: Human Media Lab, Queen's University Researchers at the Human Media Lab at Queens University in Kingston, Ontario, have developed a life-sized hologram-like telepod that uses Microsofts Kinect System and a cylindrical display for live, 3D videoconferencing. The system, called TeleHuman, allows two people to simply stand in front of their own pods and talk to 3D hologram-like images of each other. An array of six Kinect sensors mounted at the top of the display capture track 3D video and convert it into the lifesize image. Since the 3D image is visible 360 degrees around the pod, the person can walk around it to see the other persons side or back, a key advantage over flat displays. Why Skype when you can talk to a lifesize 3D holographic image of another person? said professor Roel Vertegaal, director of the Human Media Lab. TeleHuman was built primarily with existing hardware, including a 3D projector installed at the base of the 1.8 meter-tall translucent acrylic cylinder and a convex mirror. The researchers used the same pod to create another application called BodiPod, which presents an interactive 3D anatomical model of the human body. Through gestures and speech control, the model can be explored 360 degrees; and when people approach the pod, they can wave their hands to peel off layers of human tissue. While 3-D holographic video is not a new technology (Cisco and Musion Systems created an onstage holographic video conference 5 years ago), TeleHuman demonstrates that it can be done for a lot cheaper using the versatile Kinect platform and off-the-shelf hardware.

The TeleHuman is a life-size video conferencing display pod which uses ten Microsoft Kinect sensors to help "beam" your image into another pod, enabling 3D chat between two people as if they were both in the same room. Microsofts Kinect motion sensor is very popular, whether its being used in its native environment to make beloved Star Wars characters rhythmically jiggle around, or hacked to an insane, powered skateboard to provide a gesture control system, its varied abilities are certainly being exploited. Now a series of the sensors find themselves attached to the TeleHuman, a 3D display that beams a life-size representation of yourself to another pod in another location, so you can talk and interact with other people as if you were really there. The TeleHuman is a cylindrical pod that stands just under two-meters tall and has a 3D projector hidden inside its base. This projects your image onto a convex mirror, which then reflects it onto the wrap-around acrylic screen. Its a complete 360-degree representation too, enabling the viewer to walk around the TeleHuman as they would if the person were really in the room, and although its not actually holographic, it looks exactly how we imagine that technology to appear. There are six Kinect sensors mounted around the top of the TeleHuman, which capture your position, then another four are placed around the room to form a square, and they record your back and side view to complete the 3D picture.

Exciting, but expensive

Developed by the Human Media Lab at Queens University in Ontario, the TeleHuman has been designed with video-conferencing, teaching and medical work in mind. You can see one possible application in the video below, where a system called BodiPod uses gestures and an interactive 3D image to help teach anatomy. In the same way that FaceTime has been embraced by the adult entertainment industry, we can see any future, commercial versions of the TeleHuman also being adopted for more intimate video conferences too. So, the TeleHuman brings use one step closer to making our sci-fi communication dreams a reality, but there are a couple of drawbacks. First, one still has to wear 3D glasses to see the image in three dimensions, which will dim the image even more; and second, its quite expensive. Professor Roel Vertegaal, Director of the Human Media Lab, told Wired.com that a mass production TeleHuman would cost around $5,000 with at least $1,500 of that being taken up by the ten Kinect sensors and as youd need one for each person taking part in the conversation, thats a big investment. However, given that the parts used are almost all commercially available, it may not be that long until the price drops to something at least slightly more reasonable. Imagine walking up to a tall, cylindrical pod, and talking to a life-size, 3-D projection of a faraway friend, family member, or colleague. And not only do you see the person in 3-D, you can walk a full 360 degrees around the pod, and see your conversation partner from every angle just as if the person was actually standing inside the cylinder. It may sound like the science fiction of Star Trek or Star Wars, but this is exactly what a team of researchers at Queens University have developed in the Human Media Lab. Grandly dubbed TeleHuman, the life-size cylindrical pod allows users to conduct real-time, 3-D and 360-degree video conferences. Its not quite a holodeck, but its a whole lot closer than todays 2-D video-chatting options like Skype or FaceTime. (To see the system in action, check out the video below.) Its all about social proximity cues. You can walk up to a person instead of having to phone them

or go through an interface, Roel Vertegaal, director of the Human Media Lab and Professor of human-computer interaction, told Wired. Instead, theres just this pod. It might be designated for a lover or a business partner. Its near the water cooler, and the same thing is in his or her office or home. You walk up, the thing starts to glow and boom beam me up, Scotty. There are two main components to the system: the cylinder itself, which features a special display and contains an internal projection mechanism, and a set of cameras that track and capture your position as you communicate with your conversation partner. The Human Media Lab team started with a hollow, cylindrical display thats made of sandblasted acrylic and mounted on top of a wooden base. A stereoscopic DepthQ projector, along with an Nvidia 3D Vision Kit, is located at the bottom of the base, and a convex mirror is installed at the top. Shooting upward, the projector beams video of your conversation partner into the mirror, which then beams the video back onto the display. ? ?You walk up, the thing starts to glow and boom beam me up, Scotty. Roel Vertegaal? ?With the help of the embedded Kinect technology and distortion correction, TeleHuman projects a holograph-like image of a person onto the interior of its cylindrical display. And the structure is a little more than 6.5 feet tall, making it possible to project most people in life size. But thats just the display portion of the system. TeleHuman is two-way, interactive technology, so it also needs to capture image movement as well. As you walk around the display, talking with your partner, TeleHuman tracks your position and captures your image with 10 different Microsoft Kinect sensors, each containing a built-in camera. Six Kinects line the top of the pod, tracking your position around the cylinder and capturing frontfacing images. The other four Kinects are arranged in a square around you, about 8 feet away from the pod. These capture your side and rear views. Whats nice about the cylindrical form factor is that its essentially a flat display curved in one dimension, which is exactly the dimension youre walking in, Vertegaal said. You always get a good view of the person as you walk 360 degrees around the display. While TeleHuman doesnt use holographic technology, its results still looks like a hologram. To this extent, TeleHuman is a major design win as current holographic systems cant match the promise of their sci-fi inspirations. Unlike most holography-like projections, TeleHuman accurately preserves motion parallax the changing appearance of an object as you move around it. And this is an important development in terms of social interactions. In its studies, the Human Media Lab team discovered that real-worldaccurate, 360-degree views are integral to how we share ourselves with each other. Communication breaks down even with a subtle little thing, Vertegaal said. When you think about preserving human communication, its more about what you leave out rather than what you add. With this system, were trying to leave out as little as possible.

Professor Raul Vertegaal. Photo:Ariel Zambelich/Wired The team specifically studied eye gaze, pointing direction and body pose. They made sure that users could experience direct eye contact while talking a huge improvement over most current video chats, where the camera is placed higher than a persons face in the video chat interface. In one experiment, the Human Media Lab asked a yoga instructor to show off various poses to a participant in another room, using both TeleHuman pods and 2-D displays. Participants were asked to replicate the poses, and the yoga instructor then judged how well the participant performed. Results showed that users did much better when they could walk around a display and see a yoga pose in 3-D. TeleHuman can also accurately depict where people are pointing and gesturing, another benefit for real-world-accurate interactions. When you turn over to show something or point at something at the screen, people see you turn around, and its actually relative to the screen thats there in the room, Vertegaal said. Being able to merge the virtual and the physical is really, really important.? ?With glasses, the projection totally appears holographic. You get the sense that theres this person hovering in space. Roel Vertegaal? ?This has a number of real-world applications. Doctors, for example, can use the device to conduct remote examinations, seeing a patients life-size body at different angles and views. The Human Media Labs paper on TeleHuman also mentions that the technology would have useful applications in sports instruction for example, teaching golfers how to improve their swings. Naturally, it could also be used for businesses and classrooms, and even for digital theater performances.

TeleHuman isnt without its problems, though. The image is a bit too dim and is somewhat low-res, since its projected onto such a large display. And in order to see the projection in 3-D, users still need to wear stereoscopic glasses. Glasses are a bit of a problem, because you lose eye gaze, Vertegaal said. You can either use [TeleHuman] with glasses and get a fully stereoscopic view, or without glasses, but you still get motion parallax. With glasses, the projection totally appears holographic. You get the sense that theres this person hovering in space. Vertegaal says his team hopes to fix these issues and further develop the TeleHuman pod. For instance, they want to have multiple pods video-conferencing simultaneously. Roaming pods, where the devices can move around a space, is another option. Vertegaal also says that he would like to use the Kinect cameras to show the environment that a person is in currently, the cameras block out background images. How much would one of these pods cost a consumer? Vertegaal estimates that TeleHuman could be mass produced for around $5,000 per device. A single Kinect sensor costs about $150, and the device uses 10 sensors. So maybe you wont see the TeleHuman in many offices or homes Professor Roel Vertegaals Star Trek-like 3D cylindrical display is probably as close to teleportation as we will ever get. A Queen's University researcher has created a Star Trek-like human-scale 3D videoconferencing pod that allows people in different locations to video conference as if they are standing in front of each other. "Why Skype when you can talk to a life-size 3D holographic image of another person?" says professor Roel Vertegaal, director of the Human Media Lab. The technology Dr. Vertegaal and researchers at the Queen's Human Media Lab have developed is called TeleHuman and looks like something from the Star Trek holodeck. Two people simply stand in front of their own life-size cylindrical pods and talk to 3D hologram-like images of each other. Cameras capture and track 3D video and convert it into the life-size surround image. Since the 3D video image is visible 360 degrees around the Pod, the person can walk around it to see the other persons side or back. While the technology may seem like it comes from a galaxy far, far away, it's not as complicated as most would think. Dr. Vertegaal and his team used mostly existing hardware including an array of Microsoft Kinect sensors, a 3D projector, a 1.8 metre-tall translucent acrylic cylinder and a convex mirror. The research team, also consisting of John Bolton, Kibum Kim, Peng Wang and McGill Prof. Jeremy Cooperstock, used the same Pod to create another application called BodiPod, which presents an interactive 3D anatomy model of the human body. The model can be explored 360 degrees around the model through gestures and speech interactions. When people approach the Pod, they can wave in thin air to peel off layers of tissue. In X-ray mode, as users get closer to the Pod they can see deeper into the anatomy, revealing the models muscles, organs and bone structure. Voice commands such as "show brain" or "show heart" will automatically zoom into a 3D model of a brain or heart. Dr. Vertegaal will unveil TeleHuman and BodiPod at CHI 2012, the premier international conference on human-computer interaction, in Austin, Texas May 5-10. Remember all those science fiction movies and television shows that depicted people in a future time, talking to life-sized holographic images? That time is now. A research team from Queen's University in Ontario has developed a video communications system that allows you to speak with an eerie three-dimensional representation of the person on the other end of the line. Under the guidance of Professor Roel Vertegaal, director of the university's Human Media Lab, the team uses some pretty common components to create a pretty uncommon communications device.

A series of Microsoft Kinect sensors and 3D camera equipment team up to capture the threedimensional image of each conversant. On the other end of the computer communications set-up, a 3D projector casts the image within an acrylic cylinder equipped with a convex mirror. What results is two people talking to life-like projected images of each other. The team calls the it the Telehuman. Besides just the sheer awesome factor, Telehuman gives users the ability to convey body language that would be lost on a flat display, bringing the experience closer to face-to-face communications in the real world. You can even walk around the entire perimeter of the person's image and see their back and sides. "Why Skype when you can talk to a life-size 3D holographic image of another person?" asks Dr. Vertegaal. Well, you don't have to be a professor to answer that question, rhetorical as it may be. Remember when you watched the Jetsons and their awesome video phones, and couldn't wait for the real, live version? Well, it's here, but there is a downside. Instead of getting to hide behind the blissfully blind telephone, now you, your bed head, your shirt stain, the broccoli in your teeth from last night's dinner (maybe?) and the bags under your eyes get all up in the face of your boss, your coworkers and your loved ones ... and, possibly wors On the plus side, we reckon Telehuman would take webcam sexy time to a whole different level at least before you go crashing through the pod and destroy thousands of dollars of equipment in a hormone-entranced outburst. This isn't one of the two potential applications of the Telehuman technology shown in the video below. The second is called Bodipod, a medical imaging function that allows you to pull off layers of skin to study anatomy.