3-D TV Preparing the Way for Holographic TV?
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bove, V. Michael
Holographic television, or “holo-video”, has been seen by many as the ultimate development not only of holography but of electronic visual communication generally. To make widespread, successful holo-video, four things are needed: 1) content, 2) a distribution mechanism, 3) sufficient processing at the receiver, and 4) suitable electro-optics at the display, and all of these must be available at prices suitable for consumers. In the past one to two years, there has been a great interest in 3-D television, but few researchers seem to have noted that many of the recent developments in 3-D TV are also solving — or at least pointing the way to solving — problems associated with holo-video. I will examine particularly relevant developments in content capture/creation, content representation (including standardization activities), and the increased suitability of graphics processors for 3-D applications, and connect these with work at the MIT Media Laboratory in developing a holo-video display suitable for consumer use.
 Well, good morning, everyone. My name is Michael Bove. I'm from MIT, and I'm not only the chair of this session, but I'm also the first speaker. What I'd like to talk about today is the connection between 3D television, which is increasingly being pushed as a consumer technology, and holographic television, which we hope will soon be pushed as a consumer technology. And I have to give specific thanks to Quinn Smithwick, who is a postdoctoral researcher in my group at the MIT Media Lab, and to PhD students Jim Barabas and Dan Smalley. The three of them did all the work, so, but I get to come to China. Also, thanks to the people who support this work, and thank you to Steve Benton, who got me involved in holographic TV in the first place. So by way of introduction, and really just to position this talk relative to the other topics we've been discussing at this conference, I want to note that holographers have different feelings about things like stereoscopic television. And some holographers have an interest in all sorts of 3D imaging, and those of you who knew Steve know that quite well, that any sort of 3D technology he loved. And some other holographers think that stereoscopic 3D is beneath them, and I think we all know some people like that. But what I would like to make you think about today is that the current activity in 3D TV is doing some of our work for us. It's laying technical and creative groundwork that's going to apply when 3D TV is not just left-right stereo, but it's autosterioscopic, very, very wide viewing angle, and horizontal or full parallax. So that's going to help us bring the following generation of 3D TV, holographic TV, to market. Now, in my paper, I cite some references from 1950 about 3D television, and the truth is they are technically not all that much different from what is happening in 3D television displays today. So why is there suddenly excitement? Why when you go to some place like the Consumer Electronics Show in Las Vegas in January, will all the Consumer Electronics manufacturers prominently be featuring 3D video displays? Well, first of all, because high definition happened, and now industry needs a next big thing, and 3D is probably a bigger thing than increasing the pixels on the screen yet again. Also, most consumers don't remember the not all that great history of 3D television, and so they aren't put off by the fact that the last few times this has been pushed, it wasn't very good. Display technology is changing very quickly, and so this is an opportunity both to slip in 3D, since you're going to be replacing your display anyway, and to change the ways in which we think about displaying video. Unlike in the past, we don't just have fixed television broadcast standards that are bringing us all our video. We have new flexible and extensible methods of representing television, and they can carry 3D formats. On Internet video, you don't have the codec hardwired into your display, you can download the codec over the network before you download the video. Blu-ray discs have the ability to evolve, so you can change the features of Blu-ray. They aren't fixed with a particular method of representing content. Lots of things like shutter glasses or high refresh rate screens for temporarily multiplex 3D or high speed digital interconnects that used to be expensive and complicated are now very cheap. So that's enabling us to do this as well. As we've seen with the example that Frank put up on the stage, high quality cameras are now small and cheap, so you can make stereo camera rigs practical and inexpensive. And there's a lot of content that's computer graphics models, and so that can be re-rendered for 3D, and more particularly GPUs are fast enough that you can do that in real time. Now GPUs are not only fast enough to give you very, very realistic 3D video games, they're fast enough to give you realistic 3D scenes in multiple views in real time. So there are lots of ways of transmitting 3D video from a source to a destination. You can first of all just send multiple streams using something like MPEG. You could interleave either spatially or temporally, or you can anamorphically squeeze two frames into one frame and transmit that, or you can alternate frames. What's more interesting and more efficient is when you code video frames to take advantage of the redundancy, not just in time, but also from one view to another. I think many of you are familiar how codecs like MPEG use motion vectors to predict one frame from its temporal neighbors. Well, you can also predict one view from its spatial neighbors. And clearly, the future is not going to be having 64 video cables going from 64 cameras to 64 projectors as we have on the screen here. It's rather going to be coding all of those 64 views together and sending them over one cable. And what has happened in the past year that's particularly relevant to solving that problem is that H.264's AVC codec now has a backward compatible amendment that supports multi-view predictive coding. So there is a mechanism in place for sending two or more parallax views reasonably efficiently. Now, you can get more sophisticated if you have more computation available and you have ways of generating a depth map, a Z-buffer, you can send a 2D stream and a depth map, and then the expectation is the receiver will compute the parallax views by shifting things differentially based on the Z values. And ultimately, certainly most efficiently, if you have a way of generating it, you could send the full 3D model and let the display render it appropriately. And there are more technical details on these in the paper if you want to track them down. But there's an important point behind all of these. And that is unlike in the past, 3D displays, even 2D displays, are now assumed to have enough local processing that the format in which images are transmitted is decoupled from the way in which the display handles them. So you can have a lenticular display, but you could send it temporally multiplexed or anamorphically squeezed frames and the expectation is it can convert them. So there are lots and lots of activities. This is just from last year what has happened in 3D TV. And there's even more than that. But again, the one I mentioned before is MPEG adopted an amendment to H.264 for multi-view video coding. It's called MVC. I think you're going to be hearing a lot about that because that will be implemented pretty much everywhere. Now, there are some other things that still need to happen. We need to have production and post-production equipment for creating this multi-view or 3D model content. And ultimately, people need to rediscover a creative and technical sense of what works and doesn't in the 3D world because there's a lot of bad 3D out there and not pretty much good 3D out there. And so some of what was known by certain low-cost cinematographers in the 1950s needs to be rediscovered. I was heartened to note that there were starting to be books coming out this year on 3D moviemaking again. And that's a topic that hasn't had books come out for a very long time. Now, of course, this is the holy grail. Princess Leia on a table in front of you. Unfortunately, I don't know where to buy the photons that will go halfway across the room and turn left. If any of you knows where to get some, I'll take a bucket full. But this is certainly what we'd really like in terms of 3D, in terms of holo video. What we get is CNN calling this a hologram, which it definitely isn't. And enough said about that. So what holo video really looks like is it looks like a hologram. It looks like the things that are on display next door except that the hologram is moving. And the hologram can be behind what you think of as the screen, which really isn't a screen at all. It can be in front. It can straddle it. And so in this particular case, that object is actually a few centimeters in front of the screen. And so I can appear to hold it in my hand. Or to make this happen, of course, you need content. Now, I want to note, and this is something that not a lot of people seem to have thought about. If you have a holo video display, it's sort of a superset because of other 3D display technologies. Since you can modulate light and you can steer it, you can take two or many parallax views. You can take light fields from integral cameras. You can take a whole variety of ways of thinking about 3D imaging. And you could put them all in a holo video display if you can do the computation locally to generate the diffraction pattern. You need a distribution mechanism. You need computation at the display. And you need the electro-optics at the display that actually turn the light into the hologram. And if 3D TV development is going to do some of these for us, then people like me don't have to invent it all. I don't have to build the end-to-end system. Rather, I can just take advantage of these things that other people are doing for other purposes. And so that's been the goal in my group for the past several years is to use as much as possible of what's already being deployed. And there are lots of people who are working on holo video. And when I make this list by holo video, I mean actually creating a diffraction pattern, illuminating it with monochromatic or coherent light, refreshing it in real time. And I'm sorry if I didn't include some of you. We like to make the distinction that our group has an emphasis on making this a near-term consumer technology. So we're trying to make a holo video display that can display the 3D content that's going to be around already that costs a few hundred dollars to build, that can use the kinds of interconnects for video signals that are already going to be on your desktop or in your living room. And in order to do that, you know, this is as close as we get to equations in this talk. We need about two, if we have a screen that's a meter wide, holo video needs about two million pixels on every scan line for that meter wide screen, which is one of the reasons that people like me and people like Steve Benton spent a lot of time thinking about horizontal parallax only because that reduces the problem somewhat. You still need two million pixels on a scan line, but you don't also need a million scan lines in the other direction. The other thing we need to do, we have to compute the diffraction pattern at the display. And what my group has done over the past couple of years and was very nice to see a number of you doing GPU programming as well is we're taking advantage of standard graphics processors in PCs and game consoles, but we're not having them output just a 2D image or a pair of 2D images. We're having them output a diffraction pattern. So if we magically tomorrow had a holo video display in your living room, but we had existing stereoscopic TV being broadcast or sent over the Internet, how would we use the holo video display to display a stereo pair? Well, the first thing you could do is just display each view at a single angle. That works. The computation is trivial for the diffraction pattern. Of course, depending on the spacing of your eyes, you need to be at a specific distance from the screen as well as at a specific horizontal position. We'd like to relax that. Since we can steer light, we could superpose multiple diffraction patterns that would reconstruct the left and right views at multiple places so that you don't have to be at a specific distance. You still have to be more or less centered on the screen. What you then probably want to do is repeat this pair of left, right views several times so the viewer could be in multiple locations or there could be multiple viewers. Now, this isn't much harder to compute than that. And if we put a little gap, then your eyes will naturally move to a place where you see both views, and this more or less guarantees you won't see a pseudoscopic image because if you can see two views, you'll see proper stereo. I think people have probably seen this photo. This is the Mark II display that Pierre Saint-Hilaire and others built in the 1990s. It looks like a science experiment. It's the size of a dining table. It has a 19-inch rack of electronics. It's about as tall as I am. It's based on the scophany diffractive television technology which came from the 1930s. We want to move away from 1930s technology. This, by the way, is a still, I don't know if any of you saw this film. This is called International House. It's a film starring W.C. Fields about the dawn of television, and it takes place in China, and the television system looks an awful lot like Mark II. I couldn't resist putting that there, but we don't want that anymore. We want something that's a monitor that just plugs into your PC. Of course, it won't look nearly as dramatic when we turn out the lights and you have this red glow everywhere, but at the same time, it's a lot more practical. What we have been doing for the past couple of years, and in particular Dan Smalley and Quinn Smithwick, is building a new display architecture that's really simple, and unlike every other scophany architecture display that's come before it, even though it is based on an acoustic-optic modulator of special kind, it doesn't have horizontal mirrors. It just has a vertical scanner, and you just sort of look down the optics toward the light modulator, and you see a whole of video. And these are the specs. This is the proof of concept version. We have a plan to scale this up. But the specs are pretty much the same as Mark II, except that we can fold the optical path to fit in a box like a desktop CRT monitor, and it's basically VGA resolution. It's 440 scan lines. Mark II is only 144. And it's based on these very special surface acoustic wave light modulators made out of lithium niobate that Dan Smalley is making for his doctoral dissertation. These are interesting because they're really cheap. These cost maybe $10 to $20. We can make them ourselves at MIT, and they have over a giga hertz of video bandwidth. So you can actually get hundreds of thousands of pixels on a scan line. And we can even make them 2D. The way that they work is if you send a surface acoustic, you have a wave guide just under the surface of the material. You couple in the beam to the wave guide, and a single mode propagates down the wave guide. If you send a surface acoustic wave across, you diffract the light. If you send a surface acoustic wave toward the source, you actually change the mode. You convert the mode, and you can cause the light to move up and down as well. What's more particularly relevant about these devices for HoloVideo is they rotate the polarization. So the first order beam, which is the one we want, has a different polarization from the zero order beam. So if you put a polarizer at the output of it, you don't need a physical stop to get rid of the zero order light coming out of here. So how do we render? Well, this is an extension of the diffraction specific work that goes back to people like Mark Lucente. We're not doing a full physical model of the scene and modeling the interference. We're rather starting with the diffraction pattern and synthesizing a diffraction pattern that's needed to create a particular light field. So you start with a chirped grating, which creates an effective emitter either in front of or behind the screen. You modulate that chirped emitter with the intensity that you want that emitter to have from multiple viewpoints, then you sum them all up and you get a horizontal parallax only hologram. And that has a lot of advantages. You can pre-compute the chirps, GPUs can do this in real time. It just, it does all of the things correctly that are really hard in physically based models like occlusion and transparent surfaces and atmospheric effects and lighting. And you can use standard lighting and texture representations. Indeed, we, our rendering pipeline is just open GL at the front. So we can plug this into lots of other pieces of software but output an HPO hologram rather than just a 2D view. There's a natural place in the pipeline to insert parallax views. If you have pre-computed parallax views from someplace else, there's a place in the rendering pipeline. You can just put them in and there's a disadvantage. Like all holosteriograms, this doesn't provide, with one exception that we know of, this doesn't provide accommodation cues consistent with virgins. So your eyes are focused someplace different from where they're converged. And one of the things that's really nice about holography is that we ought to be able to make all of the cues to 3D, all of the perceptual cues for your eyes and your brain consistent with one another. And that's a violation of that. Well, at Photonics West next January, we're going to present a solution. We're going to present holographic stereograms that can still be computed in real time but where each point in the image focuses wherever the object is at that point. And so I will just won't explain the whole thing because I don't have time but I'll give you a little hint. When you render the stereoscopic views from the 3D model using a virtual camera, you keep the Z-buffer. And then the Z-buffer tells you where that, what depth that emitter is at at each angle. So it's not a particularly hard thing and it's a process that the GPU is already doing anyway. So this January, we reported that on GPUs, we've gone from one frame per second to greater than 15. We actually have faster results than that now which we haven't yet published in the literature. And so what I'd like to do is I would like to show you HoloVideo rendered in real time from a 3D model. This is on the old Mark II display. So this is only 144 lines and this is with a single NVIDIA GPU and the codec that we used to compress this did some violence to it as well. So my apologies that this doesn't look all that good. If you find yourself in Boston, let me know. We'll show you the real thing in a couple of months when the display is a little brighter and more focused. We'll unveil Mark III and you'll see the same model at standard definition TV resolution. I'll note that the renderer runs at full speed on Mark III and we actually just sort of throttle it down for Mark II. But enough said about that, here is a dinosaur running past some trees and that has about 30 degrees of look around and that is texture mapped. The camera didn't capture it all that well. In conclusion, if you haven't yet bought this book that Steve spent many years working on and Betsy Connors and I and a couple of other people spent three years after that trying to package and put together in a book, it's available and certain starving holographers like Betsy will benefit if you purchase the book. So thank you very much for your attention and I'll conclude there. Are there any questions at this point? Michael, are you saying that at Photonix you'll be showing us the shoebox version of Mark III? Is it going to be there or is it still going to require a superstructure that's not portable? I think the main concern is that it might be a little bit too fragile to take on an airplane and there will only be one of them then so I'm hesitant to have it leave the building. We hope to have a lot of them at some point but we will definitely be showing video of our new algorithm and I hope we'll be showing video of our new algorithm shot from the screen of Mark III rather than Mark II. And so what's the dimension of the desktop device in the end or do you expect that again? The dimension is about the size of a 19-inch desktop CRT. And so the business of plugging in existing models, I guess the model is to, you know, in the same way that vector graphics, the way that flash works is that you download an instruction set and then the data is actually already there in the device. Is that what you're proposing? No, we're actually doing it more the way that real-time gaming does it where you're updating the model. I mean, we assume you'll be transmitting geometry, transmitting surface normals, transmitting texture maps and transmitting things like directions for the virtual camera. Michael, what is the display size? The display size for Mark III is just a little bit smaller than Mark II so it's 80 by 60 by 80 millimeters. Mark II is just a little bit larger than that. But you'll notice that the view volume is actually as deep as the dimensions of the screen. And when do you think we will have something like at least 15 inches of the RML display? Well, full color. Well, full color turns out to be easy. The first generation holo video display at MIT in 1990 was full color. I mean, that's only a factor of three. Scale-up is what kills you because there's a square in there. So when will we have bigger displays? Big... If you know of a source of very big pieces of lithium niobate, we could actually make that relatively soon because Dan has an architecture for putting multiple modulators on one monolithic piece of lithium niobate to make a direct view display. But the biggest lithium niobate wafers we can get are not as big as that. So, is there a source of lithium niobate as a limit? That's certainly... The lithium niobate plus the wiring to it, there are some bandwidth problems, obviously. So right now, the Mark III display uses two DVI cables from the PC. And if you imagine a tiled display, you would need two DVI or HDMI cables for each tile of this size in order to have enough bandwidth for that much parallax. Okay, thank you. I understood that Mark III display implements diffraction-specific approach, right? Yes. And did you make any improvements of this algorithm developed by Mark III Center? Yes. If you look at the... We published a paper in optical engineering in 2007 about something called the RIP algorithm, which is a significant improvement over the original diffraction-specific algorithm. What we are now doing is a further improvement over the RIP algorithm. And then what we'll be presenting next January is a further improvement over that. So we have been continuing to improve the algorithm at the same time as we increase its efficiency on the GPUs. Thank you. Mike, hi. I just wanted to thank you because I don't think your work can be underestimated. I think it's one of the most important developments in the log-of-heat we have, you know, this century. And I hope we'll live to see 3D TV before we... With much order. Thank you. Well, thank you very, very much. I'm embarrassed.
 Well, good morning, everyone. My name is Michael Bove. I'm from MIT, and I'm not only the chair  of this session, but I'm also the first speaker. What I'd like to talk about today is the connection  between 3D television, which is increasingly being pushed as a consumer technology, and  holographic television, which we hope will soon be pushed as a consumer technology. And  I have to give specific thanks to Quinn Smithwick, who is a postdoctoral researcher in my group  at the MIT Media Lab, and to PhD students Jim Barabas and Dan Smalley. The three of them  did all the work, so, but I get to come to China. Also, thanks to the people who support  this work, and thank you to Steve Benton, who got me involved in holographic TV in the  first place. So by way of introduction, and really just to position this talk relative  to the other topics we've been discussing at this conference, I want to note that holographers  have different feelings about things like stereoscopic television. And some holographers  have an interest in all sorts of 3D imaging, and those of you who knew Steve know that  quite well, that any sort of 3D technology he loved. And some other holographers think  that stereoscopic 3D is beneath them, and I think we all know some people like that.  But what I would like to make you think about today is that the current activity in 3D TV  is doing some of our work for us. It's laying technical and creative groundwork that's going  to apply when 3D TV is not just left-right stereo, but it's autosterioscopic, very,  very wide viewing angle, and horizontal or full parallax. So that's going to help us  bring the following generation of 3D TV, holographic TV, to market. Now, in my paper,  I cite some references from 1950 about 3D television, and the truth is they are technically  not all that much different from what is happening in 3D television displays today.  So why is there suddenly excitement? Why when you go to some place like the Consumer Electronics  Show in Las Vegas in January, will all the Consumer Electronics manufacturers prominently  be featuring 3D video displays? Well, first of all, because high definition happened,  and now industry needs a next big thing, and 3D is probably a bigger thing than increasing  the pixels on the screen yet again. Also, most consumers don't remember the not all  that great history of 3D television, and so they aren't put off by the fact that the last  few times this has been pushed, it wasn't very good. Display technology is changing  very quickly, and so this is an opportunity both to slip in 3D, since you're going to  be replacing your display anyway, and to change the ways in which we think about displaying  video. Unlike in the past, we don't just have fixed television broadcast standards that  are bringing us all our video. We have new flexible and extensible methods of representing  television, and they can carry 3D formats. On Internet video, you don't have the codec  hardwired into your display, you can download the codec over the network before you download  the video. Blu-ray discs have the ability to evolve, so you can change the features  of Blu-ray. They aren't fixed with a particular method of representing content.  Lots of things like shutter glasses or high refresh rate screens for temporarily multiplex  3D or high speed digital interconnects that used to be expensive and complicated are now  very cheap. So that's enabling us to do this as well. As we've seen with the example that  Frank put up on the stage, high quality cameras are now small and cheap, so you can make stereo  camera rigs practical and inexpensive. And there's a lot of content that's computer  graphics models, and so that can be re-rendered for 3D, and more particularly GPUs are fast  enough that you can do that in real time. Now GPUs are not only fast enough to give you  very, very realistic 3D video games, they're fast enough to give you realistic 3D scenes  in multiple views in real time. So there are lots of ways of transmitting 3D video from  a source to a destination. You can first of all just send multiple streams using something  like MPEG. You could interleave either spatially or temporally, or you can anamorphically squeeze  two frames into one frame and transmit that, or you can alternate frames. What's more interesting  and more efficient is when you code video frames to take advantage of the redundancy,  not just in time, but also from one view to another. I think many of you are familiar  how codecs like MPEG use motion vectors to predict one frame from its temporal neighbors.  Well, you can also predict one view from its spatial neighbors. And clearly, the future  is not going to be having 64 video cables going from 64 cameras to 64 projectors as  we have on the screen here. It's rather going to be coding all of those 64 views together  and sending them over one cable. And what has happened in the past year that's particularly  relevant to solving that problem is that H.264's AVC codec now has a backward compatible  amendment that supports multi-view predictive coding. So there is a mechanism in place for  sending two or more parallax views reasonably efficiently. Now, you can get more sophisticated  if you have more computation available and you have ways of generating a depth map, a  Z-buffer, you can send a 2D stream and a depth map, and then the expectation is the receiver  will compute the parallax views by shifting things differentially based on the Z values.  And ultimately, certainly most efficiently, if you have a way of generating it, you could  send the full 3D model and let the display render it appropriately. And there are more  technical details on these in the paper if you want to track them down. But there's an  important point behind all of these. And that is unlike in the past, 3D displays, even 2D  displays, are now assumed to have enough local processing that the format in which images  are transmitted is decoupled from the way in which the display handles them. So you  can have a lenticular display, but you could send it temporally multiplexed or anamorphically  squeezed frames and the expectation is it can convert them. So there are lots and lots  of activities. This is just from last year what has happened in 3D TV. And there's even  more than that. But again, the one I mentioned before is MPEG adopted an amendment to H.264  for multi-view video coding. It's called MVC. I think you're going to be hearing a  lot about that because that will be implemented pretty much everywhere.  Now, there are some other things that still need to happen. We need to have production  and post-production equipment for creating this multi-view or 3D model content. And ultimately,  people need to rediscover a creative and technical sense of what works and doesn't in the 3D world  because there's a lot of bad 3D out there and not pretty much good 3D out there. And  so some of what was known by certain low-cost cinematographers in the 1950s needs to be rediscovered.  I was heartened to note that there were starting to be books coming out this year on 3D moviemaking  again. And that's a topic that hasn't had books come out for a very long time.  Now, of course, this is the holy grail. Princess Leia on a table in front of you. Unfortunately,  I don't know where to buy the photons that will go halfway across the room and turn left.  If any of you knows where to get some, I'll take a bucket full. But this is certainly  what we'd really like in terms of 3D, in terms of holo video. What we get is CNN calling  this a hologram, which it definitely isn't. And enough said about that. So what holo  video really looks like is it looks like a hologram. It looks like the things that are  on display next door except that the hologram is moving. And the hologram can be behind  what you think of as the screen, which really isn't a screen at all. It can be in front.  It can straddle it. And so in this particular case, that object is actually a few centimeters  in front of the screen. And so I can appear to hold it in my hand.  Or to make this happen, of course, you need content. Now, I want to note, and this is  something that not a lot of people seem to have thought about. If you have a holo video  display, it's sort of a superset because of other 3D display technologies. Since you can  modulate light and you can steer it, you can take two or many parallax views. You can take  light fields from integral cameras. You can take a whole variety of ways of thinking about  3D imaging. And you could put them all in a holo video display if you can do the computation  locally to generate the diffraction pattern. You need a distribution mechanism. You need  computation at the display. And you need the electro-optics at the display that actually  turn the light into the hologram. And if 3D TV development is going to do some of these  for us, then people like me don't have to invent it all. I don't have to build the end-to-end  system. Rather, I can just take advantage of these things that other people are doing for  other purposes. And so that's been the goal in my group for the past several years is  to use as much as possible of what's already being deployed. And there are lots of people  who are working on holo video. And when I make this list by holo video, I mean actually  creating a diffraction pattern, illuminating it with monochromatic or coherent light, refreshing  it in real time. And I'm sorry if I didn't include some of you. We like to make the distinction  that our group has an emphasis on making this a near-term consumer technology. So we're  trying to make a holo video display that can display the 3D content that's going to be  around already that costs a few hundred dollars to build, that can use the kinds of interconnects  for video signals that are already going to be on your desktop or in your living room.  And in order to do that, you know, this is as close as we get to equations in this talk.  We need about two, if we have a screen that's a meter wide, holo video needs about two million  pixels on every scan line for that meter wide screen, which is one of the reasons that people  like me and people like Steve Benton spent a lot of time thinking about horizontal parallax  only because that reduces the problem somewhat. You still need two million pixels on a scan  line, but you don't also need a million scan lines in the other direction. The other thing  we need to do, we have to compute the diffraction pattern at the display. And what my group  has done over the past couple of years and was very nice to see a number of you doing  GPU programming as well is we're taking advantage of standard graphics processors in PCs and  game consoles, but we're not having them output just a 2D image or a pair of 2D images. We're  having them output a diffraction pattern. So if we magically tomorrow had a holo video  display in your living room, but we had existing stereoscopic TV being broadcast or sent over  the Internet, how would we use the holo video display to display a stereo pair? Well, the  first thing you could do is just display each view at a single angle. That works. The computation  is trivial for the diffraction pattern. Of course, depending on the spacing of your eyes,  you need to be at a specific distance from the screen as well as at a specific horizontal  position. We'd like to relax that. Since we can steer light, we could superpose multiple  diffraction patterns that would reconstruct the left and right views at multiple places  so that you don't have to be at a specific distance. You still have to be more or less  centered on the screen. What you then probably want to do is repeat this pair of left, right  views several times so the viewer could be in multiple locations or there could be multiple  viewers. Now, this isn't much harder to compute than that. And if we put a little gap, then  your eyes will naturally move to a place where you see both views, and this more or less guarantees  you won't see a pseudoscopic image because if you can see two views, you'll see proper  stereo. I think people have probably seen this photo. This is the Mark II display that  Pierre Saint-Hilaire and others built in the 1990s. It looks like a science experiment.  It's the size of a dining table. It has a 19-inch rack of electronics. It's about as  tall as I am. It's based on the scophany diffractive television technology which came from the  1930s. We want to move away from 1930s technology. This, by the way, is a still, I don't know  if any of you saw this film. This is called International House. It's a film starring  W.C. Fields about the dawn of television, and it takes place in China, and the television  system looks an awful lot like Mark II. I couldn't resist putting that there, but we  don't want that anymore. We want something that's a monitor that just plugs into your  PC. Of course, it won't look nearly as dramatic when we turn out the lights and you have this  red glow everywhere, but at the same time, it's a lot more practical.  What we have been doing for the past couple of years, and in particular Dan Smalley and  Quinn Smithwick, is building a new display architecture that's really simple, and unlike  every other scophany architecture display that's come before it, even though it is based  on an acoustic-optic modulator of special kind, it doesn't have horizontal mirrors. It just  has a vertical scanner, and you just sort of look down the optics toward the light modulator,  and you see a whole of video. And these are the specs. This is the proof of concept version.  We have a plan to scale this up. But the specs are pretty much the same as Mark II, except  that we can fold the optical path to fit in a box like a desktop CRT monitor, and it's  basically VGA resolution. It's 440 scan lines. Mark II is only 144. And it's based on these  very special surface acoustic wave light modulators made out of lithium niobate that Dan Smalley  is making for his doctoral dissertation. These are interesting because they're really cheap.  These cost maybe $10 to $20. We can make them ourselves at MIT, and they have over a giga  hertz of video bandwidth. So you can actually get hundreds of thousands of pixels on a scan  line. And we can even make them 2D. The way that they work is if you send a surface acoustic,  you have a wave guide just under the surface of the material. You couple in the beam to  the wave guide, and a single mode propagates down the wave guide. If you send a surface  acoustic wave across, you diffract the light. If you send a surface acoustic wave toward  the source, you actually change the mode. You convert the mode, and you can cause the  light to move up and down as well. What's more particularly relevant about these devices  for HoloVideo is they rotate the polarization. So the first order beam, which is the one  we want, has a different polarization from the zero order beam. So if you put a polarizer  at the output of it, you don't need a physical stop to get rid of the zero order light coming  out of here. So how do we render? Well, this is an extension of the diffraction specific  work that goes back to people like Mark Lucente. We're not doing a full physical model of  the scene and modeling the interference. We're rather starting with the diffraction pattern  and synthesizing a diffraction pattern that's needed to create a particular light field.  So you start with a chirped grating, which creates an effective emitter either in front  of or behind the screen. You modulate that chirped emitter with the intensity that you  want that emitter to have from multiple viewpoints, then you sum them all up and you get a horizontal  parallax only hologram. And that has a lot of advantages. You can pre-compute the chirps,  GPUs can do this in real time. It just, it does all of the things correctly that are  really hard in physically based models like occlusion and transparent surfaces and atmospheric  effects and lighting. And you can use standard lighting and texture representations. Indeed,  we, our rendering pipeline is just open GL at the front. So we can plug this into lots  of other pieces of software but output an HPO hologram rather than just a 2D view. There's  a natural place in the pipeline to insert parallax views. If you have pre-computed parallax  views from someplace else, there's a place in the rendering pipeline. You can just put  them in and there's a disadvantage. Like all holosteriograms, this doesn't provide, with  one exception that we know of, this doesn't provide accommodation cues consistent with  virgins. So your eyes are focused someplace different from where they're converged. And  one of the things that's really nice about holography is that we ought to be able to  make all of the cues to 3D, all of the perceptual cues for your eyes and your brain consistent  with one another. And that's a violation of that. Well, at Photonics West next January,  we're going to present a solution. We're going to present holographic stereograms that can  still be computed in real time but where each point in the image focuses wherever the object  is at that point. And so I will just won't explain the whole thing because I don't have  time but I'll give you a little hint. When you render the stereoscopic views from the  3D model using a virtual camera, you keep the Z-buffer. And then the Z-buffer tells  you where that, what depth that emitter is at at each angle. So it's not a particularly  hard thing and it's a process that the GPU is already doing anyway. So this January,  we reported that on GPUs, we've gone from one frame per second to greater than 15. We  actually have faster results than that now which we haven't yet published in the literature.  And so what I'd like to do is I would like to show you HoloVideo rendered in real time  from a 3D model. This is on the old Mark II display. So this is only 144 lines and this  is with a single NVIDIA GPU and the codec that we used to compress this did some violence  to it as well. So my apologies that this doesn't look all that good. If you find yourself in  Boston, let me know. We'll show you the real thing in a couple of months when the display  is a little brighter and more focused. We'll unveil Mark III and you'll see the same model  at standard definition TV resolution. I'll note that the renderer runs at full speed  on Mark III and we actually just sort of throttle it down for Mark II. But enough said about  that, here is a dinosaur running past some trees and that has about 30 degrees of look  around and that is texture mapped. The camera didn't capture it all that well. In conclusion,  if you haven't yet bought this book that Steve spent many years working on and Betsy  Connors and I and a couple of other people spent three years after that trying to package  and put together in a book, it's available and certain starving holographers like Betsy  will benefit if you purchase the book. So thank you very much for your attention and  I'll conclude there. Are there any questions at this point?  Michael, are you saying that at Photonix you'll be showing us the shoebox version of Mark  III? Is it going to be there or is it still going to require a superstructure that's not  portable? I think the main concern is that it might be a little bit too fragile to take  on an airplane and there will only be one of them then so I'm hesitant to have it leave  the building. We hope to have a lot of them at some point but we will definitely be showing  video of our new algorithm and I hope we'll be showing video of our new algorithm shot  from the screen of Mark III rather than Mark II.  And so what's the dimension of the desktop device in the end or do you expect that again?  The dimension is about the size of a 19-inch desktop CRT.  And so the business of plugging in existing models, I guess the model is to, you know,  in the same way that vector graphics, the way that flash works is that you download an  instruction set and then the data is actually already there in the device. Is that what  you're proposing? No, we're actually doing it more the way that  real-time gaming does it where you're updating the model. I mean, we assume you'll be transmitting  geometry, transmitting surface normals, transmitting texture maps and transmitting things like  directions for the virtual camera.  Michael, what is the display size? The display size for Mark III is just a  little bit smaller than Mark II so it's 80 by 60 by 80 millimeters.  Mark II is just a little bit larger than that. But you'll notice that the view volume is  actually as deep as the dimensions of the screen.  And when do you think we will have something like at least 15 inches of the RML display?  Well, full color. Well, full color turns out to be easy.  The first generation holo video display at MIT in 1990 was full color. I mean, that's  only a factor of three. Scale-up is what kills you because there's a square in there.  So when will we have bigger displays? Big... If you know of a source of very big pieces  of lithium niobate, we could actually make that relatively soon because Dan has an architecture  for putting multiple modulators on one monolithic piece of lithium niobate to make a direct  view display. But the biggest lithium niobate wafers we can get are not as big as that.  So, is there a source of lithium niobate as a limit? That's certainly... The lithium niobate  plus the wiring to it, there are some bandwidth problems, obviously. So right now, the Mark  III display uses two DVI cables from the PC. And if you imagine a tiled display, you would  need two DVI or HDMI cables for each tile of this size in order to have enough bandwidth  for that much parallax.  Okay, thank you.  I understood that Mark III display implements diffraction-specific approach, right?  Yes.  And did you make any improvements of this algorithm developed by Mark III Center?  Yes. If you look at the... We published a paper in optical engineering in 2007 about  something called the RIP algorithm, which is a significant improvement over the original  diffraction-specific algorithm. What we are now doing is a further improvement over the  RIP algorithm. And then what we'll be presenting next January is a further improvement over  that. So we have been continuing to improve the algorithm at the same time as we increase  its efficiency on the GPUs.  Thank you.  Mike, hi. I just wanted to thank you because I don't think your work can be underestimated.  I think it's one of the most important developments in the log-of-heat we have, you know, this  century. And I hope we'll live to see 3D TV before we... With much order. Thank you.  Well, thank you very, very much. I'm embarrassed.
3-D_TV_Preparing_the_Way_for_Holographic_TV__0_59.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__60_1475.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__1476_1703.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__1704_1751.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__1752_1835.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__1836_1907.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__1908_2243.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__2244_2447.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__2448_3359.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__3360_4463.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__4464_4919.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__4920_6683.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__6684_7055.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__7056_7631.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__7632_8303.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__8304_10247.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__10248_10631.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__10632_11375.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__11376_11987.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__11988_12995.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__12996_13631.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__13632_13847.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__13848_14519.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__14520_16223.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__16224_17219.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__17220_18731.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__18732_19247.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__19248_20075.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__20076_20771.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__20772_21455.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__21456_22439.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__22440_23183.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__23184_23807.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__23808_24503.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__24504_25775.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__25776_26891.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__26892_29855.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__29856_30191.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__30192_31931.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__31932_32975.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__32976_33107.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__33108_35927.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__35928_35975.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__35976_39347.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__39348_40547.png 3-D_TV_Preparing_the_Way_for_Holographic_TV__40548_40596.png