Hybrid reality: Using 2D and 3D together in a mixed mode display
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hoffmeister, Kurt M.
Critical collaborative work session rely on sharing 2D and 3D information. Limitations in display options make it difficult to share and interact with multiple content types content simultaneously. As a result, displays capable of showing stereoscopic content are predominately used for 2D applications. This presentation will illustrate Hybrid Reality—a strategy for showing and interacting with 2D and 3D content simultaneously on the same display. Example displays, use cases, and case studies will be discussed. By using the Hybrid Reality environment, manufacturing organizations have achieved ROI with time and cost savings as well as improved collaboration for complex design problems. In higher education, Hybrid Reality displays support instruction and curriculum design by providing a process to share a wide spectrum of 2D and 3D media and applications into classroom setting. This presentation will share detailed case studies of both applications. This presentation will demonstrate how a Hybrid Reality display system can be used to effectively combine 2D and 3D content and applications for improved understanding, insight, decision making, and collaboration. 
 Our next paper is titled Hybrid Reality using 2D and 3D together in a mixed mode display. Our speaker is Kurt Hofmeister of Mekdine Corporation where he is Vice President of Technology and Innovation and he's also a co-founder. Good morning. This morning I want to talk about hybrid reality environments but first I'll give you just a quick slide background on Mekdine. Those of you not familiar with Mekdine were a turnkey solution provider for display systems, visualization software and professional services around that. Our experience goes back to 1988. We've seen the rise and fall in variations of 3D and industry and with installations around the world we've certainly seen a wide variety of applications. Just to start off here, Hybrid Reality Environment has several common features. Basically it combines 2D and 3D content and applications, supports immersive interactive visualization but also includes some of the familiar desktop applications and interaction paradigms there. Also it supports a wide variety of media and more importantly multiple participants in a collaborative type work environment. So for an agenda besides a quick introduction here I'm going to take a quick look at history, some of the triggers in the market that have moved these hybrid reality systems to the forefront where we are today with a couple examples and challenges for tomorrow. So let's start clear back in 1992 with the cave coming out of Electronic Visualization Lab University of Illinois. We're lucky to have a few people, Dan Sandin and maybe Carolina Cruz with us this morning who are instrumental along with others in starting up that system. Couple things that jump out to me about the cave too is it quickly became a gold standard for virtual reality, virtual environment type applications and it's still popular today, 20 or more like 25 years after the first system. But it was still a purpose built device monthly for stereoscopic visualization virtuality. It was typically connected to a very expensive supercomputer for graphics. The SGI typically was a major part of any investment and yet the resolution was relatively low at the time, 10 pixels per inch type resolution no more than a megapixel per wall initially. And this was a real challenge for showing text, icons, menus, the data behind the visualization sometimes was difficult to betray or work with. Even so over the years there was a steady increase in resolution from that one megapixel up to 16 megapixels per wall and beyond that's kind of constantly evolved. Around the late 1990s we started building system called a flexible cave. The intent was that you could have a fully immersive and closing cave where you were inside the data or you could open up the sidewalls and had one triple wide display where you might do 3D visualization but less immersive more for presenting to a group. What we quickly found was customers using this type system oftentimes only opened one wall. They kept a corner cave for immersive application and would use a secondary computer for this sidewall usually showing nothing more than 2D applications, PowerPoint presentations, spreadsheets, graphs, things that complemented what was being shown in the immersive simulation. So some of the triggers that became pretty apparent in the early 2000s were of course the increasing resolution and brightness. We also saw the rise of powerful graphics computers and the kind of fading out of SGI and sun super graphics computers. At the same time there was this move from cave type systems being a research device to becoming part of the daily workflow in many of the companies we worked with. So being part of that daily workflow there was a pretty high expectation to support desktop applications, different types of collaboration and your typical office uses. One thing interesting to note in 2003, many universities, Mekdine and other companies began to experiment with OpenGL Interception. I don't want to get too far off in the weeds here but the beauty of OpenGL Interception was you could run a desktop application like a CAD application that used OpenGL graphics and whether the application was aware or not you could intercept calls to the graphics library, send those to a visualization cluster and re-render it for a cave. We could take an application at that time that was not intended for stereoscopic use, make it stereoscopic, immersive, interaction, head tracking, all those things could be then added to the application. Quickly expanded the way a cave or a hybrid reality type system would then be used. So we quickly jumped to several caves now around the world that have 4K resolution, 4K by 4K on each wall. If we kind of contrast that quickly, it's a 16-fold resolution increase per surface. We went from a fairly coarse data representation where text and 2D data was difficult to combine to the extreme case where without some degree of filtering and grouping of data you could display now clouds of data so dense it became difficult to work with. And so oftentimes there'd be a 2D component of these applications even though they were immersive once you started filtering you would see 2D representation of some data subset as part of the actual environment. At the same time an early adopter of 3D visualization was the oil and gas industry and instead of partitioning their display, this is like a large three channel curve screen, they really drove forward a windowing type approach. So the curve screen was treated as one big desktop and everything on it was actually a window. It could be size and position wherever you wanted. So although there might be a stereoscopic application or even something more fully immersive, it was oftentimes intermixed with video and models and other 2D data sources. So this became a true model of a hybrid environment but typically all these windows would be related to whatever the topic at hand was but they didn't necessarily interact with each other. So the simulation would be separate from a model of the rig or some of the data being shown at the same time. We also saw a few changes in the driving mechanism behind these displays, moving from projectors which we had from the very beginning to projection cubes and today a fair amount of use after the implementation of 3D TVs, the LCD flat panels going forward OLED flat panels still with some bezel. And I really predict by the end of this year, early next year, we're going to start to see displays built from DirectView LED 3D capable used for these same hybrid environments and I'd love to talk to any of you about that more after the talk this morning. So jump forward, here's an example of a University of Arkansas research lab flexible cave even though this cave uses an array of eight projection cubes per wall, it's still often used where there's an immersive application running on part of the cave and 2D presentation segmented off to one side. It really wasn't until 2012 when University of Illinois Electronic Visualization Lab began developing cave 2 that we saw a lot of these things for hybrid use really, really come together. The cave 2 if you haven't experienced one, you're certainly welcome to visit Mekdyne or I can't speak for Electronic Visualization Lab but it's quite a system to see it's 25 foot diameter, multiple columns of flat panel arrays and some important things here is it still supports 3D immersive applications and typically it supports some type of multimedia windowing at the same time. There's maybe six of these in the world now and smaller subsets that might be a half or a quarter of the full cave 2. Interesting note cave 2 is a trademark registered to the University of Illinois just like the name cave, I hope cave 2 becomes as generic and wide used as cave has become. So to follow on from that previous example, this is a working session where they're exploring a simulation I think it's crack propagation in a glass crystal and structure. You can imagine quite a few different scenarios around this but you have one or more stereoscopic visualizations going on. You have representations of the data through different modeling, of course they brought in a paper whether they're reviewing that for reference or maybe they're actually working on the content that goes into that paper. You can really see a wide variety of media and how people are engaged here. Because it's a windowing type system, each of these participants may actually be doing some portion of the interaction through their own laptop on a portion of the screen. So it's truly a hybrid approach depending on what resources are brought into the system. A good example, I'm far from an expert on this so I can give a few high points. The NASA endurance project lowered an autonomous vehicle into Botany Lake in the Antarctica and began doing a sonar scans mapping and water data sampling. Early on they knew ahead of time but they quickly realized that the salinity of the water was a challenge for the sonar data. In near real time with feedback, they were visualizing the data as they got it while processing it, visualizing it and making decisions about changing parameters on how the sonar was collected to improve the integrity of the data. At the same time they had this visualization running with the white markers you see that were known depth markers so they could also kind of validate accuracy of the data as it was coming in. Later on this system went to kind of a post-processing phase. That's another great example of the hybrid use of the system. You can see the gentleman with dark hair in the center is running an immersive application looking at point clouded data. At the same time a lot of the other participants are looking off to the left at more 2D representations of the data and what they're doing is they're identifying potentially bad data points, trying to make a decision about how well that fits with the surrounding data and whether or not to exclude it from the data set. So this is a very interactive cleanup phase making full use of the system. To go back to one of the flexible cave systems, this is a nice example from Hemholtz Center for Environmental Research. This is a newer oil and gas application where you have seismic representation and then an oil field with a bunch of wells. What's really nice about this application is it combines the 3D immersive part that what you, where you position yourself and what you select in the virtual environment, determines what's shown on the two side screens in terms of a map and data representation. So all your 2D data in this case isn't just pulled up separate, it's keyed into the application. This application does a great job of combining the two very effectively. Some of the cave two systems that I'm aware of, Mekdine provided at least three of them, are used primarily for education and research environments. So the picture to the left there is University of Illinois Electronic Visualization Lab lecture taking place in the cave two. Again this is multimedia content, some of the things on the screen are actually being brought up by students from their own laptop, some of it's being served from the system that drives the display. So a real good mix going on there. The image to the upper right is Sunshine Coast University in Eastern Australia. This is a flipped classroom model where there's work tables scattered throughout the room, but they all tie back to one large immersive display that's 3D touch, cannot often be used for lecture sharing content from one or more of the student work tables out in the room. The bottom right picture would be typical PowerPoint presentation like we use today, but there's other tools out there Prezi is one no one has used at this conference that I was aware of yet, that is more of a web based PowerPoint that could really effectively take use of this pixel real estate on a large display. And I'd certainly challenge folks that tools like that that do your presentation on a large pixel space are somewhat lacking or custom developed. Even if we had more of those tools the interaction is still a challenge, often times the interaction paradigm comes down to the specific application. Yet the number of things we have in terms of interactive wands, touch, gesture, we all carry smartphones our own personal interaction device and of course voice. Those things are not necessarily intuitive to use in the systems that make use of them have a challenge for help and guidance and using those things. Of course my pitch I think it's important to use technology integrator who's familiar with a wide range of technologies to effectively apply some of these things as well. So main challenges going forward would be there certainly is no universal control or interaction paradigm. There's a rich area of work to go on there. Multi-participant interaction, most of the applications I've worked with really are centered around a single point of interaction. At least with a windowing control, a windowing system overlaid on it, you can have multiple content and multiple participants interacting with their own content but that's kind of the extent of it. So there's definitely continued challenges around integrating 2D and 3D applications. Final thing, the broader reality most of us collaborate with remotely scattered groups. These systems definitely need more work for supporting remote participants. Thank you. I hope I have a moment for any questions.
 Our next paper is titled Hybrid Reality using 2D and 3D together in a mixed mode display.  Our speaker is Kurt Hofmeister of Mekdine Corporation where he is Vice President of Technology  and Innovation and he's also a co-founder.  Good morning.  This morning I want to talk about hybrid reality environments but first I'll give you just  a quick slide background on Mekdine.  Those of you not familiar with Mekdine were a turnkey solution provider for display systems,  visualization software and professional services around that.  Our experience goes back to 1988.  We've seen the rise and fall in variations of 3D and industry and with installations  around the world we've certainly seen a wide variety of applications.  Just to start off here, Hybrid Reality Environment has several common features.  Basically it combines 2D and 3D content and applications, supports immersive interactive  visualization but also includes some of the familiar desktop applications and interaction  paradigms there.  Also it supports a wide variety of media and more importantly multiple participants in  a collaborative type work environment.  So for an agenda besides a quick introduction here I'm going to take a quick look at history,  some of the triggers in the market that have moved these hybrid reality systems to the  forefront where we are today with a couple examples and challenges for tomorrow.  So let's start clear back in 1992 with the cave coming out of Electronic Visualization  Lab University of Illinois.  We're lucky to have a few people, Dan Sandin and maybe Carolina Cruz with us this morning  who are instrumental along with others in starting up that system.  Couple things that jump out to me about the cave too is it quickly became a gold standard  for virtual reality, virtual environment type applications and it's still popular today,  20 or more like 25 years after the first system.  But it was still a purpose built device monthly for stereoscopic visualization virtuality.  It was typically connected to a very expensive supercomputer for graphics.  The SGI typically was a major part of any investment and yet the resolution was relatively  low at the time, 10 pixels per inch type resolution no more than a megapixel per wall initially.  And this was a real challenge for showing text, icons, menus, the data behind the visualization  sometimes was difficult to betray or work with.  Even so over the years there was a steady increase in resolution from that one megapixel  up to 16 megapixels per wall and beyond that's kind of constantly evolved.  Around the late 1990s we started building system called a flexible cave.  The intent was that you could have a fully immersive and closing cave where you were  inside the data or you could open up the sidewalls and had one triple wide display where you might  do 3D visualization but less immersive more for presenting to a group.  What we quickly found was customers using this type system oftentimes only opened one  wall.  They kept a corner cave for immersive application and would use a secondary computer for this  sidewall usually showing nothing more than 2D applications, PowerPoint presentations,  spreadsheets, graphs, things that complemented what was being shown in the immersive simulation.  So some of the triggers that became pretty apparent in the early 2000s were of course  the increasing resolution and brightness.  We also saw the rise of powerful graphics computers and the kind of fading out of SGI  and sun super graphics computers.  At the same time there was this move from cave type systems being a research device  to becoming part of the daily workflow in many of the companies we worked with.  So being part of that daily workflow there was a pretty high expectation to support  desktop applications, different types of collaboration and your typical office uses.  One thing interesting to note in 2003, many universities, Mekdine and other companies  began to experiment with OpenGL Interception.  I don't want to get too far off in the weeds here but the beauty of OpenGL Interception  was you could run a desktop application like a CAD application that used OpenGL graphics  and whether the application was aware or not you could intercept calls to the graphics library,  send those to a visualization cluster and re-render it for a cave.  We could take an application at that time that was not intended for stereoscopic use,  make it stereoscopic, immersive, interaction, head tracking, all those things could be then  added to the application.  Quickly expanded the way a cave or a hybrid reality type system would then be used.  So we quickly jumped to several caves now around the world that have 4K resolution,  4K by 4K on each wall.  If we kind of contrast that quickly, it's a 16-fold resolution increase per surface.  We went from a fairly coarse data representation where text and 2D data was difficult to combine  to the extreme case where without some degree of filtering and grouping of data you could  display now clouds of data so dense it became difficult to work with.  And so oftentimes there'd be a 2D component of these applications even though they were  immersive once you started filtering you would see 2D representation of some data subset  as part of the actual environment.  At the same time an early adopter of 3D visualization was the oil and gas industry and instead of  partitioning their display, this is like a large three channel curve screen, they really  drove forward a windowing type approach.  So the curve screen was treated as one big desktop and everything on it was actually  a window.  It could be size and position wherever you wanted.  So although there might be a stereoscopic application or even something more fully immersive,  it was oftentimes intermixed with video and models and other 2D data sources.  So this became a true model of a hybrid environment but typically all these windows would be related  to whatever the topic at hand was but they didn't necessarily interact with each other.  So the simulation would be separate from a model of the rig or some of the data being  shown at the same time.  We also saw a few changes in the driving mechanism behind these displays, moving from projectors  which we had from the very beginning to projection cubes and today a fair amount of use after  the implementation of 3D TVs, the LCD flat panels going forward OLED flat panels still  with some bezel.  And I really predict by the end of this year, early next year, we're going to start to see  displays built from DirectView LED 3D capable used for these same hybrid environments and  I'd love to talk to any of you about that more after the talk this morning.  So jump forward, here's an example of a University of Arkansas research lab flexible cave even  though this cave uses an array of eight projection cubes per wall, it's still often used where  there's an immersive application running on part of the cave and 2D presentation segmented  off to one side.  It really wasn't until 2012 when University of Illinois Electronic Visualization Lab  began developing cave 2 that we saw a lot of these things for hybrid use really, really  come together.  The cave 2 if you haven't experienced one, you're certainly welcome to visit Mekdyne  or I can't speak for Electronic Visualization Lab but it's quite a system to see it's 25  foot diameter, multiple columns of flat panel arrays and some important things here is it  still supports 3D immersive applications and typically it supports some type of multimedia  windowing at the same time.  There's maybe six of these in the world now and smaller subsets that might be a half or  a quarter of the full cave 2.  Interesting note cave 2 is a trademark registered to the University of Illinois just like the  name cave, I hope cave 2 becomes as generic and wide used as cave has become.  So to follow on from that previous example, this is a working session where they're exploring  a simulation I think it's crack propagation in a glass crystal and structure.  You can imagine quite a few different scenarios around this but you have one or more stereoscopic  visualizations going on.  You have representations of the data through different modeling, of course they brought  in a paper whether they're reviewing that for reference or maybe they're actually working  on the content that goes into that paper.  You can really see a wide variety of media and how people are engaged here.  Because it's a windowing type system, each of these participants may actually be doing  some portion of the interaction through their own laptop on a portion of the screen.  So it's truly a hybrid approach depending on what resources are brought into the system.  A good example, I'm far from an expert on this so I can give a few high points.  The NASA endurance project lowered an autonomous vehicle into Botany Lake in the Antarctica  and began doing a sonar scans mapping and water data sampling.  Early on they knew ahead of time but they quickly realized that the salinity of the  water was a challenge for the sonar data.  In near real time with feedback, they were visualizing the data as they got it while  processing it, visualizing it and making decisions about changing parameters on how the sonar  was collected to improve the integrity of the data.  At the same time they had this visualization running with the white markers you see that  were known depth markers so they could also kind of validate accuracy of the data as it  was coming in.  Later on this system went to kind of a post-processing phase.  That's another great example of the hybrid use of the system.  You can see the gentleman with dark hair in the center is running an immersive application  looking at point clouded data.  At the same time a lot of the other participants are looking off to the left at more 2D representations  of the data and what they're doing is they're identifying potentially bad data points,  trying to make a decision about how well that fits with the surrounding data and whether  or not to exclude it from the data set.  So this is a very interactive cleanup phase making full use of the system.  To go back to one of the flexible cave systems, this is a nice example from Hemholtz Center  for Environmental Research.  This is a newer oil and gas application where you have seismic representation and then an  oil field with a bunch of wells.  What's really nice about this application is it combines the 3D immersive part that  what you, where you position yourself and what you select in the virtual environment,  determines what's shown on the two side screens in terms of a map and data representation.  So all your 2D data in this case isn't just pulled up separate, it's keyed into the application.  This application does a great job of combining the two very effectively.  Some of the cave two systems that I'm aware of, Mekdine provided at least three of them,  are used primarily for education and research environments.  So the picture to the left there is University of Illinois Electronic Visualization Lab  lecture taking place in the cave two.  Again this is multimedia content, some of the things on the screen are actually being  brought up by students from their own laptop, some of it's being served from the system  that drives the display.  So a real good mix going on there.  The image to the upper right is Sunshine Coast University in Eastern Australia.  This is a flipped classroom model where there's work tables scattered throughout the room,  but they all tie back to one large immersive display that's 3D touch, cannot often be used  for lecture sharing content from one or more of the student work tables out in the room.  The bottom right picture would be typical PowerPoint presentation like we use today,  but there's other tools out there Prezi is one no one has used at this conference that  I was aware of yet, that is more of a web based PowerPoint that could really effectively  take use of this pixel real estate on a large display.  And I'd certainly challenge folks that tools like that that do your presentation on a large  pixel space are somewhat lacking or custom developed.  Even if we had more of those tools the interaction is still a challenge, often times the interaction  paradigm comes down to the specific application.  Yet the number of things we have in terms of interactive wands, touch, gesture, we all  carry smartphones our own personal interaction device and of course voice.  Those things are not necessarily intuitive to use in the systems that make use of them  have a challenge for help and guidance and using those things.  Of course my pitch I think it's important to use technology integrator who's familiar  with a wide range of technologies to effectively apply some of these things as well.  So main challenges going forward would be there certainly is no universal control or  interaction paradigm.  There's a rich area of work to go on there.  Multi-participant interaction, most of the applications I've worked with really are centered  around a single point of interaction.  At least with a windowing control, a windowing system overlaid on it, you can have multiple  content and multiple participants interacting with their own content but that's kind of  the extent of it.  So there's definitely continued challenges around integrating 2D and 3D applications.  Final thing, the broader reality most of us collaborate with remotely scattered groups.  These systems definitely need more work for supporting remote participants.  Thank you.  I hope I have a moment for any questions.
SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_0_143.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_144_287.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_288_503.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_504_1391.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_1392_2207.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_2208_2267.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_2268_2375.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_2376_2651.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_2652_3191.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_3192_5027.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_5028_6443.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_6444_8975.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_8976_9227.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_9228_10355.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_10356_12071.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_12072_13223.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_13224_13907.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_13908_15635.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_15636_17147.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_17148_18683.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_18684_19811.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_19812_21227.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_21228_23879.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_23880_24911.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_24912_26231.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_26232_26387.png SD_A_2016__Hybrid_reality__using_2D_and_3D_together_in_a_mixed_mode_display_xFGzcMKAOeM_26388_26758.png