Disparity remapping to ameliorate visual comfort of stereoscopic video
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hosik, Sohn
Jung, Yong Ju
Lee, Seong-il
Ro, Yong Man
The great success of the three-dimensional (3D) digital cinema industry has opened up a new era of 3D content services. While we have witnessed a rapid surge of stereoscopic 3D services, the issue of viewing safety remains a possible obstacle to the widespread deployment of such services. In this paper, we propose a novel disparity remapping method to reduce the visual discomfort induced by fast change in disparity. The proposed remapping approach selectively adjusts the disparities of the discomfort regions where the fast change in disparity occurs. To this purpose, the proposed approach detects visual importance regions in a stereoscopic 3D video, which may have dominant influence on visual comfort in video frames, and then locally adjust the disparity by taking into account the disparity changes in the visual importance regions. The experimental results demonstrate that the proposed approach to adjust local problematic regions can improve visual comfort while preserving naturalness of the scene. 
 Thank you for giving me an opportunity to present my work. My name is Ho-Sik Son in Kais, South Korea. The title of my paper is Dispect Remapping to Emulate Visual Comfort of Stereoscopic Video. For a couple of years, we have been working on the objective visual comfort metric to predict visual discomfort of 3D contents. Our research interests naturally to the question how to improve the visual comfort by adjusting the disparity of 3D contents. This is our pilot result of our research. The investigation of 3D industry has been reality with great economic success, as we know. We have between us great efforts by the industry to extend such success to the other 3D applications such as mobile and gaming or online 3D video. And generally, this agreed that a possible obstacle to the widespread of 3D services is related with the visual comfort issue. Indeed, the industry has dedicated their effort to address these concerns. However, the solution provided requires time-consuming and labor-intensive processing. As a result, the development of automatic tools to improve visual comfort has been gained this importance. So in stereoscopic 3D contents, excessive disparity magnitude and fast change of disparity have been known as key determinants of visual discomfort. As we know, the stereoscopic being induced accommodation versus conflict, and this conflict worsens with excessive disparity. Especially this conflict, as disparity may increase, the degree of conflict also increases so that more visual discomfort can be induced. In addition, fast change of disparity can also induce visual discomfort, possibly due to excessive demand on accommodation and versions of the linkage. So currently, disparity remapping has been suggested as a one-way to improve visual comfort of 3D contents. And in most cases, disparity remapping has been used to minimize discomfort by excessive disparity. So current one common approach is to target an internal disparity range into a comfortable range. So however, the problem is that even the same disparity is within the comfortable range, visual discomfort can be still induced. Right here. So for instance, like fast change of disparity or excessive disparity gradient or high spatial frequency, so we have many of additional content characters that can affect our visual comfort. So considering that the severity of visual discomfort is highly correlated with disparity magnitude, we can further decrease the same disparity range to minimize the negative effect those discomfort causes on visual comfort. However, in this case, the problem is that so by decreasing the disparity range of scene, we can minimize the, we can improve the visual comfort, but it can result in degradation in the depth quality and perceived disparity range, the perceived depth range. So the motivation of our work is to locally scale the disparity of local problematic reasons. So in this case, in this way, we can improve visual comfort while preserving the depth, perceived depth range. So the contribution of our paper is to, is a local disparity remapping that aims at reducing the visual comfort in this way, fast change of disparity in stereoscopic video. So to this purpose, we detect visual importance region, which have a dominant influence on our visual comfort and quantifies the level of visual discomfort in those regions. And then the local disparity remapping is generally to compress the local problematic reasons. This slide shows our framework of local disparity remapping. We first detect visual importance region from the in post-tearscopic video. And as aforementioned, visual importance region represents the reason where, which have a dominant influence on our visual comfort. And by analyzing the motion and disparity characteristics of those regions, we quantify the degree of visual discomfort in this by those regions. And local disparity remapping function is generated to compress the problematic disparity plane according to the severity of visual discomfort. So in order to detect visual discomfort region, we employed computational visual attention model. Especially, we used image motion and depth based image map to generate visual importance map. And this figure shows the example of the image motion, depth standard map of in post-tearscopic video. And considering that different modalities, and finally contribute visual comfort map, we combined this standard map using a linear combination with the conveyance. Then using a simple threshold segmentation, we obtained visual importance region from the input video. Then in order to quantify the visual discomfort in those regions, we measured the magnitude of in depth motion at each visual importance region for each frame. Then we can obtain disparity features for multiple reasons for each frame. So here we assume that the most problematic reason can dominant influence the overall visual comfort. So we took the maximum pulling strategy. So we chose the maximum feature value from each frame. Then to further quantify the visual discomfort level, we have seen individual disparity planes across different frames. So from we have multiple, especially for the feature values for several frames, and we divided that feature values according to their disparity of influence region. So here the QT represented average disparity of visual importance region where the special field motion feature was extracted. So we divide this feature value into the multiple sets, and among them we took the maximum value with an assumption of the winner-taker mechanism in temporal domain. So finally, using this information, we quantified the relative level of visual discomfort across different disparity planes in video by the normalization. This is a simple example to represent the process of visual discomfort quantification. So here in this video, the most foreground butterfly was detected as visual importance region which have very large disparity magnitude and the best change of depth of motion. So the first figure shows the special field motion feature value which approximate the magnitude of in-depth motion, and second figure shows the average disparity magnitude of visual importance region. So finally, the third figure shows the relative level of visual discomfort across different frames within a shot of the input video. We can see that since a first change of disparity occurs between the disparity range of 3 degrees to 3.5 degrees, the relative level of visual discomfort was very severe around this disparity values. So now we have the information of relative visual discomfort of the search curve video. So we use this information to compress the disparity of the search curve video. So the basic concept of the proposed method is to compress the local disparity planes according to the relative level of visual discomfort. So here the C is the information of the discomfort level we obtained in the previous slide. So we use this information to determine the amount of disparity compression at each disparity plane. So here this figure shows the example of disparity function that compress just one disparity plane. So here the x-axis represents the original disparity and y-axis represents the remap disparity. So here for disparity plane DI, the amount of disparity compression is determined by the CDI. So in this way we compress the each disparity plane according to the severe visual discomfort. And here the W, this parameter is a weight of factor that determines the absolute value that disparity to be compressed. So since each disparity plane has different amount of disparity to be compressed and compressing specific disparity planes affect the others, so it isn't necessary to determine the proper disparity value. Since we have the multiple choice for the remap disparity plane, in this case we took the minimum disparity value. So this is the example for just only close disparity. So the same the mechanism applied to the uncrossed disparity. So the reason why we took the minimum value for this interval is that we need to sufficiently compress the disparity of problematic disparity plane. And this is the final example of the disparity, local disparity mapping. So the left figure shows the relative level of visual discomfort and the right figure shows the local disparity mapping function we generated using the method previously explained. So we can see that disparity between 3 degree to 3.5 degree has been compressed. So to demonstrate the performance of the proposed local disparity mapping function, we used 18-sterescopic video which consists of 9 scenes with 2 different camera base lines. And this thumbnail shows an example of the star-screw video we used for our experiment. And these star-screw videos were generally to have very excessive disparity range and a vast change of disparity. So for our experiment, we applied two different disparity mapping methods. The first one was the global disparity mapping which linearly scales the disparity range of star-screw video. And the second one was combined global and local disparity mapping. So by comparing the subject's curve between these two sets, we proved that the additional local disparity mapping can improve visual comfort while preserving the naturalness of the scene. So to compress the disparity of the scene, it is necessary to determine the proper amount of disparity compression. So to this purpose, we used object visual comfort metric which is our previous work. So for global disparity mapping, we used a visual input space we saved for disparity magnitudes which utilize the disparity statistical visual importance region to predict the overall level of visual comfort of the input stereoscopic 3D contents. Then for the second video set, we applied the visual input space we saved for disparity magnitudes change. So according to the prediction result, the absolute amount of disparity compression is determined. Then we applied the proposed local disparity mapping method. This is a very simple example of the star-screw video after two different disparity mapping. The first one is global disparity mapping and the second one is combined global and local disparity mapping. So here you can see that after local disparity, here I record that the foreground butterfly is the visual input region in this video. So the local disparity mapping compress the most problematic region here, the butterfly. So you can see that disparity of this butterfly is decreased and while the disparity of background like tree is almost preserved. To demonstrate the local disparity mapping, we conducted our subjective assessment of visual comfort and naturalness. We used DSC-Case which randomly displayed two different versions of the same image. The one was the original image and the other was the process image by the global remapping or combined global and local disparity mapping. So we measured the visual comfort and naturalness using this score sheet. And for the analysis of the result, we converted this continuous scale from 0 to 10 and demo so as used for the analysis. This is the result of subject assessment result of visual comfort. And here the excess is represented the content index and why excess is represented the most. So in here the empty, the B ins represent the improvement of visual comfort obtained by global disparity mapping and the field bar represent the one obtained by the combined global and local approach. So we observed that both global and combined approach improved the visual comfort with statistical significance. And for global approach, average demo was 8.5 and for the combined approach, average demo was 11. And the difference of these two scores was statistically significant. And for naturalness, you observed that the overall improvement of naturalness was significant but we could not observe any difference between global and combined approach. So in this paper, we proposed the local disparity mapping to compress the local problematic display plan in the statistical video and to this purpose, we detect visual implications and quantify the level of discomfort induced by those reasons. And our subject assessment shows that the proper local disparity mapping can improve visual comfort while preserving the naturalness of scene. Okay. Okay, that's all. Thank you. Thank you. Maybe one quick question if you have any questions? So do I understand that by local, you mean local in depth disparity? Right. So you change some range of depth to some other range of depth? Yeah, right. So here local does not mean special information but depth information. So the local is in Z. Yeah, right. So please start to speak. Thank you.
 Thank you for giving me an opportunity to present my work.  My name is Ho-Sik Son in Kais, South Korea.  The title of my paper is Dispect Remapping to Emulate Visual Comfort of Stereoscopic  Video.  For a couple of years, we have been working on the objective visual comfort metric to  predict visual discomfort of 3D contents.  Our research interests naturally to the question how to improve the visual comfort by adjusting  the disparity of 3D contents.  This is our pilot result of our research.  The investigation of 3D industry has been reality with great economic success, as we  know.  We have between us great efforts by the industry to extend such success to the other 3D applications  such as mobile and gaming or online 3D video.  And generally, this agreed that a possible obstacle to the widespread of 3D services  is related with the visual comfort issue.  Indeed, the industry has dedicated their effort to address these concerns.  However, the solution provided requires time-consuming and labor-intensive processing.  As a result, the development of automatic tools to improve visual comfort has been gained  this importance.  So in stereoscopic 3D contents, excessive disparity magnitude and fast change of disparity  have been known as key determinants of visual discomfort.  As we know, the stereoscopic being induced accommodation versus conflict, and this conflict  worsens with excessive disparity.  Especially this conflict, as disparity may increase, the degree of conflict also increases  so that more visual discomfort can be induced.  In addition, fast change of disparity can also induce visual discomfort, possibly due  to excessive demand on accommodation and versions of the linkage.  So currently, disparity remapping has been suggested as a one-way to improve visual comfort  of 3D contents.  And in most cases, disparity remapping has been used to minimize discomfort by excessive  disparity.  So current one common approach is to target an internal disparity range into a comfortable  range.  So however, the problem is that even the same disparity is within the comfortable range,  visual discomfort can be still induced.  Right here.  So for instance, like fast change of disparity or excessive disparity gradient or high spatial  frequency, so we have many of additional content characters that can affect our visual comfort.  So considering that the severity of visual discomfort is highly correlated with disparity  magnitude, we can further decrease the same disparity range to minimize the negative effect  those discomfort causes on visual comfort.  However, in this case, the problem is that so by decreasing the disparity range of scene,  we can minimize the, we can improve the visual comfort, but it can result in degradation  in the depth quality and perceived disparity range, the perceived depth range.  So the motivation of our work is to locally scale the disparity of local problematic reasons.  So in this case, in this way, we can improve visual comfort while preserving the depth,  perceived depth range.  So the contribution of our paper is to, is a local disparity remapping that aims at reducing  the visual comfort in this way, fast change of disparity in stereoscopic video.  So to this purpose, we detect visual importance region, which have a dominant influence on  our visual comfort and quantifies the level of visual discomfort in those regions.  And then the local disparity remapping is generally to compress the local problematic  reasons.  This slide shows our framework of local disparity remapping.  We first detect visual importance region from the in post-tearscopic video.  And as aforementioned, visual importance region represents the reason where, which have a  dominant influence on our visual comfort.  And by analyzing the motion and disparity characteristics of those regions, we quantify  the degree of visual discomfort in this by those regions.  And local disparity remapping function is generated to compress the problematic disparity  plane according to the severity of visual discomfort.  So in order to detect visual discomfort region, we employed computational visual attention  model.  Especially, we used image motion and depth based image map to generate visual importance  map.  And this figure shows the example of the image motion, depth standard map of in post-tearscopic  video.  And considering that different modalities, and finally contribute visual comfort map,  we combined this standard map using a linear combination with the conveyance.  Then using a simple threshold segmentation, we obtained visual importance region from  the input video.  Then in order to quantify the visual discomfort in those regions, we measured the magnitude  of in depth motion at each visual importance region for each frame.  Then we can obtain disparity features for multiple reasons for each frame.  So here we assume that the most problematic reason can dominant influence the overall  visual comfort.  So we took the maximum pulling strategy.  So we chose the maximum feature value from each frame.  Then to further quantify the visual discomfort level, we have seen individual disparity planes  across different frames.  So from we have multiple, especially for the feature values for several frames, and we  divided that feature values according to their disparity of influence region.  So here the QT represented average disparity of visual importance region where the special  field motion feature was extracted.  So we divide this feature value into the multiple sets, and among them we took the maximum value  with an assumption of the winner-taker mechanism in temporal domain.  So finally, using this information, we quantified the relative level of visual discomfort across  different disparity planes in video by the normalization.  This is a simple example to represent the process of visual discomfort quantification.  So here in this video, the most foreground butterfly was detected as visual importance  region which have very large disparity magnitude and the best change of depth of motion.  So the first figure shows the special field motion feature value which approximate the  magnitude of in-depth motion, and second figure shows the average disparity magnitude of visual  importance region.  So finally, the third figure shows the relative level of visual discomfort across different  frames within a shot of the input video.  We can see that since a first change of disparity occurs between the disparity range of 3 degrees  to 3.5 degrees, the relative level of visual discomfort was very severe around this disparity  values.  So now we have the information of relative visual discomfort of the search curve video.  So we use this information to compress the disparity of the search curve video.  So the basic concept of the proposed method is to compress the local disparity planes  according to the relative level of visual discomfort.  So here the C is the information of the discomfort level we obtained in the previous slide.  So we use this information to determine the amount of disparity compression at each disparity  plane.  So here this figure shows the example of disparity function that compress just one disparity  plane.  So here the x-axis represents the original disparity and y-axis represents the remap  disparity.  So here for disparity plane DI, the amount of disparity compression is determined by  the CDI.  So in this way we compress the each disparity plane according to the severe visual discomfort.  And here the W, this parameter is a weight of factor that determines the absolute value  that disparity to be compressed.  So since each disparity plane has different amount of disparity to be compressed and compressing  specific disparity planes affect the others, so it isn't necessary to determine the proper  disparity value.  Since we have the multiple choice for the remap disparity plane, in this case we took  the minimum disparity value.  So this is the example for just only close disparity.  So the same the mechanism applied to the uncrossed disparity.  So the reason why we took the minimum value for this interval is that we need to sufficiently  compress the disparity of problematic disparity plane.  And this is the final example of the disparity, local disparity mapping.  So the left figure shows the relative level of visual discomfort and the right figure  shows the local disparity mapping function we generated using the method previously explained.  So we can see that disparity between 3 degree to 3.5 degree has been compressed.  So to demonstrate the performance of the proposed local disparity mapping function, we used  18-sterescopic video which consists of 9 scenes with 2 different camera base lines.  And this thumbnail shows an example of the star-screw video we used for our experiment.  And these star-screw videos were generally to have very excessive disparity range and  a vast change of disparity.  So for our experiment, we applied two different disparity mapping methods.  The first one was the global disparity mapping which linearly scales the disparity range  of star-screw video.  And the second one was combined global and local disparity mapping.  So by comparing the subject's curve between these two sets, we proved that the additional  local disparity mapping can improve visual comfort while preserving the naturalness of  the scene.  So to compress the disparity of the scene, it is necessary to determine the proper amount  of disparity compression.  So to this purpose, we used object visual comfort metric which is our previous work.  So for global disparity mapping, we used a visual input space we saved for disparity  magnitudes which utilize the disparity statistical visual importance region to predict the overall  level of visual comfort of the input stereoscopic 3D contents.  Then for the second video set, we applied the visual input space we saved for disparity  magnitudes change.  So according to the prediction result, the absolute amount of disparity compression  is determined.  Then we applied the proposed local disparity mapping method.  This is a very simple example of the star-screw video after two different disparity mapping.  The first one is global disparity mapping and the second one is combined global and  local disparity mapping.  So here you can see that after local disparity, here I record that the foreground butterfly  is the visual input region in this video.  So the local disparity mapping compress the most problematic region here, the butterfly.  So you can see that disparity of this butterfly is decreased and while the disparity of background  like tree is almost preserved.  To demonstrate the local disparity mapping, we conducted our subjective assessment of  visual comfort and naturalness.  We used DSC-Case which randomly displayed two different versions of the same image.  The one was the original image and the other was the process image by the global remapping  or combined global and local disparity mapping.  So we measured the visual comfort and naturalness using this score sheet.  And for the analysis of the result, we converted this continuous scale from 0 to 10 and demo  so as used for the analysis.  This is the result of subject assessment result of visual comfort.  And here the excess is represented the content index and why excess is represented the most.  So in here the empty, the B ins represent the improvement of visual comfort obtained by  global disparity mapping and the field bar represent the one obtained by the combined  global and local approach.  So we observed that both global and combined approach improved the visual comfort with  statistical significance.  And for global approach, average demo was 8.5 and for the combined approach, average demo  was 11.  And the difference of these two scores was statistically significant.  And for naturalness, you observed that the overall improvement of naturalness was significant  but we could not observe any difference between global and combined approach.  So in this paper, we proposed the local disparity mapping to compress the local problematic  display plan in the statistical video and to this purpose, we detect visual implications  and quantify the level of discomfort induced by those reasons.  And our subject assessment shows that the proper local disparity mapping can improve  visual comfort while preserving the naturalness of scene.  Okay.  Okay, that's all.  Thank you.  Thank you.  Maybe one quick question if you have any questions?  So do I understand that by local, you mean local in depth disparity?  Right.  So you change some range of depth to some other range of depth?  Yeah, right.  So here local does not mean special information but depth information.  So the local is in Z.  Yeah, right.  So please start to speak.  Thank you.
Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_0_83.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_84_155.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_156_1187.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_1188_2603.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_2604_2855.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_2856_3119.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_3120_3419.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_3420_3503.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_3504_4007.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_4008_4115.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_4116_6119.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_6120_6587.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_6588_7439.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_7440_8519.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_8520_9767.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_9768_10883.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_10884_12539.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_12540_13175.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_13176_14627.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_14628_16859.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_16860_18191.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_18192_19067.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_19068_19931.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_19932_20459.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_20460_20795.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_20796_22547.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_22548_23867.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_23868_25127.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_25128_26615.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_26616_26987.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_26988_29027.png Disparity_remapping_to_ameliorate_visual_comfort_of_stereoscopic_video__8648-33___SD_A2013__CibZ_RoO_29028_29111.png