Trends in S3D movies quality as evaluated on 105 movies and 10 quality metrics
CC Attribution - NoDerivatives 2.0 UK: England & Wales:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Vatolin, Dimitry
Bokov, Alexander
Erifeev, Mikhail
Napadovsky, Vyacheslav
1) OBJECTIVE: The main objective of the large-scale quality analysis of S3D movies is to gain a better understanding of how quality control was performed in different movies. Also several novel quality metrics are presented, including channel swap detection, evaluation of temporal shifts between stereoscopic views and depth continuity. 2) METHOD: The main technical obstacle that we had to overcome was an enormous amount of computation and disc space required by such an analysis. Evaluation of one movie could take up to 4 weeks and required over 40GB for the source Blu-ray only. To maximize the efficiency we had to rewrite all of our metrics to exploit the multicore architecture of contemporary CPUs. We have also developed a system that efficiently distributes the computations across the cluster of up to 17 computers working in parallel. It enabled us to finish the evaluation of 105 movies in about 6 months. 3) RESULTS: An evaluation of 105 S3D movies’ technical quality has been conducted that span over 50 years of the stereoscopic cinema history. Our main observations are as follows: According to our measurements, “Avatar” in fact had a superior technical quality compared to the most S3D movies of the previous decade. So it is not surprising that it was positively received by the viewers. S3D quality improvement over the years is fairly obvious from the conducted evaluation, e.g. the results of average-quality movies from 2010 correspond to the results of the 2014 movies with nearly the worst technical quality. A more important conclusion from the analysis, however, is that it gradually becomes possible to produce low-budget movies with excellent technical quality, that was previously within reach only for high-budget blockbusters. We hope that new objective quality metrics like the channel mismatch metric will find their applications in production pipelines. It can further decrease the number of viewers experiencing discomfort and give a start to the new surge of S3D popularity. 4) CONCLUSION: Objective S3D quality metrics make it easier to find problematic frames or entire shots in movies, that could potentially lead to discomfort of a significant fraction of the audience. Our analysis have already revealed thousands of such scenes in real S3D movies. But to directly estimate this discomfort subjective evaluations are necessary. We have organized several of such evaluations with the help of volunteers, that were asked to watch some of the scenes with the worst technical quality according to our analysis. These experiments allow us to further improve the metrics and to develop a universal metric that could directly predict a percentage of the audience experiencing a noticeable discomfort. It is already clear that the development of such universal metric is a very challenging problem, so we are looking for collaboration. It is also clear to us that the majority of problems could be fixed in post-production with minimal user intervention, if not entirely automatically. Some of these techniques are not widely employed just because the problem itself is not considered important enough to require correction. We hope our work could help shed the light on the problem and more attention will be drawn to correcting the S3D production issues. 
 Okay. Hello, everybody. I will describe our results and we'll announce a lot of our other results because we have very limited time in this pretty funny project. Okay. You know that currently it's a big problem in industry because as Forbes wrote, it's great 3D fiasco and 3D guys kill their own cash cow. It's bad news generally, but it's good news that looks like currently it's possible to solve more and more problems that cause low quality 3D. And when we talk with guys from science and guys from industry, looks like these guys live in different universe. So this is one world of guys from Australia, and so on, and another world parallel, absolutely the parallel world of guys, of scientists. And it's a very big question how to make bridge through all these walls and we try to do this. Okay. What we do? This is our project team and this is Blu-ray disks that was analyzed. We analyze all Blu-rays, all 3D Blu-rays with a budget noted on IMDB.com. This means that this is all non-documentary Blu-rays. And we measure trends and we work in this area, in area of measurement of quality of 3D, the light last eight years. So this is a pretty long project and we measure 3D quality previously, but there was very common question, you measure some will you? Is it good or is it wrong? What can you comment this? And currently we can answer to this question and we can consider different values. Also we propose new metrics, we create a challenge match metric and publish results something like four years ago. But currently we create practical challenge match. This means that it's become approximately 100 times faster and more robust simultaneously. So it looks like we are the first guys in the world who create a practical challenge match metric that's so fast that it's possible to measure 100 movies. And also we create new temporal sheets metric and update other metrics and so on and so on. Oh, okay, prolaxis. Everybody know here what's this. And we analyze movies and you see very, very, very big parallax on big screen, your eyes will be damaged. And we measure trends and you see here trends on 100 movies during last years. Here is zero level. It's negative parallax, it's positive parallax. So it's closer to viewer. It's behind screen. And this is level of avatar. And this is values of front object and first object in scene, average value for movie. So you see how good depth, how good visible depth have different movies. And this is trend lines, trend lines and bad news that you see that depths become smaller and smaller and smaller. Here was avatar and you see that after avatar we can see movies that sometimes have four or three times less visual depth than avatar. This means that this movies will have good perceptive depth only in cinema. And when you will buy this Blu-ray, like we are, and try to see this Blu-ray on small screen, visible depth will be very small. Be careful. So it's better to go to the cinema. Okay, and there are a lot of other interesting results after this analysis, but anyway, let's move forward. This is, by the way, this is all movies sorted by visible depth. And you can see red color mean converted movie, blue color mean captured movie. And you can see, of course, most of converted movies have small depth, every professional understand why. And this is avatar. And this is Titanic. Titanic have one of the best visible depth in comparison with all other converted movies. Really, really great job. Really great job. And that's why people go to the cinema, see avatar, see Titanic, see clearly visible depth, clearly visible, and enjoy. And you understand that people who will go to this movie, for example, you understand. So I just unknown this result, we also analyze general histogram of depth in movies and red color again, converted movies. And you can see that many converted movies have not so... Have some strange steps in depth, so it's interesting to analyze how this not so smooth depth will decrease your visual performance of 3D movie. Okay. So, as I told you, it looks like we have first guys who create practical channel mismatch measure. And this measure can find not only fully swappered views, but also some special effects that put it on wrong depths. For example, here you can see this guy's on first plan, and this candle on the background and this special effect put it at the depth of candle, so at the background. But obviously in front of this guy. So at this moment, you will see absolutely impossible picture. And your brain will be damaged a little bit. And of course, we find several things with absolutely swappered channels. Here is also absolutely swappered channels. And this is histogram of movies with swappered channels. Good news from this analyzed 100 Blu-rays, 23 have at least one scene with swappered channels. So 21% probability that you will see at least one swappered channel in movie. So enough big. And good news that the biggest amount, the biggest percentage of such movies was in 2010. And I suppose that you understand why a lot of guys appear in this industry who never previously worked with stereo. And percentage of such movies here was even bigger for previous movies. And currently, this parameter become lower and lower and lower. So I suppose that with our metric, this will be zero in future. It's possible. And why? It's a good reason to do this. And we also analyze visual discomfort for the scene. And currently, we are able to predict in short words. We can predict visual discomfort from scene with swappered channels. Because if this duck scene, if this shot scene, if this flat scene, visual discomfort from swappered channels will be lower. And it's possible to predict this. And this is trend line for my presentation is about trends, you remember. This is trend line for channel is much by release date. And we see that unfortunately some movies may be not so good, but generally, situation become better. And the same situation with the budget. Of course, commonly, so opportunity have movies with smaller budget. That's the understand why. Okay. Color mismatch. Everybody understand what this is? And here are some samples. For example, like this. So 3D or journey. Also clearly visible. Color mismatch. And on trend line, we can see that generally, situation become better, fortunately. And it's interesting to analyze resident level first and second. We see that situation become better. And step up situation also become better. So situation become better. But of course, maybe we have good question to not so good movies now. And generally we can see that this is avatar. This is avatar. And we see that avatar was one of the best movies when it was released. But currently level in color mismatch for avatar become in red zone for now. So I'm absolutely sure that avatar to 0, to 0. It will be essentially better than first one. And the same analysis for budgets. Of course, low budget movies have lower values and high budget movie are accurate. And you can see that avatar is the worth on this parameter in comparison with all high budget movies. So this means that new high budget movies are more accurate than avatar. Good news. And by the way, here we can see that some low budget movies have the same parameters as high budget movies. So currently it's possible to create even low budget movies on the same level of quality. So again, good news. Special symmetry. Everybody understand what this is when we see something like this. And generally this is not so painful but not so good. And again, timeline. And we can see that step up becomes naturally better from one to 10. And resident evil have the same values. So it looks like these guys began to use new technologies and these guys have not. And if you will analyze by budget also, of course, the same situation a lot of. And by budget you can see that step up, move it from red zone to green zone for this budget. So it's interesting analysis. Rotation. Currently we began to analyze separately. Rotation, zoom and other things because even small, generally a small shift, vertical shift is not so big problem. But when it's rotation, even for one degree, it's really painful. And you can see, for example, rotation here, shark knight. And rotation here, silent hill, crazy rotation. By the way, here you see that we show the worst sense and you maybe see that it's very common situation when this is horror movies. So we prove that this movies will be not only scary but also painful. Okay and again release dates. And again, step up. You see that step up was here in red zone and become essentially better than resident evil. Unfortunately not. And here is situation for budget and you see that again avatar have one of the worst values currently. Scale mismatch. Here is, we was really surprised how many scale mismatch sense was in computer graphics. And Piranha, space station again computer graphics. Guys. Start track. Again computer graphics. Crazy. And fortunately situation become better and the same situation on budget. Sharpness mismatch. When you see sharp, it's really painful and the same situation become better. Temporal shift. Pretty funny things. It's now a new metric. You see clearly visible. Temporal shift. Hugo, also clearly visible temporal shift. Step up. Clearly visible temporal shift. It's we found more than 500 scenes with temporal shift. So pretty big amount. By the way, it's funny. Very fast. I motion. And again situation. We can say that situation become better in temporal domain and in budget chart. And we create our reports. Currently it was published. And we have something close to 3,000 page in this reports. And report thousands of problems. A lot of downloaded, 28 contributed stereographers, their names. And next report will be published soon. Maybe in first quarter even. And we plan to reports during this year. And we have a lot of plans. And our goal is to predict fatigue by movie. So to predict percentage of disappointed viewers. It's very ambitious and challenging goal. But we move in this direction. And this you see that it's composite with 3D bluradisks. We see that no pain will be good gain for this industry. Thank you very much.
 Okay.  Hello, everybody.  I will describe our results and we'll announce a lot of our other results because we have  very limited time in this pretty funny project.  Okay.  You know that currently it's a big problem in industry because as Forbes wrote, it's  great 3D fiasco and 3D guys kill their own cash cow.  It's bad news generally, but it's good news that looks like currently it's possible to  solve more and more problems that cause low quality 3D.  And when we talk with guys from science and guys from industry, looks like these guys live  in different universe.  So this is one world of guys from Australia, and so on, and another world parallel, absolutely  the parallel world of guys, of scientists.  And it's a very big question how to make bridge through all these walls and we try to do this.  Okay.  What we do?  This is our project team and this is Blu-ray disks that was analyzed.  We analyze all Blu-rays, all 3D Blu-rays with a budget noted on IMDB.com.  This means that this is all non-documentary Blu-rays.  And we measure trends and we work in this area, in area of measurement of quality of  3D, the light last eight years.  So this is a pretty long project and we measure 3D quality previously, but there was very  common question, you measure some will you?  Is it good or is it wrong?  What can you comment this?  And currently we can answer to this question and we can consider different values.  Also we propose new metrics, we create a challenge match metric and publish results  something like four years ago.  But currently we create practical challenge match.  This means that it's become approximately 100 times faster and more robust simultaneously.  So it looks like we are the first guys in the world who create a practical challenge match  metric that's so fast that it's possible to measure 100 movies.  And also we create new temporal sheets metric and update other metrics and so on and so  on.  Oh, okay, prolaxis.  Everybody know here what's this.  And we analyze movies and you see very, very, very big parallax on big screen, your eyes  will be damaged.  And we measure trends and you see here trends on 100 movies during last years.  Here is zero level.  It's negative parallax, it's positive parallax.  So it's closer to viewer.  It's behind screen.  And this is level of avatar.  And this is values of front object and first object in scene, average value for movie.  So you see how good depth, how good visible depth have different movies.  And this is trend lines, trend lines and bad news that you see that depths become smaller  and smaller and smaller.  Here was avatar and you see that after avatar we can see movies that sometimes have four  or three times less visual depth than avatar.  This means that this movies will have good perceptive depth only in cinema.  And when you will buy this Blu-ray, like we are, and try to see this Blu-ray on small  screen, visible depth will be very small.  Be careful.  So it's better to go to the cinema.  Okay, and there are a lot of other interesting results after this analysis, but anyway, let's  move forward.  This is, by the way, this is all movies sorted by visible depth.  And you can see red color mean converted movie, blue color mean captured movie.  And you can see, of course, most of converted movies have small depth, every professional  understand why.  And this is avatar.  And this is Titanic.  Titanic have one of the best visible depth in comparison with all other converted movies.  Really, really great job.  Really great job.  And that's why people go to the cinema, see avatar, see Titanic, see clearly visible depth,  clearly visible, and enjoy.  And you understand that people who will go to this movie, for example, you understand.  So I just unknown this result, we also analyze general histogram of depth in movies and red  color again, converted movies.  And you can see that many converted movies have not so...  Have some strange steps in depth, so it's interesting to analyze how this not so smooth  depth will decrease your visual performance of 3D movie.  Okay.  So, as I told you, it looks like we have first guys who create practical channel mismatch  measure.  And this measure can find not only fully swappered views, but also some special effects that put  it on wrong depths.  For example, here you can see this guy's on first plan, and this candle on the background  and this special effect put it at the depth of candle, so at the background.  But obviously in front of this guy.  So at this moment, you will see absolutely impossible picture.  And your brain will be damaged a little bit.  And of course, we find several things with absolutely swappered channels.  Here is also absolutely swappered channels.  And this is histogram of movies with swappered channels.  Good news from this analyzed 100 Blu-rays, 23 have at least one scene with swappered  channels.  So 21% probability that you will see at least one swappered channel in movie.  So enough big.  And good news that the biggest amount, the biggest percentage of such movies was in 2010.  And I suppose that you understand why a lot of guys appear in this industry who never  previously worked with stereo.  And percentage of such movies here was even bigger for previous movies.  And currently, this parameter become lower and lower and lower.  So I suppose that with our metric, this will be zero in future.  It's possible.  And why?  It's a good reason to do this.  And we also analyze visual discomfort for the scene.  And currently, we are able to predict in short words.  We can predict visual discomfort from scene with swappered channels.  Because if this duck scene, if this shot scene, if this flat scene, visual discomfort from  swappered channels will be lower.  And it's possible to predict this.  And this is trend line for my presentation is about trends, you remember.  This is trend line for channel is much by release date.  And we see that unfortunately some movies may be not so good, but generally, situation  become better.  And the same situation with the budget.  Of course, commonly, so opportunity have movies with smaller budget.  That's the understand why.  Okay.  Color mismatch.  Everybody understand what this is?  And here are some samples.  For example, like this.  So 3D or journey.  Also clearly visible.  Color mismatch.  And on trend line, we can see that generally, situation become better, fortunately.  And it's interesting to analyze resident level first and second.  We see that situation become better.  And step up situation also become better.  So situation become better.  But of course, maybe we have good question to not so good movies now.  And generally we can see that this is avatar.  This is avatar.  And we see that avatar was one of the best movies when it was released.  But currently level in color mismatch for avatar become in red zone for now.  So I'm absolutely sure that avatar to 0, to 0.  It will be essentially better than first one.  And the same analysis for budgets.  Of course, low budget movies have lower values and high budget movie are accurate.  And you can see that avatar is the worth on this parameter in comparison with all high  budget movies.  So this means that new high budget movies are more accurate than avatar.  Good news.  And by the way, here we can see that some low budget movies have the same parameters  as high budget movies.  So currently it's possible to create even low budget movies on the same level of quality.  So again, good news.  Special symmetry.  Everybody understand what this is when we see something like this.  And generally this is not so painful but not so good.  And again, timeline.  And we can see that step up becomes naturally better from one to 10.  And resident evil have the same values.  So it looks like these guys began to use new technologies and these guys have not.  And if you will analyze by budget also, of course, the same situation a lot of.  And by budget you can see that step up, move it from red zone to green zone for this budget.  So it's interesting analysis.  Rotation.  Currently we began to analyze separately.  Rotation, zoom and other things because even small, generally a small shift, vertical shift  is not so big problem.  But when it's rotation, even for one degree, it's really painful.  And you can see, for example, rotation here, shark knight.  And rotation here, silent hill, crazy rotation.  By the way, here you see that we show the worst sense and you maybe see that it's very common  situation when this is horror movies.  So we prove that this movies will be not only scary but also painful.  Okay and again release dates.  And again, step up.  You see that step up was here in red zone and become essentially better than resident  evil.  Unfortunately not.  And here is situation for budget and you see that again avatar have one of the worst values  currently.  Scale mismatch.  Here is, we was really surprised how many scale mismatch sense was in computer graphics.  And Piranha, space station again computer graphics.  Guys.  Start track.  Again computer graphics.  Crazy.  And fortunately situation become better and the same situation on budget.  Sharpness mismatch.  When you see sharp, it's really painful and the same situation become better.  Temporal shift.  Pretty funny things.  It's now a new metric.  You see clearly visible.  Temporal shift.  Hugo, also clearly visible temporal shift.  Step up.  Clearly visible temporal shift.  It's we found more than 500 scenes with temporal shift.  So pretty big amount.  By the way, it's funny.  Very fast.  I motion.  And again situation.  We can say that situation become better in temporal domain and in budget chart.  And we create our reports.  Currently it was published.  And we have something close to 3,000 page in this reports.  And report thousands of problems.  A lot of downloaded, 28 contributed stereographers, their names.  And next report will be published soon.  Maybe in first quarter even.  And we plan to reports during this year.  And we have a lot of plans.  And our goal is to predict fatigue by movie.  So to predict percentage of disappointed viewers.  It's very ambitious and challenging goal.  But we move in this direction.  And this you see that it's composite with 3D bluradisks.  We see that no pain will be good gain for this industry.  Thank you very much.
SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_0_3107.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_3108_5579.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_5580_8255.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_8256_11003.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_11004_11267.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_11268_12035.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_12036_12419.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_12420_16127.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_16128_16427.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_16428_19307.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_19308_19511.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_19512_22031.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_22032_22763.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_22764_23267.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_23268_23411.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_23412_23819.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_23820_23867.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_23868_24095.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_24096_24683.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_24684_24839.png SD_A_2016__Trends_in_S3D_movies_quality_as_evaluated_on_105_movies_and_10_quality_metrics_UVe9DVuxqG_24840_27355.png