<story><title>MH370 Was ‘Manipulated’ Off Course to Its End, Report Says</title><url>https://www.bloomberg.com/news/articles/2018-07-30/mh370-investigation-unable-to-determine-cause-of-disappearance</url></story><parent_chain><item><author>froindt</author><text>I think there are a couple orders of magnitude difference between your idea and what it&amp;#x27;d take to process.&lt;p&gt;My googling shows 148.94 trillion square meters, with 18.6 trillion&lt;p&gt;We&amp;#x27;ll being generous to say &amp;quot;good enough to spot a car&amp;quot; is 2 pixels with 1 meter pixel size.&lt;p&gt;To be moderately confident it&amp;#x27;s a car, I&amp;#x27;d guesstimate we&amp;#x27;d need quite a bit higher resolution.&lt;p&gt;With no compression and RGB 24 bit encoding, that&amp;#x27;s 3 bytes&amp;#x2F;pixel, or 446 TB of data per &amp;quot;frame&amp;quot; of our realtime recording.  Of course that could be compressed down a TON, but regardless it&amp;#x27;s a staggeringly large quantity of data to work with.&lt;p&gt;&amp;gt; some machine learning it should be possible to skip most of oceans forests and deserts (unless some unusual object appears in that area).&lt;p&gt;As far as I can think through, you&amp;#x27;d need to process the data in the oceans, forests, and deserts if you want to find unusual objects.  We can&amp;#x27;t do a shortcut and say only 1&amp;#x2F;8 of the world is inhabited&amp;#x2F;developed in some way and that&amp;#x27;s all we&amp;#x27;re going to monitor.  Perhaps you take 1&amp;#x2F;4 or 1&amp;#x2F;8 of the data from the vast amounts of uninteresting data and look for larger anomalies.&lt;p&gt;No matter how you look at it, there&amp;#x27;d be a ridiculous amount of data and processing required to do object detection on that scale.</text></item><item><author>comboy</author><text>I wonder when are we going to have some low resolution (good enough to spot a car) realtime recording of the full globe. I know it&amp;#x27;s tons of data but with some machine learning it should be possible to skip most of oceans forests and deserts (unless some unusual object appears in that area).</text></item></parent_chain><comment><author>staticfloat</author><text>This is actually a pretty good argument for sparse data handling; push some intelligence out to the edges of your network (e.g. the spy satellites or what have you) that are collecting the data in the first place, and then only store &amp;quot;interesting&amp;quot; pixels.  E.g. regions where there are movement; regions already determined to be of interest (densely populated areas, war zones, trading vessel corridors, etc....)&lt;p&gt;There is research going on right now to build native camera systems that emit data (at the hardware level) that is not dense but rather sparse; information is transmitted as &amp;quot;blocks of pixels that changed between frame two and three&amp;quot; rather than &amp;quot;here is the state of all pixels at frame two, and here is the state of all pixels at frame three&amp;quot;.  In essence, doing a small amount of delta compression within the hardware itself.&lt;p&gt;This seems like a really natural fit for an application such as this one; and could potentially cut the amount of data needed to be paid attention to down by orders of magnitude.&lt;p&gt;Many of these ideas are similar to what is used to compress video down in the first place; it would be interesting to see a combination of video compression technology and &amp;quot;region of interest&amp;quot; ML pruning, where the ML models are applied only to interesting regions, as identified by the compression codec.  The codec already has done the analysis to figure out which portions of the video have changed, which portions of the video are simply shifted versions of information from the last frame, etc....  You could significantly speed up ML analysis by making use of that kind of information.</text></comment>
<story><title>MH370 Was ‘Manipulated’ Off Course to Its End, Report Says</title><url>https://www.bloomberg.com/news/articles/2018-07-30/mh370-investigation-unable-to-determine-cause-of-disappearance</url></story><parent_chain><item><author>froindt</author><text>I think there are a couple orders of magnitude difference between your idea and what it&amp;#x27;d take to process.&lt;p&gt;My googling shows 148.94 trillion square meters, with 18.6 trillion&lt;p&gt;We&amp;#x27;ll being generous to say &amp;quot;good enough to spot a car&amp;quot; is 2 pixels with 1 meter pixel size.&lt;p&gt;To be moderately confident it&amp;#x27;s a car, I&amp;#x27;d guesstimate we&amp;#x27;d need quite a bit higher resolution.&lt;p&gt;With no compression and RGB 24 bit encoding, that&amp;#x27;s 3 bytes&amp;#x2F;pixel, or 446 TB of data per &amp;quot;frame&amp;quot; of our realtime recording.  Of course that could be compressed down a TON, but regardless it&amp;#x27;s a staggeringly large quantity of data to work with.&lt;p&gt;&amp;gt; some machine learning it should be possible to skip most of oceans forests and deserts (unless some unusual object appears in that area).&lt;p&gt;As far as I can think through, you&amp;#x27;d need to process the data in the oceans, forests, and deserts if you want to find unusual objects.  We can&amp;#x27;t do a shortcut and say only 1&amp;#x2F;8 of the world is inhabited&amp;#x2F;developed in some way and that&amp;#x27;s all we&amp;#x27;re going to monitor.  Perhaps you take 1&amp;#x2F;4 or 1&amp;#x2F;8 of the data from the vast amounts of uninteresting data and look for larger anomalies.&lt;p&gt;No matter how you look at it, there&amp;#x27;d be a ridiculous amount of data and processing required to do object detection on that scale.</text></item><item><author>comboy</author><text>I wonder when are we going to have some low resolution (good enough to spot a car) realtime recording of the full globe. I know it&amp;#x27;s tons of data but with some machine learning it should be possible to skip most of oceans forests and deserts (unless some unusual object appears in that area).</text></item></parent_chain><comment><author>DrJokepu</author><text>I don’t disagree with anything you said here, but it reminds me an argument about IRC privacy from the ’90s. People frequently accused IRC networks of secretly logging private conversations, and the usual argument was that this would require such an immense amount of storage that any such effort would be astronomically expensive and entirely infeasible.&lt;p&gt;I don’t know if that was really true even back then, but today it’s completely doable and done all the time (think Google Chat or Facebook Chat).&lt;p&gt;My point is, times change, it’s entirely possible that in 20 years storing that much data will be completely possible.</text></comment>