<story><title>IBM Invests to Help Apache Spark</title><url>http://bits.blogs.nytimes.com/2015/06/15/ibm-invests-to-help-open-source-big-data-software-and-itself/</url><text></text></story><parent_chain></parent_chain><comment><author>rpalmaotero</author><text>For everyone that wants to start working with Spark and Big Data, I recommend them to enrole into this MOOC published by UC Berkeley at EDX: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.edx.org&amp;#x2F;course&amp;#x2F;introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.edx.org&amp;#x2F;course&amp;#x2F;introduction-big-data-apache-spar...&lt;/a&gt;</text></comment>
<story><title>IBM Invests to Help Apache Spark</title><url>http://bits.blogs.nytimes.com/2015/06/15/ibm-invests-to-help-open-source-big-data-software-and-itself/</url><text></text></story><parent_chain></parent_chain><comment><author>zaroth</author><text>I read about an interesting technique, an &amp;quot;all-or-nothing tracker&amp;quot; in a blog post from an Apache Spark engineer.&lt;p&gt;You dispatch n jobs, where n is quite large, and you want to know; have all n jobs have completed, or has less than n jobs have completed. How to do so with a small fixed number of bytes with very high probability?&lt;p&gt;Give each job a random 128-bit ID number. XOR each ID number together as you start each job, and XOR into the same value as each job completes. If all the jobs have completed, the result is 0. The chance of zero turning up randomly if all jobs are not complete is negligible.&lt;p&gt;The technique is mentioned here under &amp;#x27;Lineage Tracking&amp;#x27;: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;highlyscalable.wordpress.com&amp;#x2F;2013&amp;#x2F;08&amp;#x2F;20&amp;#x2F;in-stream-big-data-processing&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;highlyscalable.wordpress.com&amp;#x2F;2013&amp;#x2F;08&amp;#x2F;20&amp;#x2F;in-stream-bi...&lt;/a&gt;  but there&amp;#x27;s a better blog post I remembered reading but can&amp;#x27;t find at the moment...</text></comment>