<story><title>Command-line tools can be faster than a Hadoop cluster (2014)</title><url>http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html</url></story><parent_chain><item><author>cs702</author><text>Most &amp;quot;big data&amp;quot; problems are really &amp;quot;small data&amp;quot; by the standards of modern hardware:&lt;p&gt;* Desktop PCs with up to 6TB of RAM and many dozens of cores have been available for over a year.[1]&lt;p&gt;* Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced.[2]&lt;p&gt;&lt;i&gt;CORRECTION: THE FIGURE IS 60TB, NOT 100TB.  See MagnumOpus&amp;#x27;s comment below.  In a haste, I searched Google and mistakengly linked to an April Fool&amp;#x27;s story.  Now I feel like a fool, of course.&lt;/i&gt;  Still, the point is valid.&lt;p&gt;* Four Nvidia Titan X GPUs can give you up to 44 Teraflops of 32-bit FP computing power in a single desktop.[3]&lt;p&gt;Despite this, the number of people who have unnecessarily spent money and&amp;#x2F;or complicated their lives with tools like Hadoop is pretty large, particularly in &amp;quot;enterprise&amp;quot; environments.  A lot of &amp;quot;big data&amp;quot; problems can be handled by a single souped-up machine that fits under your desk.&lt;p&gt;[1] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.alphr.com&amp;#x2F;news&amp;#x2F;enterprise&amp;#x2F;387196&amp;#x2F;intel-xeon-e7-v2-servers-support-6tb-of-ram&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.alphr.com&amp;#x2F;news&amp;#x2F;enterprise&amp;#x2F;387196&amp;#x2F;intel-xeon-e7-v2...&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.storagenewsletter.com&amp;#x2F;rubriques&amp;#x2F;hard-disk-drives&amp;#x2F;incredible-record-of-100tb-into-3-5-inch-hdd-with-hamrhelium&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.storagenewsletter.com&amp;#x2F;rubriques&amp;#x2F;hard-disk-drives&amp;#x2F;...&lt;/a&gt;&lt;p&gt;[3] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=12141334&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=12141334&lt;/a&gt;</text></item></parent_chain><comment><author>rdtsc</author><text>Well basically as soon as big data hit the news everyone stopped doing data and started doing big data.&lt;p&gt;Same thing with data science. Opened a Jupyter notebook, loaded 100 rows, and displayed a graph - &amp;quot;I am a data scientist&amp;quot;.</text></comment>
<story><title>Command-line tools can be faster than a Hadoop cluster (2014)</title><url>http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html</url></story><parent_chain><item><author>cs702</author><text>Most &amp;quot;big data&amp;quot; problems are really &amp;quot;small data&amp;quot; by the standards of modern hardware:&lt;p&gt;* Desktop PCs with up to 6TB of RAM and many dozens of cores have been available for over a year.[1]&lt;p&gt;* Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced.[2]&lt;p&gt;&lt;i&gt;CORRECTION: THE FIGURE IS 60TB, NOT 100TB.  See MagnumOpus&amp;#x27;s comment below.  In a haste, I searched Google and mistakengly linked to an April Fool&amp;#x27;s story.  Now I feel like a fool, of course.&lt;/i&gt;  Still, the point is valid.&lt;p&gt;* Four Nvidia Titan X GPUs can give you up to 44 Teraflops of 32-bit FP computing power in a single desktop.[3]&lt;p&gt;Despite this, the number of people who have unnecessarily spent money and&amp;#x2F;or complicated their lives with tools like Hadoop is pretty large, particularly in &amp;quot;enterprise&amp;quot; environments.  A lot of &amp;quot;big data&amp;quot; problems can be handled by a single souped-up machine that fits under your desk.&lt;p&gt;[1] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.alphr.com&amp;#x2F;news&amp;#x2F;enterprise&amp;#x2F;387196&amp;#x2F;intel-xeon-e7-v2-servers-support-6tb-of-ram&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.alphr.com&amp;#x2F;news&amp;#x2F;enterprise&amp;#x2F;387196&amp;#x2F;intel-xeon-e7-v2...&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.storagenewsletter.com&amp;#x2F;rubriques&amp;#x2F;hard-disk-drives&amp;#x2F;incredible-record-of-100tb-into-3-5-inch-hdd-with-hamrhelium&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.storagenewsletter.com&amp;#x2F;rubriques&amp;#x2F;hard-disk-drives&amp;#x2F;...&lt;/a&gt;&lt;p&gt;[3] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=12141334&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=12141334&lt;/a&gt;</text></item></parent_chain><comment><author>MagnumOpus</author><text>&amp;gt; Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced&lt;p&gt;That is an April Fools story.&lt;p&gt;(Of course you can still get a Synology DS2411+&amp;#x2F;DX1211 24-bay NAS combo for a few thousand bucks, but it will take up a lot of space under your desk and keep your legs toasty...)</text></comment>