<story><title>Investigating the 900GB “Collection #1-5” password leaks with R</title><url>https://timogrossenbacher.ch/2019/03/big-data-journalism-with-spark-and-r/</url></story><parent_chain></parent_chain><comment><author>aw3c2</author><text>&amp;gt; We’re talking about 31 million lines of data here (all lines with Swiss email addresses in the leak). On our workstation, reducing this to the distinct email addresses (approx. 3,3 million) took a mere 10.099 seconds (4 + 6 + .099).&lt;p&gt;That&amp;#x27;s just 700MB of data... On my 5 year old middle-class PC this took 12 seconds with sort and uniq.&lt;p&gt;&lt;pre&gt;&lt;code&gt;    $ wc -l email30m
    31000000 email30m
    
    $ time sort email30m | uniq &amp;gt; email30m.uniq
    
    real 0m11.498s
    user 0m33.833s
    sys 0m0.933s
    
    $ wc -l email30m.uniq 
    2172320 email30m.uniq
&lt;/code&gt;&lt;/pre&gt;
This automatically split the task into all available CPU cores.</text></comment>
<story><title>Investigating the 900GB “Collection #1-5” password leaks with R</title><url>https://timogrossenbacher.ch/2019/03/big-data-journalism-with-spark-and-r/</url></story><parent_chain></parent_chain><comment><author>girst</author><text>You can actually get a lot just from grep (well, zgrep, which searches compressed files)&lt;p&gt;&lt;pre&gt;&lt;code&gt;    zgrep -A &amp;#x27;@\[^.]\+\.ch&amp;#x27; &amp;#x2F;path&amp;#x2F;to&amp;#x2F;collections&amp;#x2F;**&amp;#x2F;*.gz
&lt;/code&gt;&lt;/pre&gt;
will spit out (almost) all swiss email addresses. sprinkle some grep -v, wc and sort|uniq here and there and bob&amp;#x27;s your uncle :-)&lt;p&gt;(I&amp;#x27;m in no way trying to dismiss the efforts--my method won&amp;#x27;t get you as much detail, but is enough to find you and your friends who asked to be searched in the dumps)</text></comment>