<story><title>Engineers Shouldn’t Write ETL</title><url>http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl</url></story><parent_chain><item><author>darksaints</author><text>This is so true. I do business intelligence at Amazon, and I&amp;#x27;ve seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I&amp;#x27;ve really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, service logs), ETL can get really absurdly complicated. But that is still no excuse to introduce big data if your data isn&amp;#x27;t actually big.&lt;p&gt;I really hate pat-myself-on-the-back stories, but I&amp;#x27;m really proud of this moment, so I&amp;#x27;m gonna share. One time a principal engineer came to me with a data analysis request and told me that the data would be available to me soon, only to come to me an hour later with the bad news that the data was 2 terabytes and I&amp;#x27;d probably have to spin up an EMR cluster. I borrowed a spinning disk USB drive, loaded all the data into a SQLite database, and had his analysis done before he could even set up a cluster with Spark. The proud moment comes when he tells his boss that we already had the analysis done despite his warning that it might take a few days because &amp;quot;big data&amp;quot;. It was then that I got to tell him about this phenomenal new technology called SQLite and he set up a seminar where I got to teach big data engineers how to use it :)&lt;p&gt;P.S. If you do any of this sort of large dataset analysis in SQLite, upgrade to the latest version with every release, even if it means you have to `make; make install;` Seemingly every new release since about 3.8.0 has given me usable new features and noticeable query optimizations that are relevant for large query data analysis.</text></item><item><author>mikestew</author><text>&lt;i&gt;... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.&lt;/i&gt;&lt;p&gt;OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there&amp;#x27;s this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh, wait, I need Cassandra on my resume, so Mongo sux0r now...displayed with some of the worst bowl of spaghetti Android code I&amp;#x27;ve witnessed. The technical debt hole was dug so deep you could hide an Abrams tank in there. To this day I could not tell you the confused thinking that led them to believe this was necessary rather than just slapping it all into PostGres and calling it a day.&lt;p&gt;All because they were processing data sets sooooo huge, that they would fit on my laptop.&lt;p&gt;I quit reading about the time the article turned into a pitch for Stitch Fix, but leading up to that point it made a good case for what happens when companies think they have &amp;quot;big data&amp;quot; when they really don&amp;#x27;t. In summary, either a company hires skills they don&amp;#x27;t really need and the hires end up bored, or you hire mediocre people that make the convoluted mess I worked with.</text></item></parent_chain><comment><author>Buttons840</author><text>Me and a coworker were laughing at the parent comment, and I told him:&lt;p&gt;&amp;quot;I guarantee that somewhere, sometime, an engineer has been like &amp;#x27;hay guys, I loaded our big data into SQLite on my laptop and it ended up being faster than our fancy cluster&amp;#x27;&amp;quot;. We then joked that the engineer would be fired a few weeks later for not being a &amp;quot;cultural fit&amp;quot;.&lt;p&gt;A few minutes later you commented with your story. I hope you didn&amp;#x27;t get fired? :)</text></comment>
<story><title>Engineers Shouldn’t Write ETL</title><url>http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl</url></story><parent_chain><item><author>darksaints</author><text>This is so true. I do business intelligence at Amazon, and I&amp;#x27;ve seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I&amp;#x27;ve really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, service logs), ETL can get really absurdly complicated. But that is still no excuse to introduce big data if your data isn&amp;#x27;t actually big.&lt;p&gt;I really hate pat-myself-on-the-back stories, but I&amp;#x27;m really proud of this moment, so I&amp;#x27;m gonna share. One time a principal engineer came to me with a data analysis request and told me that the data would be available to me soon, only to come to me an hour later with the bad news that the data was 2 terabytes and I&amp;#x27;d probably have to spin up an EMR cluster. I borrowed a spinning disk USB drive, loaded all the data into a SQLite database, and had his analysis done before he could even set up a cluster with Spark. The proud moment comes when he tells his boss that we already had the analysis done despite his warning that it might take a few days because &amp;quot;big data&amp;quot;. It was then that I got to tell him about this phenomenal new technology called SQLite and he set up a seminar where I got to teach big data engineers how to use it :)&lt;p&gt;P.S. If you do any of this sort of large dataset analysis in SQLite, upgrade to the latest version with every release, even if it means you have to `make; make install;` Seemingly every new release since about 3.8.0 has given me usable new features and noticeable query optimizations that are relevant for large query data analysis.</text></item><item><author>mikestew</author><text>&lt;i&gt;... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.&lt;/i&gt;&lt;p&gt;OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there&amp;#x27;s this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh, wait, I need Cassandra on my resume, so Mongo sux0r now...displayed with some of the worst bowl of spaghetti Android code I&amp;#x27;ve witnessed. The technical debt hole was dug so deep you could hide an Abrams tank in there. To this day I could not tell you the confused thinking that led them to believe this was necessary rather than just slapping it all into PostGres and calling it a day.&lt;p&gt;All because they were processing data sets sooooo huge, that they would fit on my laptop.&lt;p&gt;I quit reading about the time the article turned into a pitch for Stitch Fix, but leading up to that point it made a good case for what happens when companies think they have &amp;quot;big data&amp;quot; when they really don&amp;#x27;t. In summary, either a company hires skills they don&amp;#x27;t really need and the hires end up bored, or you hire mediocre people that make the convoluted mess I worked with.</text></item></parent_chain><comment><author>chubot</author><text>That&amp;#x27;s awesome... I&amp;#x27;m working with some 15-20 GB sqlite databases.  Though 2 TB sounds kind of big?&lt;p&gt;Was it 2 TB before compression?  Because sqlite does blow up the data size over the raw data usually (depending on the original format obviously).  It can be kind of wasteful, and I ended up storing some fields as compressed JSON for this reason (that actually beats the sqlite format).&lt;p&gt;Also, the sqlite insert speed can be much slower than the disk&amp;#x27;s sequential write speed (even if you make sure you&amp;#x27;re not committing&amp;#x2F;flushing on every row, and if you have no indices to update, etc.)&lt;p&gt;So I think inserting and laying on the data could be nontrivial.  But the queries should be fast as long as they are indexed.  In theory, sqlite queries should be slower for a lot of use cases because it is row-oriented, but in practice distributed systems usually add 10x overhead themselves anyway...</text></comment>