<story><title>Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas?</title><text>Hi HN,&lt;p&gt;I&amp;#x27;m a seasoned Python software developer. Recently I have found a new obsession with data processing, management, engineering etc ... I&amp;#x27;d like to (eventually) branch off into that field but I find the lack of beginner friendly resources is slowing me down. All I can find is spark, hadoop related articles (I know these are prominent in the field, but I want to learn to walk before I run). So If any of you have pointers, websites, project ideas I can start to get a good grasp of all the fundamental concepts, I&amp;#x27;d really appreciate it.&lt;p&gt;Thanks a lot in advance</text></story><parent_chain><item><author>3pt14159</author><text>Spark, etc, are great, but honestly if you&amp;#x27;re just getting started I would forget all about existing tooling that is geared towards people working at 300 person companies and I would read The Data Warehouse ETL Toolkit by Kimball:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;gp&amp;#x2F;product&amp;#x2F;1118530802&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;gp&amp;#x2F;product&amp;#x2F;1118530802&amp;#x2F;&lt;/a&gt;&lt;p&gt;I learned from the second edition, but I&amp;#x27;ve heard even better things about the third. As you&amp;#x27;re working through it, create a project with real data and from-scratch re-implement a data warehouse as you go. It doesn&amp;#x27;t really matter what you tackle, but I personally like ETLing either data gather from web crawling a single site[0] or push in a weekly gathered wikipedia dump. You&amp;#x27;ll learn many of the foundational reasons for all the tools the industry uses, which will make it very easy for you to get up to speed on them and to make the right choices about when to introduce them. I personally tend to favour tools that have an API or CLI so I can coordinate tasks without needing to click around, but many others like a giant GUI so they can see data flows graphically. Most good tools have at least some measure of both.&lt;p&gt;[0] Use something like Scrapy for python (or Mechanize for ruby) with CSS selectors and use the extension Inspector Gadget to quickly generate CSS selectors.</text></item></parent_chain><comment><author>endlessvoid94</author><text>Agreed. This book will give you a fantastic way to think about ETL strategy rather than simply pointing you to the latest library.&lt;p&gt;Some of the recent popular toolkits &amp;#x2F; services aren&amp;#x27;t &amp;quot;real&amp;quot; ETL -- they simply move data from one place to another. This is obviously a crucial part of ETL, but it&amp;#x27;s not the hard part. And without an understanding of data warehousing such as from this book, it will not be easy to discern the difference.&lt;p&gt;(This is based on many conversations with people on both sides of the table.)</text></comment>
<story><title>Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas?</title><text>Hi HN,&lt;p&gt;I&amp;#x27;m a seasoned Python software developer. Recently I have found a new obsession with data processing, management, engineering etc ... I&amp;#x27;d like to (eventually) branch off into that field but I find the lack of beginner friendly resources is slowing me down. All I can find is spark, hadoop related articles (I know these are prominent in the field, but I want to learn to walk before I run). So If any of you have pointers, websites, project ideas I can start to get a good grasp of all the fundamental concepts, I&amp;#x27;d really appreciate it.&lt;p&gt;Thanks a lot in advance</text></story><parent_chain><item><author>3pt14159</author><text>Spark, etc, are great, but honestly if you&amp;#x27;re just getting started I would forget all about existing tooling that is geared towards people working at 300 person companies and I would read The Data Warehouse ETL Toolkit by Kimball:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;gp&amp;#x2F;product&amp;#x2F;1118530802&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;gp&amp;#x2F;product&amp;#x2F;1118530802&amp;#x2F;&lt;/a&gt;&lt;p&gt;I learned from the second edition, but I&amp;#x27;ve heard even better things about the third. As you&amp;#x27;re working through it, create a project with real data and from-scratch re-implement a data warehouse as you go. It doesn&amp;#x27;t really matter what you tackle, but I personally like ETLing either data gather from web crawling a single site[0] or push in a weekly gathered wikipedia dump. You&amp;#x27;ll learn many of the foundational reasons for all the tools the industry uses, which will make it very easy for you to get up to speed on them and to make the right choices about when to introduce them. I personally tend to favour tools that have an API or CLI so I can coordinate tasks without needing to click around, but many others like a giant GUI so they can see data flows graphically. Most good tools have at least some measure of both.&lt;p&gt;[0] Use something like Scrapy for python (or Mechanize for ruby) with CSS selectors and use the extension Inspector Gadget to quickly generate CSS selectors.</text></item></parent_chain><comment><author>dionidium</author><text>I interned for a data warehousing team when I was in college (a random assignment) and this is the book everybody there lived by and recommended.</text></comment>