<story><title>Fast analysis with DuckDB and Pyarrow</title><url>https://tech.gerardbentley.com/python/data/intermediate/2022/04/26/holy-duck.html</url></story><parent_chain></parent_chain><comment><author>minimaxir</author><text>The fact that most data analysis&amp;#x2F;ETL tutorials on the internet have converged on the same CSV&amp;#x2F;pandas tactics over the past decade is disappointing when newer tools demonstrated here such as DuckDB&amp;#x2F;Arrow have practical advantages without much code complexity overhead.&lt;p&gt;This post also links to another discussion about the Parquet data format (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;pythonspeed.com&amp;#x2F;articles&amp;#x2F;pandas-read-csv-fast&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;pythonspeed.com&amp;#x2F;articles&amp;#x2F;pandas-read-csv-fast&amp;#x2F;&lt;/a&gt;), also supported by Arrow, which is also extremely useful but I never see anyone talking about it. Granted, Parquet data can&amp;#x27;t natively be imported into Excel which is likely the main cause.</text></comment>
<story><title>Fast analysis with DuckDB and Pyarrow</title><url>https://tech.gerardbentley.com/python/data/intermediate/2022/04/26/holy-duck.html</url></story><parent_chain></parent_chain><comment><author>jagtesh</author><text>First of all, thanks for sharing this OP! So glad to see a way to query a df using SQL without further transformation.&lt;p&gt;Arrow has been truly revolutionary in this regard, providing a solid in-memory data format (with performant APIs in many languages) for interchange between different engines and even formats.&lt;p&gt;You can go from ORC to Parset to CSV on a local FS or S3.&lt;p&gt;With DuckDB, it’s like you can build your own AWS Athena at likely a fraction of the cost. Now if only someone would integrate vaex with DuckDB, it will make your powerful Apple Silicon machines a compelling alternative to running a full fledged Spark&amp;#x2F;Hadoop cluster.</text></comment>