<story><title>DuckDB – An in-process SQL OLAP database management system</title><url>https://duckdb.org/</url></story><parent_chain></parent_chain><comment><author>SnowflakeOnIce</author><text>DuckDB is terrific. I&amp;#x27;m bullish on its potential for simplifying many big data pipelines. Particularly, it&amp;#x27;s plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc.&lt;p&gt;I have successfully used DuckDB like above for preparing an ML dataset from about 100GB of input.&lt;p&gt;DuckDB is undergoing rapid development these days. There have been format-breaking changes and bugs that could lose data. I would not yet trust DuckDB for long-term storage or archival purposes. Parquet is a better choice for that.</text></comment>
<story><title>DuckDB – An in-process SQL OLAP database management system</title><url>https://duckdb.org/</url></story><parent_chain></parent_chain><comment><author>d_watt</author><text>I&amp;#x27;m duck-curious.&lt;p&gt;Looking at how it&amp;#x27;s deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB.&lt;p&gt;EG if you deployed new code&lt;p&gt;1. Do you have a stateful machine you&amp;#x27;re doing an old school &amp;quot;Kill the old process, start the new process&amp;quot; deploy, and there&amp;#x27;s some duckdb file on disk that is maintained?&lt;p&gt;2. Or do you back that duckdb file in some sort of shared disk (Eg EBS), and have a rolling deploy where multiple applications access the same DB at the same time?&lt;p&gt;3. Or is DuckDB is treated as ephemeral, and you&amp;#x27;re using it to process data on the fly, so persisted state isn&amp;#x27;t an issue?</text></comment>