<story><title>Downloading files from S3 with multithreading and Boto3</title><url>https://emasquil.github.io/posts/multithreading-boto3/</url></story><parent_chain></parent_chain><comment><author>vtuulos</author><text>It is a bit of a hidden gem but Metaflow includes a Boto-based highly parallelized, error-tolerant S3 client that Netflix uses routinely to get 10-20Gbps throughput between EC2 and S3.&lt;p&gt;Technically it is independent from Metaflow, so you could use it as a stand-alone, high-performance S3 client.&lt;p&gt;See docs here &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.metaflow.org&amp;#x2F;metaflow&amp;#x2F;data#store-and-load-objects-in-a-metaflow-flow&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.metaflow.org&amp;#x2F;metaflow&amp;#x2F;data#store-and-load-objec...&lt;/a&gt;&lt;p&gt;And code here &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Netflix&amp;#x2F;metaflow&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;metaflow&amp;#x2F;datatools&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Netflix&amp;#x2F;metaflow&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;metaflow&amp;#x2F;dat...&lt;/a&gt;&lt;p&gt;(I wrote it originally - AMA if curious)</text></comment>
<story><title>Downloading files from S3 with multithreading and Boto3</title><url>https://emasquil.github.io/posts/multithreading-boto3/</url></story><parent_chain></parent_chain><comment><author>ak217</author><text>The AWS Go SDK now has a connection pool based S3 download&amp;#x2F;upload manager API that allows saturating your (e.g. 40Gbit&amp;#x2F;s EC2-S3) network connection using far less memory and CPU than is possible with Python.&lt;p&gt;A colleague of mine developed this tool to make this functionality available in a CLI: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;chanzuckerberg&amp;#x2F;s3parcp&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;chanzuckerberg&amp;#x2F;s3parcp&lt;/a&gt;</text></comment>