<story><title>Gcsfuse: A user-space file system for interacting with Google Cloud Storage</title><url>https://github.com/GoogleCloudPlatform/gcsfuse</url></story><parent_chain><item><author>carbocation</author><text>I do scientific computing in google cloud. When I first got started, I heavily relied on GCSFuse. Over time, I have encountered enough trouble that I no longer use it for the vast majority of my work. Instead, I explicitly localize the files I want to the machine that will be operating on them, and this has eliminated a whole class of slowdown bugs and availability bugs.&lt;p&gt;The scale of data for my work is modest (~50TB, ~1 million files total, about 50k files per &amp;quot;directory&amp;quot;).</text></item></parent_chain><comment><author>ashishbijlani</author><text>FUSE does not work well with a large number of small files (due to high metadata ops such as inode&amp;#x2F;dentry lookups).&lt;p&gt;ExtFUSE (optimized FUSE with eBPF) [1] can offer you much higher performance. It caches metadata in the kernel to avoid lookups in user space. Disclaimer: I built it.&lt;p&gt;1. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;extfuse&amp;#x2F;extfuse&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;extfuse&amp;#x2F;extfuse&lt;/a&gt;</text></comment>
<story><title>Gcsfuse: A user-space file system for interacting with Google Cloud Storage</title><url>https://github.com/GoogleCloudPlatform/gcsfuse</url></story><parent_chain><item><author>carbocation</author><text>I do scientific computing in google cloud. When I first got started, I heavily relied on GCSFuse. Over time, I have encountered enough trouble that I no longer use it for the vast majority of my work. Instead, I explicitly localize the files I want to the machine that will be operating on them, and this has eliminated a whole class of slowdown bugs and availability bugs.&lt;p&gt;The scale of data for my work is modest (~50TB, ~1 million files total, about 50k files per &amp;quot;directory&amp;quot;).</text></item></parent_chain><comment><author>laurencerowe</author><text>These file systems are not a good fit for large numbers of small files. Their sweet spot is working with large (~GB+) files which are mostly read from beginning to end. I’ve mostly used them for bioinformatics stuff.</text></comment>