<story><title>Intel Shuts Down Lustre File System Business</title><url>https://www.nextplatform.com/2017/04/20/intel-shuts-lustre-file-system-business/</url></story><parent_chain><item><author>epistasis</author><text>This entire space is littered with hard to use and&amp;#x2F;or flawed products. It&amp;#x27;s extremely difficult to get right. And even things like HDFS, which redefine the problem into something much much more manageable and with better semantics for distributed computing, have had their own issues.&lt;p&gt;Take, for example, my favorite storage system Ceph. As I understand it was originally going to be CephFS, with multiple metadata servers and lots of distributed POSIX goodness. However, in the 10+ years its been in development, the parts that have gotten tons of traction and have widespread use are seemingly one-off side projects from the underlying storage system: object storage and the RBD block device interfaces. Only in the past 12 months is CephFS becoming production ready. But only with a single metadata server, and the multiple metadata servers are still being debugged.&lt;p&gt;With Ceph, some of these timing issues are that the market for object store and network-based block devices are dwarf the market for  distributed POSIX. But I bring it up to point out that distributed POSIX is also just a really really hard problem, with limited use cases. It&amp;#x27;s super convenient for getting an existing Unix executable to run on lots of machines at once. But that convenience may not be worth the challenges it imposes on the infrastructure.</text></item></parent_chain><comment><author>X-Istence</author><text>Object Storage and layered on top of that RBD are much easier to get right.&lt;p&gt;CephFS simply didn&amp;#x27;t gain as much traction because it made sense to just store objects in many cases, and let something else worry about what is stored and where. A massive distributed file system is not nearly as necessary as people make it out to be for a lot of different workloads.&lt;p&gt;RBD works so well because it can replace iSCSI&amp;#x2F;FC storage and provide failover and redundancy, and as you scale servers it gets faster. We had 20 Gbit&amp;#x2F;sec bandwidth for our storage network (to each hypervisor), with 20 Gbit&amp;#x2F;sec for the front-end Ceph access network, and Ceph RBD easily saturated it on the hypervisor side. Spinning up VM&amp;#x27;s in OpenStack was a pleasure and soooooo quick (RBD backed images too).&lt;p&gt;We looked at increasing the hypervisor to 40 Gbit&amp;#x2F;sec, but decided against it for cost reasons, and in reality we had very little workloads that actually needed that much storage IO.</text></comment>
<story><title>Intel Shuts Down Lustre File System Business</title><url>https://www.nextplatform.com/2017/04/20/intel-shuts-lustre-file-system-business/</url></story><parent_chain><item><author>epistasis</author><text>This entire space is littered with hard to use and&amp;#x2F;or flawed products. It&amp;#x27;s extremely difficult to get right. And even things like HDFS, which redefine the problem into something much much more manageable and with better semantics for distributed computing, have had their own issues.&lt;p&gt;Take, for example, my favorite storage system Ceph. As I understand it was originally going to be CephFS, with multiple metadata servers and lots of distributed POSIX goodness. However, in the 10+ years its been in development, the parts that have gotten tons of traction and have widespread use are seemingly one-off side projects from the underlying storage system: object storage and the RBD block device interfaces. Only in the past 12 months is CephFS becoming production ready. But only with a single metadata server, and the multiple metadata servers are still being debugged.&lt;p&gt;With Ceph, some of these timing issues are that the market for object store and network-based block devices are dwarf the market for  distributed POSIX. But I bring it up to point out that distributed POSIX is also just a really really hard problem, with limited use cases. It&amp;#x27;s super convenient for getting an existing Unix executable to run on lots of machines at once. But that convenience may not be worth the challenges it imposes on the infrastructure.</text></item></parent_chain><comment><author>zzzcpan</author><text>&amp;quot;This entire space is littered with hard to use and&amp;#x2F;or flawed products.&amp;quot;&lt;p&gt;None of the internet facing services actually need a strongly consistent POSIX filesystem that scales across multiple datacenters and a huge chunk of them won&amp;#x27;t put up with corresponding latencies of something like that for mere convinience. So the products are not really flawed, they just don&amp;#x27;t need to do those things.</text></comment>