<story><title>How We Knew It Was Time to Leave the Cloud</title><url>https://about.gitlab.com/2016/11/10/why-choose-bare-metal/?</url></story><parent_chain><item><author>ocdtrekkie</author><text>At a certain scale, it&amp;#x27;s better to just take that responsibility rather than rely on someone to be able to handle that burden. When something fails, the worst thing is being out of control of it and unable to do anything about it.&lt;p&gt;Last time I had a failed storage controller, HP delivered a replacement in four hours. Last time a service provider went down, I had no visibility into the repair process, and had to explain there was nothing I could do... For five days.&lt;p&gt;It seems like you&amp;#x27;ve outright admitted you can&amp;#x27;t guarantee what they need, yet you still urge them not to leave your business model. Maybe come back when your business model can meet their needs.</text></item><item><author>boulos</author><text>See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency &amp;lt; 1ms!), but high-performance, guaranteed networked storage is just tricky.&lt;p&gt;As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don&amp;#x27;t offer the guarantee, because we don&amp;#x27;t want you to believe it can&amp;#x27;t fail.</text></item><item><author>Dwolb</author><text>I think what you&amp;#x27;re seeing in this post has product implications for Google Cloud.&lt;p&gt;The OP is asking for guaranteed IOPS&amp;#x2F;latency SLA for which they&amp;#x27;re willing to pay HUGE money for by ROLLING THEIR OWN.&lt;p&gt;Possibly think about high service pricing tiers for systems that require it.  This is a standard operations problem that can co-exist within pooled service models.</text></item><item><author>boulos</author><text>I appreciate that you didn&amp;#x27;t sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for the storage layer.&lt;p&gt;For what it&amp;#x27;s worth, hardware doesn&amp;#x27;t provide an IOPS&amp;#x2F;latency SLA either ;). In all seriousness, we (Google, all providers) struggle with deciding what we can strictly promise. Offering you a &amp;quot;guaranteed you can hit 50k IOPS&amp;quot; SLA isn&amp;#x27;t much comfort if we know that all it takes is a single ToR failure for that to be not true (providers could still offer it, have you ask for a refund if affected, etc. but your experience isn&amp;#x27;t changed).&lt;p&gt;All that said, I would encourage you to reconsider. I know you&amp;#x27;re frustrated, but rolling your own infrastructure just means you have to build systems even better than the providers. On the plus side, when it&amp;#x27;s your fault, it&amp;#x27;s your fault (or the hardware vendor, or the colo facility). You&amp;#x27;ve been through a lot already, but I&amp;#x27;d suggest you&amp;#x27;d be better off returning to AWS or coming to us (Google) [Note: Our PD offering historically allowed up to 10 TiB per disk and is now a full 64 TiB, I&amp;#x27;m sorry if the docs were confusing].&lt;p&gt;Again, I&amp;#x27;m not saying this to have you come to us or another cloud provider, but because I honestly believe this would be a huge time sink for GitLab. Instead of focusing on your great product, you&amp;#x27;d have to play &amp;quot;Let&amp;#x27;s order more storage&amp;quot; (honestly managing Ceph has a similar annoyance). I&amp;#x27;m sorry you had a bad experience with your provider, but it&amp;#x27;s not all the same. Feel free to reach out to me or others, if you want to chat further.&lt;p&gt;Disclosure: I work on Google Cloud.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gitlab.com&amp;#x2F;gitlab-com&amp;#x2F;www-gitlab-com&amp;#x2F;merge_requests&amp;#x2F;1097&amp;#x2F;diffs&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gitlab.com&amp;#x2F;gitlab-com&amp;#x2F;www-gitlab-com&amp;#x2F;merge_requests&amp;#x2F;...&lt;/a&gt;</text></item></parent_chain><comment><author>ckozlowski</author><text>One of the things I stress when talking to customers is that when one is moving to cloud, it&amp;#x27;s not just the business model that changes, but the architecture model.&lt;p&gt;That may sound like cold comfort on the face of it, but it&amp;#x27;s key to getting the most out of cloud and exceed the possibilities of an on-prem architecture. Rule #1 is, everything fails. The key advantage to a good cloud provider (and there are many) is not that they can deliver a guarantee against failure (as boulos stated correctly) but that they&amp;#x27;ll allow you to design for failure. The issue becomes when the architecture in the cloud resembles that which was on-premise. While there&amp;#x27;s still some advantages, they&amp;#x27;re markedly fewer, and as you said, there&amp;#x27;s nothing you can do to prioritize your fix.&lt;p&gt;They key to having a good cloud deployment is effectively utilizing the features that eliminate single points of failure so that the same storage controller failure that might knock you out in your on-prem can&amp;#x27;t knock you out in the cloud, even though the repair time for the latter might be longer. That brings its own challenges, but brings huge advantages when it comes together.&lt;p&gt;Disclosure: I work for AWS.</text></comment>
<story><title>How We Knew It Was Time to Leave the Cloud</title><url>https://about.gitlab.com/2016/11/10/why-choose-bare-metal/?</url></story><parent_chain><item><author>ocdtrekkie</author><text>At a certain scale, it&amp;#x27;s better to just take that responsibility rather than rely on someone to be able to handle that burden. When something fails, the worst thing is being out of control of it and unable to do anything about it.&lt;p&gt;Last time I had a failed storage controller, HP delivered a replacement in four hours. Last time a service provider went down, I had no visibility into the repair process, and had to explain there was nothing I could do... For five days.&lt;p&gt;It seems like you&amp;#x27;ve outright admitted you can&amp;#x27;t guarantee what they need, yet you still urge them not to leave your business model. Maybe come back when your business model can meet their needs.</text></item><item><author>boulos</author><text>See my note about SLAs and ToR failures. We probably could promise something for our Local SSD offering (tail latency &amp;lt; 1ms!), but high-performance, guaranteed networked storage is just tricky.&lt;p&gt;As I said, rolling your own will not give you a guarantee, it will just give you the responsibility for failure. We don&amp;#x27;t offer the guarantee, because we don&amp;#x27;t want you to believe it can&amp;#x27;t fail.</text></item><item><author>Dwolb</author><text>I think what you&amp;#x27;re seeing in this post has product implications for Google Cloud.&lt;p&gt;The OP is asking for guaranteed IOPS&amp;#x2F;latency SLA for which they&amp;#x27;re willing to pay HUGE money for by ROLLING THEIR OWN.&lt;p&gt;Possibly think about high service pricing tiers for systems that require it.  This is a standard operations problem that can co-exist within pooled service models.</text></item><item><author>boulos</author><text>I appreciate that you didn&amp;#x27;t sling mud at Azure, but re-reading the commit for the move to Azure [1] there were tell tale signs then that it might be bumpy for the storage layer.&lt;p&gt;For what it&amp;#x27;s worth, hardware doesn&amp;#x27;t provide an IOPS&amp;#x2F;latency SLA either ;). In all seriousness, we (Google, all providers) struggle with deciding what we can strictly promise. Offering you a &amp;quot;guaranteed you can hit 50k IOPS&amp;quot; SLA isn&amp;#x27;t much comfort if we know that all it takes is a single ToR failure for that to be not true (providers could still offer it, have you ask for a refund if affected, etc. but your experience isn&amp;#x27;t changed).&lt;p&gt;All that said, I would encourage you to reconsider. I know you&amp;#x27;re frustrated, but rolling your own infrastructure just means you have to build systems even better than the providers. On the plus side, when it&amp;#x27;s your fault, it&amp;#x27;s your fault (or the hardware vendor, or the colo facility). You&amp;#x27;ve been through a lot already, but I&amp;#x27;d suggest you&amp;#x27;d be better off returning to AWS or coming to us (Google) [Note: Our PD offering historically allowed up to 10 TiB per disk and is now a full 64 TiB, I&amp;#x27;m sorry if the docs were confusing].&lt;p&gt;Again, I&amp;#x27;m not saying this to have you come to us or another cloud provider, but because I honestly believe this would be a huge time sink for GitLab. Instead of focusing on your great product, you&amp;#x27;d have to play &amp;quot;Let&amp;#x27;s order more storage&amp;quot; (honestly managing Ceph has a similar annoyance). I&amp;#x27;m sorry you had a bad experience with your provider, but it&amp;#x27;s not all the same. Feel free to reach out to me or others, if you want to chat further.&lt;p&gt;Disclosure: I work on Google Cloud.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gitlab.com&amp;#x2F;gitlab-com&amp;#x2F;www-gitlab-com&amp;#x2F;merge_requests&amp;#x2F;1097&amp;#x2F;diffs&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gitlab.com&amp;#x2F;gitlab-com&amp;#x2F;www-gitlab-com&amp;#x2F;merge_requests&amp;#x2F;...&lt;/a&gt;</text></item></parent_chain><comment><author>boulos</author><text>That is quite honestly amazing turn around. Most people aren&amp;#x27;t in a position to get a replacement delivered from a vendor that quickly, but if you are, awesome. Again though, there&amp;#x27;s a big gap between simple, local storage (we could probably provide a tighter SLO&amp;#x2F;SLA on Local SSD for example) and networked storage as a service.&lt;p&gt;As someone else alluded to downthread: anyone claiming they can provide guaranteed throughput and latency on networked block devices at arbitrary percentiles in the face of hardware failure is misleading you. I don&amp;#x27;t disagree that you might feel better and have more visibility into what&amp;#x27;s going on when it&amp;#x27;s your own hardware, but it&amp;#x27;s an apples and oranges comparison.</text></comment>