<story><title>Kubernetes Failure Stories</title><url>https://srcco.de/posts/kubernetes-failure-stories.html</url></story><parent_chain></parent_chain><comment><author>m0zg</author><text>It&amp;#x27;s not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can&amp;#x27;t re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE&amp;#x27;s make infrastructure &amp;quot;just work&amp;quot;. I wish there was something drastically simpler. I don&amp;#x27;t need three dozen persistent volume providers, or the ability to e.g. replace my network plugin or DNS provider, or load balancer. I want a sane set of defaults built-in. I want easy access to persistent data (currently a bit of a nightmare to set up in your own cluster). I want a configuration setup that can take command line params without futzing with templating and the like. As horrible and inconsistent as Borg&amp;#x27;s BCL is, it&amp;#x27;s, IMO, an improvement over what K8S uses.&lt;p&gt;Most importantly: I want a lot fewer moving parts than it currently has. Being &amp;quot;extensible&amp;quot; is a noble goal, but at some point cognitive overhead begins to dominate. Learn to say &amp;quot;no&amp;quot; to good ideas.&lt;p&gt;Unfortunately there&amp;#x27;s a lot of K8S configs and specific software already written, so people are unlikely to switch to something more manageable. Fortunately if complexity continues to proliferate, it may collapse under its own weight, leaving no option but to move somewhere else.</text></comment>
<story><title>Kubernetes Failure Stories</title><url>https://srcco.de/posts/kubernetes-failure-stories.html</url></story><parent_chain></parent_chain><comment><author>dvnguyen</author><text>Having used Docker Compose&amp;#x2F;Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn&amp;#x27;t really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments.&lt;p&gt;The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook Air 2013 and a Ubuntu Thinkpad. Both have VTx enabled and Docker and VirtualBox running flawlessly. Their online interactive tutorial was good though, enough for the learning purpose.&lt;p&gt;Production setup is a bigger disappointment. The only easy and reliable ways to have a production grade Kubernetes cluster are to lock yourself into either a big player cloud provider, or an enterprise OS (Redhat&amp;#x2F;Ubuntu), or introduce a new layer on top of Kubernetes [1]. Locking myself into enterprise Ubuntu&amp;#x2F;Redhad is expensive, and I&amp;#x27;m not comfortable with adding a new, moving, unreliable layer on top of Kubernestes which is built on top of Docker. One thing I like about the Docker movement is that they commoditize infrastructure and reduce lock-ins. I can design my infrastructure so it can utilize an open source based cloud product first and easily move to others or self-host if needed. With Kubernetes, things are going the other way. Even if I never moved out of the big 3 (AWS&amp;#x2F;Azure&amp;#x2F;GCloud), the migration process could be painful since their Kubernetes may introduce further lock-ins for logging, monitoring, and so on.&lt;p&gt;[1]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;kubernetes.io&amp;#x2F;docs&amp;#x2F;setup&amp;#x2F;pick-right-solution&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;kubernetes.io&amp;#x2F;docs&amp;#x2F;setup&amp;#x2F;pick-right-solution&amp;#x2F;&lt;/a&gt;</text></comment>