How Habitat can Boost Your Chef Ecosystem
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Sharma, Amulya
Attend this to learn how you can take advantage of Habitat and enhance your current Chef-based ecosystem. We will share our journey and learnings from building "software as a service delivery system" with Chef and how Habitat's packaging, supervision and service discovery made it simpler, faster and more reliable. You will take home: Where to use Chef and where to introduce Habitat. Benefits of using Habitat. How distributed and complex system deployment can be simplified with Habitat. You will see Chef and Habitat live, in action, delivering a highly available ELK stack (ElasticSearch, Logstash and Kibana) within minutes.
 Hello everyone. Good afternoon. So out of the two names here, I am Amuria, he is Deepak. Before I start, let me ask you how many of you are using Chef in production? Awesome. That's what I was expecting. How many of you are using Habitat in production? Oh, amazing. It's pretty obvious that not many people are using it in production. But how many of you used Habitat in some capacity? Like in on your local machine to do something, create a package? So in this session, we primarily cover how we were using Chef and how we are using Chef and from at what places we introduced Habitat. Just let's get right into it. It is a technical talk. So there will be some code, there will be some demos, node convergence and things like that. So please try to stay awake. Do not fall asleep and get the best out of it. I would like to start with this code from Tim O'Reilly, from O'Reilly media that Silicon Valley is underestimating the impact of IoT, which is kind of crazy. Like why Silicon Valley is underestimating, they are all for it, right? So I'd like to explain why he thinks that we are underestimating with something called as a power of 1%. So let's get V at GE, have many business units. One of the business units is GE aviation. Every two seconds, the GE jet engine takes off and it is 70% likely the flight which you took was running on a GE jet engine to come to Austin because 70% of the planes are running GE jet engine. Every engine generate 500 GB of data per flight, which is insane amount of data. What if we can get all the data from the engine into our cloud, do some analytics on it and try to understand the patterns and behaviors which engine is running through. If we can analyze all of that, maybe there is a potential opportunity to just save 1% of fuel and if we can do that, that's equal to 100 million gallons of fuel saved every year just in US. So that's the power of 1% which I was talking about. Same with the locomotive, GE has GE transportation, 1% more on time performance, 100,000 minutes saved just in UK. Energy, 1% more efficiency, you can generate more energy to power Canada for the whole year. Healthcare, 1% more efficiency in the healthcare devices because of the data collection and the analytics on it, 45,000 additional life saved. Production, same thing, 10 billion more output just by 1%. So that's what the power of 1%. So this is a quote from our CEO. He says that every industrial company need to become a technology company or information technology company because that's the only way to guarantee the uptime, to guarantee the asset is performing at the desired performance level. Whether you like it or not, but you have to become an information technology company and that's the transition GE is going through right now. But the problem is how do we make all it happen? So many data is generating at the asset 500 GB per flight. How can we get all of this and do those analytics which we wanted to do? It's very simple. It takes just three simple steps. Edge, you create a firm software which runs on the edge, in this case GE, winterbind and then you create the biggest possible cloud because you need to absorb like a lot of data and then you run the analytics on the data. It's that simple, like three simple steps. Anybody can do that. Super easy. You can do AI and ML because we all love artificial intelligence and machine learning. So we've been doing sort of these three simple steps and the team from we are coming from is mostly focused on the cloud, the middle, the biggest possible cloud you can think of. A little bit of our cloud, the name of our cloud offering is predicts. It's available in multiple point of presence throughout the globe. We have tens of thousands of machines running in our cloud with tens of thousands of developers developing and running tens of thousands of applications. We also offer software services tax. We have thousands of software services tax running in our cloud. It's just to give you the scale like what we are operating in. When we were building this cloud, we need to solve two problems. The first problem one was what we are calling as a stateless problem. We need to have a stateless environment where any developer can push its web applications. The other problem we need to solve was the stateful where developer can also persist their data like database as a service, caching as a service and so on. For it is stateless. There were many solutions in the industry when we started building the cloud like cloud foundry, Kubernetes, missiles. It was very easy to pick just one of these and put together as a stateless runtime. But for a stateful runtime, there were not many solutions around. The state of the container with the persisted layer was not that matured. So it was a problem for us to how do we solve this stateful problem? How do we create a runtime which can keep the data if the container or the machine moves somewhere else? In order to solve this problem, we created our own runtime in which we call as a buffet. The reason why we call it buffet because all the names from the kitchen was already taken by Chef. So we need to do something just outside the kitchen and say, okay, what comes to the kitchen from outside the kitchen? That's the kitchen of the buffet. So with Chef, knife and all those things, you prepare a buffet and then serve it. A little bit of our buffet, about the buffet. Buffet is built on Chef from ground up. Chef is at the core of the buffet. We use Chef for reasoning and configuration management with Chef. For the VM creations and the storage allocation, we use infrastructure as a service. And buffet can run on three virtualization, Azure, Amazon Web Services and VMware. For configuration management on the machine, we use Chef. And with all of this, we were able to create persisted services, software services, for example, Redis, Elk and Rabbit MQ. These are the three services which we run with buffet in production today. A little bit in detail. So to start interacting with our system, there's an API. Everything starts with an API. So you can say, I need a stack by sending a create request. The create request has a payload which has like, hey, I need three elastic search nodes, two log stash, three Kibana. So you specify what you want. That request goes to a queue. At the same time, it also gets persisted to a Chef server in the Chef data bag. Then the queue item is being picked by the worker machines. The worker machine's job is to read the payload and then start a Chef provisioning recipe. Now, the job of the Chef provisioning recipe is to just create the stack, like, okay, the number of machines I was requesting, just create the machine, create the load balancer. That's it. At that point of time, Chef comes into the picture and the job of Chef is run the service cookbook and put that particular software on those machines. So in this case, putting elastic search, log stash and Kibana. And Chef node search come into the picture and cluster the machines together. So again, zoomed in in very simple terms. Chef provisioning, create all the machines and the load balancer. Chef comes in, put the software on it and cluster them together. Life is good. But life is actually not that great because by doing all of this dynamic cluster allocation in Chef is not that simple, not that easy. And we'll go through why. The problem number one is the dependency. In order to converge these software in this example, elastic search, log stash and Kibana on a machine, you need to download the dependencies. And these are the dependency looks like just for these three software. You need to have Java, node, Python, all these software. And this is the cookbook, compile cookbook list on the machine. So it needs to download literally the whole internet to converge one machine. That is uncool. The problem with that is it's time consuming. And S3 goes down from time to time. We all experienced when the S3 went down, all the package vanished. Nothing was coming down on the machine. All the convergence start failing for no reason because S3 went down and the whole internet is on S3. And the third problem is security. Security doesn't like it at all. They hate you for downloading stuff from internet, right? All over the internet. And you don't even know what you're going to download. So these are the problems. The second problem is there is no good service discovery built into the Chef or Chef convergence. In order to do the service discovery, what you need to do is you need to wait till your node gets registered to the Chef server. And then you need to do a Chef search to get the IP of the node and then write a configuration file and distribute it among all your machines and restart the services. That's kind of an ugly service discovery model. But that's what we have been doing because that's the only way you can do totally in Chef. The advanced use cases like finding a leader is not also being thought through in the Chef implementation, just Chef implementation. So for example, we run Redis and Redis is one of the use cases, the finding the leader. So from the master, I just want to publish the domain name from the master node only. In order to do that, either I need to have an HAProxy in front of it or I need to write some custom logic in my cookbook that figure out how it has to figure out where the master is and then publish the route. So advanced use cases like finding leader is not very well handled in just Chef recipes. You need to write a complex logic within your recipe. The last is the configuration change. In order, let's say if there is a configuration change in your services stack, let's say you are changing the elastic search heap from 1GB to 4GB. In order to do that, you also need to write that logic in your recipes or in your cookbook. You need to first make the configuration file, change the value and notify the service that, hey, I changed this value, you better restart. It's your responsibility to do that. So that all makes your cookbook and recipes a little bit complex and ugly. So from life is good to life is crap. I have all these problems. I have the best tool in the world, Chef, and I cannot do these advanced use cases or these advanced problems which everybody have in cluster management or distributed systems. And then, thankfully, this happened last year. Adam presented that he has something which can fit better in these use cases which we've been going through. Then Habitat introduced this new features like build your application packages with hard packages, quick config changes. You don't have to worry about the supervisor takes care of the config changes. Intelligent runtime environment, if you forget to add a service, you just did to, you know, as simple as dropping a file and new service comes up on that machine. Rolling upgrades. So let's say if you want to change something, all of your machines are not getting restarted or reconfigured the same time. You can do one at a time which was not there in the case of Chef. So we all get very excited. Now, at this point, I would like to call Deepak to take you how we took advantage of these awesome features and change the system which was just introduced, the buffet, to take advantage of Habitat. In fact, how we used Habitat to boost our Chef based system. Okay. So we saw our buffet. We were serving some cool services. And then after Habitat was launched, we decided to bring Habitat into the buffet. It might seem weird that there's a Habitat being served with the buffet, but let's see what we changed here. So as we saw earlier that buffet was taking some requests, there were some workers picking up those requests and doing its job. So till the time we bring up our infrastructure based on the request we have received, everything is same. And after that, when it comes to prepare your service on that particular instance on those machines, we bring in Habitat. So now instead of writing service cookbooks, we started writing Habitat plans. We have a plan for Elasticsearch, Logstash and Kibana. After that, we need to integrate all these services together. My Logstash needs to know where Elasticsearch lives. I need not to do anything extra now. I like doing a search with the Chef. Supervisor will send the rumors to every peer in the ring and will tell that, okay, now there is one Elasticsearch available and do a configuration accordingly. So at the end of our runtime solution, we introduced Habitat here. So we'll do a comparison versus Chef versus Chef plus Habitat based runtime which we are running at GE. So there are basically four layers. The first layer is provisioning layer. So in both the cases, we still use Chef provisioning and yes, we use Chef provisioning. You might be thinking that, you know, there are so many other tools also. Chef provisioning fits really good for our use case, the way we are using it and it gives us a really good infrastructure as a code. The second piece is configuration. So in the Chef world, we had so many service cookbooks written and in the Habitat world, now we have all those cookbooks converted as a plan. Third is integration. To do integration of my services in Chef world, I had to do some search, figure out how many nodes, how many clusters are there with registered with the Chef server and I find them and integrate. But with the Habitat, I just leave it on the Habitat supervisor and the Gossy protocol will send the rumors to everyone in the ring and then every node, every supervisor running on a machine will figure out who other services are there and whom I need to bind to. And the last thing is that I have done all these things. Now I need to make sure everything keeps running on my machine. It's not that I configured and I can forget about it. What if it goes down? I need to make sure that I have a desired state configuration all the time on that machine. So we run Chef client as a service at a particular interval on those machines. But with Habitat, again, I need no to worry about running anything because there is already a supervisor which is responsible for all those services to be up and running. If anything goes down, supervisor will make sure that it gets restarted again. So that's the comparison of our buffet old model with just with the Chef and then how we are leveraging Habitat. So a couple of issues or the problems which we were facing, we were discussed. Now we are going to see how did we leverage Habitat to overcome those things. So first thing with Habitat is better packaging. We get to create our own packages and every dependency which is required for that package is embedded. So consider this example. If you have to go manually install elastic search on a machine, you first do update because you want your Ubuntu to be updated. Then install Java, install elastic search. If you need to add a repo, add a repo and all that stuff. There are a couple of steps. But let's say you have built already a package with Habitat for elastic search. What do you need to do? Just one command. Habstart and the elastic search. And it will make sure that your package gets downloaded on the machine and it gets started right away. So I need no to hit even more than one command. Just one command and your service is up. This is in the form of cookbooks. So when we were trying to install elastic search, there were a couple of recipes needs to be included and the resources from elastic search community cookbook. The same thing when we tried to do with Habitat, we just had to write two resources. First to make sure Habitat is installed on the machine and the second to install the package of the service which we want to run. So by doing this, so we had already these cookbooks written for our services because we were running on all of these. Just to install elastic search, we had 150 lines of code in our cookbook. It included some of the standard code which was required by our runtime solution buffet. And when we rewrote all the elastic search cookbook with the Habitat, we saw that we removed 100 lines of the code from there. And actually out of these 50 lines, only 10 lines are for the Habitat and the other 40 are for our standard requirement for the buffet. So we saw a code reduction with the better packaging system. Second solution or the leverage we are getting from Habitat is that it's fast. We can converse faster. So in the Chef world, now we are taking the example of Elk. I need to connect my LogStash to the elastic search. And how I'm going to do that? By doing a search. But if elastic search is still not registered to the Chef server, I cannot find it. So I have to break my convergence in three steps. The first step is I'll bring my infrastructure. The second is I will just bring my elastic search nodes so that I can register them, configure them and service can start, right? And the third convergence would be configuring LogStash and then doing a search to the Chef server and integrating my LogStash to the elastic search. So basically, like, from overall, like, there are three steps, provisioning then elastic search and then the lastly I'll go and configure my LogStash. But if I had to do the same thing with Habitat, first thing is, again, the same convergence, I have to bring my instances, right? But after that, I won't wait until my elastic search is up and running. I would just do one convergence and put everything on each machine. Because if I tell my LogStash to find an elastic search and integrate with it, supervisor will make sure that until it finds an elastic search service in your ring, in your cluster, it won't let LogStash start. It will keep waiting and as soon as there is elastic search available, the rumors are sent to every other peer and they come to know, they will say, okay, now we have some extra nodes which says they are elastic search, let's recompile our configuration and then the configuration file for the LogStash gets updated and the service gets started. So I reduce one convergence and by, so we'll see the time, actual time which we were seeing in the production. When we were doing the first time on-demand service creation or the instance creation for L cluster, it was taking around 10 minutes. But when we switched to Habitat, it was reduced by 30% and we saw that it's getting converged in 7 minutes. This is the first convergence. Now let's consider something happened to my machines. All my machines went down. My data disk is still available because I have to attach the same with the service and service would be restored. So by doing a disaster recovery of machines, machine failover with the Chef, it was the same because it's all thing again. I have to bring up machines, install all the softwares, wait one by one, everything comes up and then attach the disk and start the services. But with the Habitat, doing a disaster, we saw another 30 to 40% reduction because of the packaging of Habitat. Everything with what Habitat requires to run a particular service was within a particular defined space which we could mount to an external device and then only machines went down. We could reattach the same disk and when we converged our recipes, they say, oh, we already have everything. So up to date. And we could recover a cluster within 3 minutes. Disaster recovery failure or maybe let's say there is a huge OS patch upgrade you have to do, right? So with this, it's pretty easy. Scrap the machine, bring a new machine with the new OS and reattach the same disk and Habitat will be happy to run your services again. The third thing, which we had to do, figure out how many nodes or which services are there, right? We had to do the search. How do we do it in Habitat? So doing a search, it's a server operation, right? Your client, your Chef client will make a call to the Chef server. There will be a solar operation and then results will be like sent back to your client and then you do the configuration, right? It's a server, central server operation made by each client. But with Habitat, they have the information locally available with the node, right? They have a census data available on every supervisor. All they need to do is just look up at that census and figure out how many other nodes are there, which I need to configure to. And this thing was moved from the cookbooks to the plants. So it's, you only need to worry about while writing a new service. You need not to worry about when you go and deploy it on a machine. So Habitat has some helper methods, which will let you figure out how many alive members are there for a particular binding, which you want to do for your, a particular service or an application. And then you can configure those things easily in your plants. Finding the leader. So finding a master because master is the only person who can write my data for some software like databases and all that is, right? We at predicts wanted to provide a single way to connect to your service, whether it's a dedicated instance or it's a clustered instance. We would provide you a single connection string URL. And then no matter if your master went down for some reason, your string, the connection string won't change. And behind the scene, we would make sure that, you know, that's connection string is always pointing to the master, right? So how do we do that? Habitat really helped us over there because it has another helper method, which always tells you about a leader. So let me tell you a little bit about leader in Habitat, right? So when you say I have to start a particular service and it is in a leader follower topology, it needs at least three nodes to be there of that particular service in your ring, in your cluster of Habitat supervisors. And as soon as three nodes become available and saying that, you know, I am publishing elastic search, right? The Habitat will start an election and a leader gets elected. So that leader will act as a master for your services. So let's see a demo here. So we have a small application running and we have a simple question to ask that, you know, the Habitat tell us who my leader is right now. So it's telling me a particular IP of the leader from the super habitat ring. And what I'm going to do is that I'll go to this machine and I would stop the supervisor. And as soon as I stop the supervisor, I would see on other nodes in my ring that they would trigger another election to find a new leader because the current leader went down. And if I'm able to see the election nodes logs here. Okay, so this went down the hooks for recompile because one service supervisor is down that sends the message to other nodes. And now let's go back to the UI and try to refresh it. Okay, let me see. Oh, so I'm still recompiling. There are three nodes in the cluster. So that's cool. Okay. Okay, I think go to the UI and like, like you get the idea like what, so if you kill a member in the ring, the leader election happens again, and the habitat supervisor help you elect another leader, which you can use for use cases like Redis, specifically where we were using. You can go to the... Okay, so... It worked. Perfect. Okay, so it took some time. Maybe, you know, the file which I'm trying to use here. So this file, it's a basic HTML file. It's just using that first method, which is always pointing to the leader. And as soon as the new election happens, that method gets updated, like that's returning a new value, and here we see it's a new leader available, right? Okay, so... I had one image for this. It's like there was one leader, and then suddenly it stepped down, and there was a new leader, so... Who will tell me this? Don't worry, there is a habitat to tell us about it. Okay, so next thing we gain from habitat is intelligent runtime, right? Every service which we install, what we do, we write a configuration file for it to start the service as we want, right? So, habitat, again, we can provide the configuration files, or some kind of configuration, which is called as spec files for Supervisor, to tell Supervisor that what kind of service, and in which fashion, like what topology, and what, like, update strategy I have to use with a particular service, and in any services you want to run, right? So all these configurations for the services, you can provide to the Supervisor, and then just leave everything up to the Supervisor, because then Supervisor will make sure that based on the given configuration, it runs each and every service on that particular instance. So there's a default directory, haveSoup default specs, which Supervisor always listens to, or keep a watch on it, and as soon as you drop a new file with some specifications to run a particular service, it will start running that service, right? And, you know, there might be a use case that I'm running a couple of services on a machine, then suddenly I might need to move that service to a different machine, or maybe I might to start a new service on a particular machine, right? So it's very easy. The Habitat makes this thing for us very easy, because all we need to do is drop a file, and then it will bring up a new service, right? We can see this in a demo, and this will work. So in this particular machine, we have configured an Nginx, which was supposed to listen to a Kibana as an upstream server, and we are seeing a bad gateway, so let's go to the machine and see what happened here. So if you see the directory, specs directory, we see that Elasticsearch is there, Logstress, Nginx, and a couple of other services are there, but we don't see a Kibana, right? So I need to install Kibana on this machine. How easy it would be if I had to do with a Chef cookbook, right? I had to write a wrapper cookbook, or maybe use a community cookbook, include that recipe into my existing recipe, push a new version, and then start a Chef client. That would start my service. It will take some time, right? But what I'm going to do here is, I would write a Kibana spec file. I can give you a little bit idea about a spec file, what it contains. The first thing it contains is, you need to tell supervisor what service it has to run. Then there is a group configuration where you can identify or segregate your services in different groups. You can run multiple Elasticsearch in the same supervisor ring, but they can work as a different cluster. You can define the topology if you want to run them as a standalone applications, or you want them to be clustered with each other. Update strategy, bindings. Do they need to bind with someone else? Any other application, database, or service? And a few other configurations. Disad's state is like, I want it to be up all the time, and start style is persistent. That tells the supervisor that when you stop, don't delete this file, because when I will start you again, I want these services to get started again. So all I have to do right now is that pick this file and drop here. And now if I go and check the log, I can see a log coming from Kibana. Hooks got recompiled. It's trying to initialize the service. And finally, a process is getting started, and then server is running at a 5.1.6.0 port. If I go back to my UI, I'm seeing a Kibana starting. I couldn't have done this within the same time if I had to write a new cookbook. But in this case, just six, seven lines in a file, and drop a file on a machine. That's it. Let me see how we are doing before Rick throw me out of the room. So last, but not the least, one of the great benefits which we had in our existing runtime solution is rolling upgrades. So let's say we are running Chef Client as a service on each machine in our cluster. I have to change something or there is a new package or any change in the configuration for a particular service or let's consider the elastic search service. I will write a new cookbook, push it to the Chef server, and then next time when the Chef Client will run, it will download the update and restart my service accordingly. It is very likely that all three nodes of my cluster can start Chef Client at the same time. And if my service gets restarted at the same time, I'll have a downtime. And nobody will be happy about that. So ideally, what I should do is I have to figure out a way that I can send my updates one by one. And Habitat comes in very handy doing that. As soon as you write a new package and you push it to the depot, Habitat will start. So apart from the leader election, which is your master of the cluster, Habitat or Supervisor does another type of election, which is update leader. The leader which is designated to look for a new package on the depot and as soon as a new package is available, download it, install it, and then tell others that this is a new package, it's good, I was able to restart properly. You also do the same. So we have a video to save the time. In this video, what we did that we pushed a new version, and then you can see the node 3, because the node 3 happened to be the update leader, started updating, and you can see the heap size is getting changed. Right now it's like 50.0 GB. Now it told its followers, so node 1 is trying to update, and then the heap size changed to 9 GB, and then the last remaining node will start updating itself, and the heap size will change, because in the change we pushed, we are going to decrease the heap size, so there was the config change, and how it happened? It happened in a rolling fashion, one node at a time, and service was all the time up and running. I think at this time we can take some questions if you guys have any questions. Yeah, that's all. That's how we boosted our Chef ecosystem with Habitat. Thank you. I'd like to ask about the election logic of the topology, the leader and the follower. Is it necessary to keep the number of the cluster to the older number? From my MongoDB experience, the pre-cathet node number should be kept in the older number, because of the little majority in terms of election. No, so all Habitat requires at least three nodes. After that it could be any number. At least three nodes, and I can add as I like. So you said you were able to eliminate the Chef Client step of having to run Chef Client as a service. How are you managing the Hab binary? Are you building golden images? No, the Chef Client is the only responsibility to start the Hab process, and that's it. After that, it's all Hab. You can do it with the images if you really want to eliminate it, but we are not eliminating, because somebody needs to make sure the Hab process is running all the time, and that Chef Client is responsible for. So this is VM runtime, and we do use Chef to manage your VM runtime. You saw in the keynote in the morning that if you are using Kubernetes, you can make sure using the images file, right, and using Kubernetes, you can pass your configuration. So in this scenario, we are using Chef to push Habitat and make sure that Habitat is there, and after that services, Habitat is responsible for them. And Chef is doing another job, basically. Chef is also placing the Habitat spec files with the templates. So there's a cookbook which puts the Habitat binary. It starts the Habitat and plays the files into the spec directory for the desired services on that machine. There are three things Chef is still doing, so we still have Chef in the mix. I was curious, along with that earlier question, about how you actually do the updates to the Hab binaries. Maybe there's a new version that comes out. How you're actually doing the updates to the Hab binary themselves. So there is a cookbook. So there's a cookbook which puts the Hab binary on it. So that cookbook, the only job in the earlier question is that cookbook takes care of. So you need to change the version of the cookbook if you want to roll out the new binary of Habitat. So the question is, how do you manage the DNS name if the load member from one leader to another leader gets elected, right? So we have our own service written to publish a domain route to an app-based routing. So we use the HTTP header. So we publish the route, and how do we decide from where, from which machine we need to publish the route, is based on the leader elections of Habitat. So that service runs in a topology leader follower model, and the plan is written in such a way that it will get executed only on a leader machine. So as soon as the leader is switched, the hooks get recompiled, the service gets restarted, and it's only running on the new leader. So you are publishing the route only from the machine, which is the leader, and how do you decide that using Habitat? And how do you publish the route? We have our custom software which uses the HTTP host name and the app-based routing. We will be at the Habitat corner if you have any other questions about our system. We, like, if you want to see another demo or see some action, services in action with the Habitat, because we run, like, lots of services on our platform. So we would be available to the corner. Yeah, one last thing is that if you like what we are doing at GE, we are always hiring. You guys are very welcome. Thank you.
 Hello everyone. Good afternoon. So out of the two names here, I am Amuria, he is Deepak.  Before I start, let me ask you how many of you are using Chef in production? Awesome.  That's what I was expecting. How many of you are using Habitat in production? Oh, amazing.  It's pretty obvious that not many people are using it in production. But how many of you  used Habitat in some capacity? Like in on your local machine to do something, create  a package? So in this session, we primarily cover how we were using Chef and how we are  using Chef and from at what places we introduced Habitat. Just let's get right into it. It  is a technical talk. So there will be some code, there will be some demos, node convergence  and things like that. So please try to stay awake. Do not fall asleep and get the best  out of it. I would like to start with this code from Tim O'Reilly, from O'Reilly media  that Silicon Valley is underestimating the impact of IoT, which is kind of crazy. Like  why Silicon Valley is underestimating, they are all for it, right? So I'd like to explain  why he thinks that we are underestimating with something called as a power of 1%. So  let's get V at GE, have many business units. One of the business units is GE aviation.  Every two seconds, the GE jet engine takes off and it is 70% likely the flight which  you took was running on a GE jet engine to come to Austin because 70% of the planes are  running GE jet engine. Every engine generate 500 GB of data per flight, which is insane  amount of data. What if we can get all the data from the engine into our cloud, do some  analytics on it and try to understand the patterns and behaviors which engine is running through.  If we can analyze all of that, maybe there is a potential opportunity to just save 1%  of fuel and if we can do that, that's equal to 100 million gallons of fuel saved every  year just in US. So that's the power of 1% which I was talking about. Same with the locomotive,  GE has GE transportation, 1% more on time performance, 100,000 minutes saved just in  UK. Energy, 1% more efficiency, you can generate more energy to power Canada for the whole  year. Healthcare, 1% more efficiency in the healthcare devices because of the data collection  and the analytics on it, 45,000 additional life saved. Production, same thing, 10 billion  more output just by 1%. So that's what the power of 1%. So this is a quote from our CEO.  He says that every industrial company need to become a technology company or information  technology company because that's the only way to guarantee the uptime, to guarantee  the asset is performing at the desired performance level. Whether you like it or not, but you  have to become an information technology company and that's the transition GE is going through  right now. But the problem is how do we make all it happen? So many data is generating  at the asset 500 GB per flight. How can we get all of this and do those analytics which  we wanted to do? It's very simple. It takes just three simple steps. Edge, you create  a firm software which runs on the edge, in this case GE, winterbind and then you create  the biggest possible cloud because you need to absorb like a lot of data and then you  run the analytics on the data. It's that simple, like three simple steps. Anybody can do that.  Super easy. You can do AI and ML because we all love artificial intelligence and machine  learning. So we've been doing sort of these three simple steps and the team from we are  coming from is mostly focused on the cloud, the middle, the biggest possible cloud you  can think of. A little bit of our cloud, the name of our cloud offering is predicts. It's  available in multiple point of presence throughout the globe. We have tens of thousands of machines  running in our cloud with tens of thousands of developers developing and running tens  of thousands of applications. We also offer software services tax. We have thousands of  software services tax running in our cloud. It's just to give you the scale like what  we are operating in. When we were building this cloud, we need to solve two problems.  The first problem one was what we are calling as a stateless problem. We need to have a  stateless environment where any developer can push its web applications. The other problem  we need to solve was the stateful where developer can also persist their data like database  as a service, caching as a service and so on. For it is stateless. There were many solutions  in the industry when we started building the cloud like cloud foundry, Kubernetes, missiles.  It was very easy to pick just one of these and put together as a stateless runtime.  But for a stateful runtime, there were not many solutions around. The state of the container  with the persisted layer was not that matured. So it was a problem for us to how do we solve  this stateful problem? How do we create a runtime which can keep the data if the container  or the machine moves somewhere else? In order to solve this problem, we created our own  runtime in which we call as a buffet. The reason why we call it buffet because all the  names from the kitchen was already taken by Chef. So we need to do something just outside  the kitchen and say, okay, what comes to the kitchen from outside the kitchen? That's  the kitchen of the buffet. So with Chef, knife and all those things, you prepare a buffet  and then serve it. A little bit of our buffet, about the buffet.  Buffet is built on Chef from ground up. Chef is at the core of the buffet. We use Chef  for reasoning and configuration management with Chef. For the VM creations and the storage  allocation, we use infrastructure as a service. And buffet can run on three virtualization,  Azure, Amazon Web Services and VMware. For configuration management on the machine, we  use Chef. And with all of this, we were able to create persisted services, software services,  for example, Redis, Elk and Rabbit MQ. These are the three services which we run with buffet  in production today. A little bit in detail. So to start interacting with our system, there's  an API. Everything starts with an API. So you can say, I need a stack by sending a create  request. The create request has a payload which has like, hey, I need three elastic search  nodes, two log stash, three Kibana. So you specify what you want. That request goes to  a queue. At the same time, it also gets persisted to a Chef server in the Chef data bag. Then  the queue item is being picked by the worker machines. The worker machine's job is to  read the payload and then start a Chef provisioning recipe. Now, the job of the Chef provisioning  recipe is to just create the stack, like, okay, the number of machines I was requesting,  just create the machine, create the load balancer. That's it. At that point of time, Chef comes  into the picture and the job of Chef is run the service cookbook and put that particular  software on those machines. So in this case, putting elastic search, log stash and Kibana.  And Chef node search come into the picture and cluster the machines together. So again,  zoomed in in very simple terms. Chef provisioning, create all the machines and the load balancer.  Chef comes in, put the software on it and cluster them together. Life is good. But life  is actually not that great because by doing all of this dynamic cluster allocation in  Chef is not that simple, not that easy. And we'll go through why. The problem number one  is the dependency. In order to converge these software in this example, elastic search,  log stash and Kibana on a machine, you need to download the dependencies. And these are  the dependency looks like just for these three software. You need to have Java, node, Python,  all these software. And this is the cookbook, compile cookbook list on the machine. So it  needs to download literally the whole internet to converge one machine. That is uncool. The  problem with that is it's time consuming. And S3 goes down from time to time. We all  experienced when the S3 went down, all the package vanished. Nothing was coming down  on the machine. All the convergence start failing for no reason because S3 went down  and the whole internet is on S3. And the third problem is security. Security doesn't like  it at all. They hate you for downloading stuff from internet, right? All over the internet.  And you don't even know what you're going to download. So these are the problems. The  second problem is there is no good service discovery built into the Chef or Chef convergence.  In order to do the service discovery, what you need to do is you need to wait till your  node gets registered to the Chef server. And then you need to do a Chef search to get the  IP of the node and then write a configuration file and distribute it among all your machines  and restart the services. That's kind of an ugly service discovery model. But that's  what we have been doing because that's the only way you can do totally in Chef. The  advanced use cases like finding a leader is not also being thought through in the Chef  implementation, just Chef implementation. So for example, we run Redis and Redis is  one of the use cases, the finding the leader. So from the master, I just want to publish  the domain name from the master node only. In order to do that, either I need to have  an HAProxy in front of it or I need to write some custom logic in my cookbook that figure  out how it has to figure out where the master is and then publish the route. So advanced  use cases like finding leader is not very well handled in just Chef recipes. You need  to write a complex logic within your recipe. The last is the configuration change. In order,  let's say if there is a configuration change in your services stack, let's say you are  changing the elastic search heap from 1GB to 4GB. In order to do that, you also need  to write that logic in your recipes or in your cookbook. You need to first make the  configuration file, change the value and notify the service that, hey, I changed this value,  you better restart. It's your responsibility to do that. So that all makes your cookbook  and recipes a little bit complex and ugly. So from life is good to life is crap. I have  all these problems. I have the best tool in the world, Chef, and I cannot do these advanced  use cases or these advanced problems which everybody have in cluster management or distributed  systems. And then, thankfully, this happened last year. Adam presented that he has something  which can fit better in these use cases which we've been going through. Then Habitat introduced  this new features like build your application packages with hard packages, quick config  changes. You don't have to worry about the supervisor takes care of the config changes.  Intelligent runtime environment, if you forget to add a service, you just did to, you know,  as simple as dropping a file and new service comes up on that machine. Rolling upgrades.  So let's say if you want to change something, all of your machines are not getting restarted  or reconfigured the same time. You can do one at a time which was not there in the case  of Chef. So we all get very excited. Now, at this point, I would like to call Deepak  to take you how we took advantage of these awesome features and change the system which  was just introduced, the buffet, to take advantage of Habitat. In fact, how we used Habitat  to boost our Chef based system. Okay. So we saw our buffet. We were serving  some cool services. And then after Habitat was launched, we decided to bring Habitat  into the buffet. It might seem weird that there's a Habitat being served with the buffet,  but let's see what we changed here. So as we saw earlier that buffet was taking some  requests, there were some workers picking up those requests and doing its job. So till  the time we bring up our infrastructure based on the request we have received, everything  is same. And after that, when it comes to prepare your service on that particular instance  on those machines, we bring in Habitat. So now instead of writing service cookbooks,  we started writing Habitat plans. We have a plan for Elasticsearch, Logstash and Kibana.  After that, we need to integrate all these services together. My Logstash needs to know  where Elasticsearch lives. I need not to do anything extra now. I like doing a search  with the Chef. Supervisor will send the rumors to every peer in the ring and will tell that,  okay, now there is one Elasticsearch available and do a configuration accordingly. So at  the end of our runtime solution, we introduced Habitat here. So we'll do a comparison versus  Chef versus Chef plus Habitat based runtime which we are running at GE. So there are basically  four layers. The first layer is provisioning layer. So in both the cases, we still use  Chef provisioning and yes, we use Chef provisioning. You might be thinking that, you know, there  are so many other tools also. Chef provisioning fits really good for our use case, the way  we are using it and it gives us a really good infrastructure as a code.  The second piece is configuration. So in the Chef world, we had so many service cookbooks  written and in the Habitat world, now we have all those cookbooks converted as a plan.  Third is integration. To do integration of my services in Chef world, I had to do some  search, figure out how many nodes, how many clusters are there with registered with the  Chef server and I find them and integrate. But with the Habitat, I just leave it on the  Habitat supervisor and the Gossy protocol will send the rumors to everyone in the ring  and then every node, every supervisor running on a machine will figure out who other services  are there and whom I need to bind to. And the last thing is that I have done all these  things. Now I need to make sure everything keeps running on my machine. It's not that  I configured and I can forget about it. What if it goes down? I need to make sure that  I have a desired state configuration all the time on that machine. So we run Chef client  as a service at a particular interval on those machines. But with Habitat, again, I need  no to worry about running anything because there is already a supervisor which is responsible  for all those services to be up and running. If anything goes down, supervisor will make  sure that it gets restarted again. So that's the comparison of our buffet old model with  just with the Chef and then how we are leveraging Habitat. So a couple of issues or the problems  which we were facing, we were discussed. Now we are going to see how did we leverage Habitat  to overcome those things. So first thing with Habitat is better packaging. We get to create  our own packages and every dependency which is required for that package is embedded.  So consider this example. If you have to go manually install elastic search on a machine,  you first do update because you want your Ubuntu to be updated. Then install Java, install  elastic search. If you need to add a repo, add a repo and all that stuff. There are a  couple of steps. But let's say you have built already a package with Habitat for elastic  search. What do you need to do? Just one command. Habstart and the elastic search. And it will  make sure that your package gets downloaded on the machine and it gets started right away.  So I need no to hit even more than one command. Just one command and your service is up. This  is in the form of cookbooks. So when we were trying to install elastic search, there were  a couple of recipes needs to be included and the resources from elastic search community  cookbook. The same thing when we tried to do with Habitat, we just had to write two resources.  First to make sure Habitat is installed on the machine and the second to install the  package of the service which we want to run. So by doing this, so we had already these  cookbooks written for our services because we were running on all of these. Just to install  elastic search, we had 150 lines of code in our cookbook. It included some of the standard  code which was required by our runtime solution buffet. And when we rewrote all the elastic  search cookbook with the Habitat, we saw that we removed 100 lines of the code from there.  And actually out of these 50 lines, only 10 lines are for the Habitat and the other 40  are for our standard requirement for the buffet. So we saw a code reduction with the better  packaging system. Second solution or the leverage we are getting from Habitat is that it's  fast. We can converse faster. So in the Chef world, now we are taking the example of Elk.  I need to connect my LogStash to the elastic search. And how I'm going to do that? By doing  a search. But if elastic search is still not registered to the Chef server, I cannot find  it. So I have to break my convergence in three steps. The first step is I'll bring my infrastructure.  The second is I will just bring my elastic search nodes so that I can register them,  configure them and service can start, right? And the third convergence would be configuring  LogStash and then doing a search to the Chef server and integrating my LogStash to the elastic  search. So basically, like, from overall, like, there are three steps, provisioning  then elastic search and then the lastly I'll go and configure my LogStash. But if I had  to do the same thing with Habitat, first thing is, again, the same convergence, I have to  bring my instances, right? But after that, I won't wait until my elastic search is up  and running. I would just do one convergence and put everything on each machine. Because  if I tell my LogStash to find an elastic search and integrate with it, supervisor will make  sure that until it finds an elastic search service in your ring, in your cluster, it  won't let LogStash start. It will keep waiting and as soon as there is elastic search available,  the rumors are sent to every other peer and they come to know, they will say, okay, now  we have some extra nodes which says they are elastic search, let's recompile our configuration  and then the configuration file for the LogStash gets updated and the service gets started.  So I reduce one convergence and by, so we'll see the time, actual time which we were seeing  in the production. When we were doing the first time on-demand service creation or the  instance creation for L cluster, it was taking around 10 minutes. But when we switched to  Habitat, it was reduced by 30% and we saw that it's getting converged in 7 minutes.  This is the first convergence. Now let's consider something happened to my machines. All my  machines went down. My data disk is still available because I have to attach the same  with the service and service would be restored. So by doing a disaster recovery of machines,  machine failover with the Chef, it was the same because it's all thing again. I have  to bring up machines, install all the softwares, wait one by one, everything comes up and then  attach the disk and start the services. But with the Habitat, doing a disaster, we saw  another 30 to 40% reduction because of the packaging of Habitat. Everything with what  Habitat requires to run a particular service was within a particular defined space which  we could mount to an external device and then only machines went down. We could reattach  the same disk and when we converged our recipes, they say, oh, we already have everything.  So up to date. And we could recover a cluster within 3 minutes. Disaster recovery failure  or maybe let's say there is a huge OS patch upgrade you have to do, right? So with this,  it's pretty easy. Scrap the machine, bring a new machine with the new OS and reattach  the same disk and Habitat will be happy to run your services again.  The third thing, which we had to do, figure out how many nodes or which services are there,  right? We had to do the search. How do we do it in Habitat? So doing a search, it's  a server operation, right? Your client, your Chef client will make a call to the Chef server.  There will be a solar operation and then results will be like sent back to your client and  then you do the configuration, right? It's a server, central server operation made by  each client. But with Habitat, they have the information locally available with the node,  right? They have a census data available on every supervisor. All they need to do is just  look up at that census and figure out how many other nodes are there, which I need to  configure to. And this thing was moved from the cookbooks to the plants. So it's, you  only need to worry about while writing a new service. You need not to worry about when  you go and deploy it on a machine. So Habitat has some helper methods, which will let you  figure out how many alive members are there for a particular binding, which you want to  do for your, a particular service or an application. And then you can configure those things easily  in your plants. Finding the leader. So finding a master because master is the only person  who can write my data for some software like databases and all that is, right? We at predicts  wanted to provide a single way to connect to your service, whether it's a dedicated  instance or it's a clustered instance. We would provide you a single connection string  URL. And then no matter if your master went down for some reason, your string, the connection  string won't change. And behind the scene, we would make sure that, you know, that's  connection string is always pointing to the master, right? So how do we do that? Habitat  really helped us over there because it has another helper method, which always tells  you about a leader. So let me tell you a little bit about leader in Habitat, right? So when  you say I have to start a particular service and it is in a leader follower topology, it  needs at least three nodes to be there of that particular service in your ring, in your  cluster of Habitat supervisors. And as soon as three nodes become available and saying  that, you know, I am publishing elastic search, right? The Habitat will start an election  and a leader gets elected. So that leader will act as a master for your services. So  let's see a demo here. So we have a small application running and we have a simple question  to ask that, you know, the Habitat tell us who my leader is right now. So it's telling  me a particular IP of the leader from the super habitat ring. And what I'm going to  do is that I'll go to this machine and I would stop the supervisor. And as soon as I stop  the supervisor, I would see on other nodes in my ring that they would trigger another  election to find a new leader because the current leader went down. And if I'm able to  see the election nodes logs here. Okay, so this went down the hooks for recompile because  one service supervisor is down that sends the message to other nodes. And now let's  go back to the UI and try to refresh it. Okay, let me see. Oh, so I'm still recompiling.  There are three nodes in the cluster. So that's cool.  Okay.  Okay, I think go to the UI and like,  like you get the idea like what,  so if you kill a member in the ring,  the leader election happens again,  and the habitat supervisor help you elect another leader,  which you can use for use cases like Redis,  specifically where we were using.  You can go to the...  Okay, so...  It worked.  Perfect.  Okay, so it took some time.  Maybe, you know, the file which I'm trying to use here.  So this file, it's a basic HTML file.  It's just using that first method,  which is always pointing to the leader.  And as soon as the new election happens,  that method gets updated, like that's returning a new value,  and here we see it's a new leader available, right?  Okay, so...  I had one image for this.  It's like there was one leader,  and then suddenly it stepped down,  and there was a new leader, so...  Who will tell me this?  Don't worry, there is a habitat to tell us about it.  Okay, so next thing we gain from habitat is intelligent runtime, right?  Every service which we install,  what we do,  we write a configuration file for it  to start the service as we want, right?  So, habitat, again,  we can provide the configuration files,  or some kind of configuration,  which is called as spec files for Supervisor,  to tell Supervisor that what kind of service,  and in which fashion, like what topology,  and what, like, update strategy I have to use  with a particular service,  and in any services you want to run, right?  So all these configurations for the services,  you can provide to the Supervisor,  and then just leave everything up to the Supervisor,  because then Supervisor will make sure  that based on the given configuration,  it runs each and every service on that particular instance.  So there's a default directory,  haveSoup default specs,  which Supervisor always listens to,  or keep a watch on it,  and as soon as you drop a new file  with some specifications to run a particular service,  it will start running that service, right?  And, you know, there might be a use case  that I'm running a couple of services on a machine,  then suddenly I might need to move that service  to a different machine,  or maybe I might to start a new service  on a particular machine, right?  So it's very easy.  The Habitat makes this thing for us very easy,  because all we need to do is drop a file,  and then it will bring up a new service, right?  We can see this in a demo,  and this will work.  So in this particular machine,  we have configured an Nginx,  which was supposed to listen to a Kibana as an upstream server,  and we are seeing a bad gateway,  so let's go to the machine and see what happened here.  So if you see the directory, specs directory,  we see that Elasticsearch is there,  Logstress, Nginx, and a couple of other services are there,  but we don't see a Kibana, right?  So I need to install Kibana on this machine.  How easy it would be if I had to do with a Chef cookbook, right?  I had to write a wrapper cookbook,  or maybe use a community cookbook,  include that recipe into my existing recipe,  push a new version, and then start a Chef client.  That would start my service.  It will take some time, right?  But what I'm going to do here is,  I would write a Kibana spec file.  I can give you a little bit idea about a spec file,  what it contains.  The first thing it contains is,  you need to tell supervisor what service it has to run.  Then there is a group configuration  where you can identify or segregate your services  in different groups.  You can run multiple Elasticsearch in the same supervisor ring,  but they can work as a different cluster.  You can define the topology if you want to run them  as a standalone applications,  or you want them to be clustered with each other.  Update strategy, bindings.  Do they need to bind with someone else?  Any other application, database, or service?  And a few other configurations.  Disad's state is like, I want it to be up all the time,  and start style is persistent.  That tells the supervisor that when you stop,  don't delete this file,  because when I will start you again,  I want these services to get started again.  So all I have to do right now is that  pick this file and drop here.  And now if I go and check the log,  I can see a log coming from Kibana.  Hooks got recompiled.  It's trying to initialize the service.  And finally, a process is getting started,  and then server is running at a 5.1.6.0 port.  If I go back to my UI, I'm seeing a Kibana starting.  I couldn't have done this within the same time  if I had to write a new cookbook.  But in this case, just six, seven lines in a file,  and drop a file on a machine. That's it.  Let me see how we are doing before Rick throw me out of the room.  So last, but not the least,  one of the great benefits which we had in our  existing runtime solution is rolling upgrades.  So let's say we are running Chef Client as a service  on each machine in our cluster.  I have to change something or there is a new package  or any change in the configuration for a particular service  or let's consider the elastic search service.  I will write a new cookbook, push it to the Chef server,  and then next time when the Chef Client will run,  it will download the update and restart my service accordingly.  It is very likely that all three nodes of my cluster  can start Chef Client at the same time.  And if my service gets restarted at the same time,  I'll have a downtime.  And nobody will be happy about that.  So ideally, what I should do is  I have to figure out a way that I can send my updates one by one.  And Habitat comes in very handy doing that.  As soon as you write a new package and you push it to the depot,  Habitat will start.  So apart from the leader election, which is your master of the cluster,  Habitat or Supervisor does another type of election,  which is update leader.  The leader which is designated to look for a new package on the depot  and as soon as a new package is available, download it, install it,  and then tell others that this is a new package, it's good,  I was able to restart properly.  You also do the same.  So we have a video to save the time.  In this video,  what we did that we pushed a new version,  and then you can see the node 3,  because the node 3 happened to be the update leader, started updating,  and you can see the heap size is getting changed.  Right now it's like 50.0 GB.  Now it told its followers,  so node 1 is trying to update,  and then the heap size changed to 9 GB,  and then the last remaining node will start updating itself,  and the heap size will change,  because in the change we pushed,  we are going to decrease the heap size,  so there was the config change,  and how it happened?  It happened in a rolling fashion, one node at a time,  and service was all the time up and running.  I think at this time we can take some questions  if you guys have any questions.  Yeah, that's all.  That's how we boosted our Chef ecosystem with Habitat.  Thank you.  I'd like to ask about the election logic of the topology,  the leader and the follower.  Is it necessary to keep the number of the cluster to the older number?  From my MongoDB experience,  the pre-cathet node number should be kept in the older number,  because of the little majority in terms of election.  No, so all Habitat requires at least three nodes.  After that it could be any number.  At least three nodes, and I can add as I like.  So you said you were able to eliminate the Chef Client step  of having to run Chef Client as a service.  How are you managing the Hab binary?  Are you building golden images?  No, the Chef Client is the only responsibility  to start the Hab process, and that's it.  After that, it's all Hab.  You can do it with the images if you really want to eliminate it,  but we are not eliminating,  because somebody needs to make sure the Hab process is running all the time,  and that Chef Client is responsible for.  So this is VM runtime, and we do use Chef to manage your VM runtime.  You saw in the keynote in the morning that if you are using Kubernetes,  you can make sure using the images file, right,  and using Kubernetes, you can pass your configuration.  So in this scenario, we are using Chef to push Habitat  and make sure that Habitat is there,  and after that services, Habitat is responsible for them.  And Chef is doing another job, basically.  Chef is also placing the Habitat spec files with the templates.  So there's a cookbook which puts the Habitat binary.  It starts the Habitat and plays the files into the spec directory  for the desired services on that machine.  There are three things Chef is still doing,  so we still have Chef in the mix.  I was curious, along with that earlier question,  about how you actually do the updates to the Hab binaries.  Maybe there's a new version that comes out.  How you're actually doing the updates to the Hab binary themselves.  So there is a cookbook.  So there's a cookbook which puts the Hab binary on it.  So that cookbook, the only job in the earlier question  is that cookbook takes care of.  So you need to change the version of the cookbook  if you want to roll out the new binary of Habitat.  So the question is, how do you manage the DNS name  if the load member from one leader to another leader  gets elected, right?  So we have our own service written to publish a domain route  to an app-based routing.  So we use the HTTP header.  So we publish the route, and how do we decide from where,  from which machine we need to publish the route,  is based on the leader elections of Habitat.  So that service runs in a topology leader follower model,  and the plan is written in such a way that it will get executed  only on a leader machine.  So as soon as the leader is switched,  the hooks get recompiled, the service gets restarted,  and it's only running on the new leader.  So you are publishing the route only from the machine,  which is the leader, and how do you decide that using Habitat?  And how do you publish the route?  We have our custom software which uses the HTTP host name  and the app-based routing.  We will be at the Habitat corner if you have any other questions  about our system.  We, like, if you want to see another demo or see some action,  services in action with the Habitat,  because we run, like, lots of services on our platform.  So we would be available to the corner.  Yeah, one last thing is that if you like what we are doing at GE,  we are always hiring.  You guys are very welcome.  Thank you.
How_Habitat_can_Boost_Your_Chef_Ecosystem_0_131.png How_Habitat_can_Boost_Your_Chef_Ecosystem_132_1331.png How_Habitat_can_Boost_Your_Chef_Ecosystem_1332_3935.png How_Habitat_can_Boost_Your_Chef_Ecosystem_3936_5855.png How_Habitat_can_Boost_Your_Chef_Ecosystem_5856_7727.png How_Habitat_can_Boost_Your_Chef_Ecosystem_7728_13655.png How_Habitat_can_Boost_Your_Chef_Ecosystem_13656_18539.png How_Habitat_can_Boost_Your_Chef_Ecosystem_18540_18899.png How_Habitat_can_Boost_Your_Chef_Ecosystem_18900_29171.png How_Habitat_can_Boost_Your_Chef_Ecosystem_29172_29411.png How_Habitat_can_Boost_Your_Chef_Ecosystem_29412_29495.png How_Habitat_can_Boost_Your_Chef_Ecosystem_29496_29675.png How_Habitat_can_Boost_Your_Chef_Ecosystem_29676_36815.png How_Habitat_can_Boost_Your_Chef_Ecosystem_36816_39035.png How_Habitat_can_Boost_Your_Chef_Ecosystem_39036_39683.png How_Habitat_can_Boost_Your_Chef_Ecosystem_39684_39851.png How_Habitat_can_Boost_Your_Chef_Ecosystem_39852_40055.png How_Habitat_can_Boost_Your_Chef_Ecosystem_40056_40499.png How_Habitat_can_Boost_Your_Chef_Ecosystem_40500_40667.png How_Habitat_can_Boost_Your_Chef_Ecosystem_40668_40955.png How_Habitat_can_Boost_Your_Chef_Ecosystem_40956_41039.png How_Habitat_can_Boost_Your_Chef_Ecosystem_41040_41195.png How_Habitat_can_Boost_Your_Chef_Ecosystem_41196_41411.png How_Habitat_can_Boost_Your_Chef_Ecosystem_41412_41555.png How_Habitat_can_Boost_Your_Chef_Ecosystem_41556_41723.png How_Habitat_can_Boost_Your_Chef_Ecosystem_41724_42251.png How_Habitat_can_Boost_Your_Chef_Ecosystem_42252_42335.png How_Habitat_can_Boost_Your_Chef_Ecosystem_42336_42479.png How_Habitat_can_Boost_Your_Chef_Ecosystem_42480_42899.png How_Habitat_can_Boost_Your_Chef_Ecosystem_42900_43427.png How_Habitat_can_Boost_Your_Chef_Ecosystem_43428_44363.png How_Habitat_can_Boost_Your_Chef_Ecosystem_44364_45167.png How_Habitat_can_Boost_Your_Chef_Ecosystem_45168_45623.png How_Habitat_can_Boost_Your_Chef_Ecosystem_45624_45731.png How_Habitat_can_Boost_Your_Chef_Ecosystem_45732_46343.png How_Habitat_can_Boost_Your_Chef_Ecosystem_46344_47183.png How_Habitat_can_Boost_Your_Chef_Ecosystem_47184_48467.png How_Habitat_can_Boost_Your_Chef_Ecosystem_48468_48539.png How_Habitat_can_Boost_Your_Chef_Ecosystem_48540_48587.png How_Habitat_can_Boost_Your_Chef_Ecosystem_48588_48635.png How_Habitat_can_Boost_Your_Chef_Ecosystem_48636_48887.png How_Habitat_can_Boost_Your_Chef_Ecosystem_48888_51587.png How_Habitat_can_Boost_Your_Chef_Ecosystem_51588_51647.png How_Habitat_can_Boost_Your_Chef_Ecosystem_51648_51719.png How_Habitat_can_Boost_Your_Chef_Ecosystem_51720_51851.png How_Habitat_can_Boost_Your_Chef_Ecosystem_51852_52031.png How_Habitat_can_Boost_Your_Chef_Ecosystem_52032_52115.png How_Habitat_can_Boost_Your_Chef_Ecosystem_52116_55511.png How_Habitat_can_Boost_Your_Chef_Ecosystem_55512_57131.png How_Habitat_can_Boost_Your_Chef_Ecosystem_57132_57599.png How_Habitat_can_Boost_Your_Chef_Ecosystem_57600_63676.png