Chef Automate: Visibility Feature, Q&A Panel - July 13, 2016
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Duffield, Tom
Hartmann, Christoph
Dennard, Mark
Drew, Tasha
Learn about the visibility feature of Chef Automate. Gain insight into operational, compliance, and workflow events. There is a query language available through the user interface and customizable dashboards. Insight into your network and development processes has never been easier.
 Okay, we'll get started. Welcome everyone. This is a Q&A session focused on Chef Automate focusing on the visibility side of the product and then the next session at three will be on the workflow side. So I'd like to introduce you to some of the team. Starting with Tasha and I think she'll introduce the others. Thanks. Awesome. Hi, I'm Tasha Drew. I'm the product manager for our visibility and analytics platform. Hello? Okay. Hi, I'm Tasha Drew and I'm the product manager for our visibility and analytics platform. As you all know, we launched that yesterday as part of Chef Automate. And yeah, this is Tom. Hi, I'm Tom. I was the lead engineer on the visibility side of the Automate product and this is... Hey, I'm Chris. I'm mostly responsible for the compliance part in here. So back to Tasha. And over here we have Mark who is head of UX. Hi. So basically... So basically today we are briefly going to go over the product just because I don't know if many of you have seen much of it. So we're going to quickly show you the product, some mocks, some stuff we're building right now that we anticipate having included in the product within the next month which we think is cool, which is our full compliance data integration. And then we are totally open for questions or whatever you all are interested in talking more about, probably keep it pretty succinct. Yeah, I don't know what to say. So I'm just going to yell. So what I'm going to show you here is actually a series of mocks that emulate the actual experience of the product. But what this has in it that the product today doesn't is where we're going, let's say, in the next month or so, which specifically is really around compliance. So if you saw the demo yesterday, then you know that essentially what this is is a... What the screens that we're going to be looking at are screens that allow you to see sort of the state of your fleet and your nodes, whether things are passing or failing or missing or being skipped. And just the first thing I think I'd point out is that the way that data is getting into here is right now you're allowed to sort of look by all Chef servers or select individual Chef servers and that will globally filter what you're seeing, the same as true with orgs as well. And if you come down here, you'll see this list, which is showing your nodes and their status in convergence. This radial is really just showing you the proportion of success and failure. You can filter based upon these nav elements here. So essentially if you wanted to see all of your failed nodes, you'd click here and you'd be able to scroll through this list and see them. There's also this trend, which displays a few things. So one, what you're seeing is sort of hour over hour for the past 12 hours, what's converging and failing, but you're also seeing based upon this right here where cookbooks were uploaded or changes were delivered through workflow. And that's pretty much what we got to show here. You can also sort of filter by environment and role, but I think probably the most interesting features are up here in this area right here because we've sort of developed a query language to allow you to put together more complicated queries based upon, let's see, what do we have in there? Resources, attributes, node names, recipes, cookbooks, that's it for now, right? And the plan is that this would be open so we can be able to add to it so that queries can become more and more robust over time. So that's essentially what this is about up here. There's also, sorry, there's also a few other things. So the ability to go and execute this, save searches and also share them. So for instance, imagine you put together a query that then you filtered by environment or role and then only failed nodes and you wanted to share exactly that with a colleague. So what we're allowing you to do is sort of click up here, click this and essentially that link will be copied to your clipboard so you can share in whatever form that you would like, like for instance for an email or chat or whatever. And you could also save a search and be able to access it later. That's what's going on here. Now the other part of this is compliance. So this is a similar dashboard to what you saw before, but this is in the context, your nodes in the context of their compliance state. So again, you can sort of see here all my own compliance nodes and here's sort of the scoring. So for instance, you have a series of checks that are critical. You can sort on these column matters, et cetera. And you can also again view when compliance profiles were uploaded or changes were delivered and then click to view details. So that's what this is. And essentially this is showing you a run history over here that you can filter on and you can click through to see the individual runs and see what do you call it? Like yeah, exactly. And view the full logs and the backtrace. And you can filter on these resources to see what the problem was and see the error for that. And that's what we have for now. We're moving towards building compliance into this as well. And that's where we're headed next. Cool. And just from an integration perspective, basically right now we can collect data from all of your Chef servers, Chef clients, and Chef solos. And we gather data from the workflow aspect. So that's how you're seeing when changes are delivered through workflow. We do plan to expand on those visualizations in the next quarter or two. And we also are integrated with InSpec. And so the data integrations are in the back end and we're working on delivering the front end visualizations that Mark just showed. And another aspect that we have is if you already have a dashboarding solution that you're really happy with and you just want all of the data that we're gathering, you can export directly to that using a very simple thing that we're calling a data tap, which is a queue of events that if you grab, you can export to another system. Do you guys want to go over anything? Okay. Cool. Does anyone have any questions off the top? Yes. Green shirt. So the client before actually brings you a PDF or some form of executive report for where you could say, hey, here's the physical proof that it has to follow. That's an excellent question. I think we had that before in Chef Compliance as well. And that's especially what we're trying to achieve here. So visibility is collecting the reports and storing it also about history. So Chef Compliance is already good in showing you the current state, but the history feature was not awesome. So this is like something that we're looking forward to bring it in. So in specific, this is also coming down to how do we do PDF reports? How do we share that? And that is something that we have to consider properly. And the challenge here is though that normally you need to adapt it to your corporate needs. So you need to have a template for that. So it sounds easy, but it's not. So and that's the challenge to make it right and to make it adaptable to your specific needs to make it customizable. And this is currently that we tried to find out how to do it properly. So reach out to our sales team so that we can find out how we can deliver that. Just quick show of hands for my own information. Who in here is currently using Chef Compliance? And then who is currently using Chef Server? Okay. Both? Oh, cool. And then anyone use Chef Solo slash zero? Yeah. Okay. Cool. Awesome. So, yeah, I mean basically what we have in production today is very focused on the Chef Server, Chef Solos, and Chef Client data integrations. And then the compliance piece we anticipate we'll have out probably in the beginning of August. But for now, basically what you do is set up Chef Automate and then very easy system level integration. And we start collecting data from all of your endpoints and then displaying them the way you see here. Yes. So does this connect these with Chef Server, the data from there, the Chef Solo, where do we point the data? So the data collection mechanism is on Automate there is a data collector endpoint and it takes data being sent to it from your other products. So for example, in the Chef Client there is a data collector, some data collector configuration that you can configure and you give it like a token and an endpoint and it will send the requisite data to that endpoint, the same with the Chef Server. So Automate isn't grabbing data from things, it's being sent data by things. So your Chef Server sending data to Automate. Correct, yes, so you can kind of do whatever networking magic you want to in the middle. It's just a restful API that all of the data is coming in on. So there is a 443 restful API on Automate that you send the data to and then it goes into the back end. Not yet. So we anticipate that we will be adding that very soon. With the attribute data, people love seeing all of their attributes in the system and I think they will enjoy it for about a week and realize they only need three of them and they like to save a lot on storage so we anticipate adding that functionality pretty soon. The saved searches are saved to the back end database so it's not cookie. So it's connected with your user account. Yes. As an export? Yes, yes, the data tab allows you to export the data to any third party dashboarding tools that you want to. Yeah. In the back. So that is in the roadmap and probably Q4 target. Yes. So I'm having trouble wrapping my head around what's happening because as an engineer I don't ever look at it really. I think my life is right. So I'm having, is this just not your to me or am I, I'm coming at this the wrong way. Can you help me? Do you want to talk to me? Well, okay, so this interface is primarily geared towards ops practitioners rather than engineers per se. So this interface is primarily geared towards ops practitioners rather than engineers per se. So for instance, a director of operations has a fleet of nodes that they have to monitor constantly and that person then has a team that has to fix the problems that are already arise with that. Well, so this is primarily around where are my fires and then what is the character of the fire for any series of individual nodes. So this may not speak to you. It's primarily not geared towards an application engineer. It could be both in that sense. It could be both, but in a lot of ways like it's geared somewhat more towards people who are managing the work. Is it currently just on prem or is it, are you guys going off of this in a hope to manage it? Yeah, it's currently on prem. We have made it, out of the gate we do allow you to use Elasticsearch as a service if you don't want to have Elasticsearch hosted in your data center. And we are looking at making it very cloud friendly as part of some of our key operational deliverables. It's not part of the hosted offering currently. And we are looking at how to make it more of a cloud first offering. So if you want it on prem, you can have it on prem. Everything's on prem. But then also having a hybrid model for people who want to use a lot of services and don't want to directly manage databases themselves. And then having a fully automated solution in the cloud for people who want that. My timeline for v0 is by the end of the year. Yeah. Yes. So as of right now, the access control is observer, everyone has observer privileges. There's really no right capabilities with this particular interface. In the future we've discussed, but I don't know the specifics on timeline, around being able to limit certain views or certain information to different roles. But as of right now, there's no role based access. Is it related to the words that on the number of, like C whatever, whichever word type of number I see, what's going on, how does it work, and where is it for the word on the number of, like C. You can see the sweep of all. Right now you can see all of the data. You can see all of the data that's available in the system, not just your, the organizations that you belong to. Right now. The authentication right now is handled through the, what was the delivery authentication, which is, but is now kind of just the automated authentication, which can sync up with SAML and LDAP and all of that stuff. But this interface doesn't have its own role based access or user accounts. It's the same, automate has its user base and those users get organized in whatever structure they're organized in at any given time. And right now you get to see all of it. Yeah. That is the default view right now. Yeah. But if you want, you can save it within using either searches or user preferences. If you don't want to see all of it, you can drill down and we'll save those preferences. If you wanted to limit somebody to seeing only their particular organization, you have to select the server. That, for now, yeah, until we have our back, that would be the only way to do it. I guess I'm kind of curious on understanding a little more about that use case too. So it sounds like there is some need for only people who are part of an organization should be able to see data from that organization on the Chef server. Okay. Cool. Yeah, that's good to know. We've been playing with the idea of RBAC, but because visibility is a read-only system, whereas the previous products had been write access, people were less worried about it than who we talked to. Yeah. Yeah, I can definitely see that sort of chipping away at the value of having everything in one place if you have role requirements that keep that from actually being useful without them. That's cool. Okay. In the back. We've talked about policy files, but direct integration with that isn't on the roadmap right now just because not enough people have told us we're using them for it to be at the top of our hit list for an integration. So if I have problems with systems that are sub-solo, sub-solo, how do I bring all of that into the data directory and not to worry about the agent based? No. So the solo agent itself knows how to send data to automate, like in the Chef client binary itself. So there's no additional agent that you need to install to send the data to automate. The Chef client, the executable, the binary just knows how to do it out of the box as of, I think, 12, 11 something. So we can just go discover things? It doesn't. You have to know the, there's like a config.d client, or sorry, like the client.rb, there is some, you have to point it to the automate endpoint for the data collector and give it a security token so that it's not just accepting all data from anyone. And then you just run it and it'll send the data to it. I'll just push that to all the client. Yep. So all of your nodes that are running Chef solo, you just either add the client.d sub-configuration or modify the client.rb to add the data collector information and it'll start sending that data. You wouldn't have to bootstrap them, not with the new client.d stuff. So you can, if you already have like 10,000 nodes, you would just need to run some script that just iterates through them and drops off a file in a certain location and it'll just work. That's how we did it internally. It's just a knife SSH with a whole bunch of stuff, but you can do an SSH loop of any kind. Right. So anytime the Chef client runs, it sends the data. So it's kind of like reporting. So reporting, it would send the data to reporting. And instead of sending the data to reporting, it's just sending it to visibility. So it runs when the Chef client runs. So it depends on how frequently you run it. Yeah. Yeah. So basically incorporate, are you thinking incorporate system level information into the workflow as you're making builds or where are you thinking? That is, that's right in the wheelhouse of workflow. Like that is workflows of bread and butter is catching those errors before they hit your important production systems. The automate server is so yes. As of right now, yes. We were trying to design it to be more service oriented such that if you wanted to, you could break them up. But it comes as an omnibus package with everything that you need. So you can just install it and go. And you will offer probably in the future more complex installation options. But yeah, right now it's everything is on the box and then you can go. Yes. No. Not that I know. So not yet. Cool. I think I saw some questions in the back. Yeah. So I'm going to answer, I'm going to provide an answer and you can let me know if it's the answer, if it answers your question. Right. The visibility side of automate can work independently of a chef server. So it can receive data from any number of chef servers. Automate itself requires a single chef server in order to operate. And that chef server is its own standalone instance. So as part of the automate installation, you need a chef server as of right now. But visibility can collect data from any number of chef servers. The automate installer. Yes. But you can use an existing chef server as part of that. It just needs that so that for the workflow side, it uses search and all that kind of stuff for the back end on the workflow side of things. Yeah. Yep. In the chef server RB configuration, you input the similar data collector configuration that you're doing for your chef clients. And the data that the chef server sends to analytics like all of your CRUD operations and all that kind of stuff then gets sent to visibility. There is, I don't know it off the top of my head. If you've been streaming the data to us, then yeah. I think the specifics of that, there are complexities that come with that amount of data and that's something that I know we are working on. So the goal is yes to provide time, be able to view what has happened over a period of time. So you just haven't quite nailed that down yet. Yeah. So the back end we're using makes it really easy to drop or export data if you don't want it on the local elastic search cluster. So basically with elastic search if you want to either ship it to Amazon Glacier or put it somewhere else when you're done with it because it's out of a time window, it's very simple to do that. Or you can just say clean it up. I don't want it anymore. Yeah. At a technical level each day we have elastic search indices for each day. So you can archive days of like elastic search days or clean them up. There's nothing built into it right now that will automatically delete it. So right now that would be an external step but it's really easy to do. Yeah I think we're going to definitely be adding a lot around data retention policies both at the what data do I really care about which we anticipate will be different for different customers and then also the how long do I care about it for. Yeah. And there's a lot of complexity in there because we have to support a lot of different retention policies and different types of retention policies. So we want to make sure we get that right so that we don't end up forcing you into a retention policy that you don't or can't follow. So that is has been put at the top of my list. Sorry the question was when when will Habitat data be in the system. And that is on the top of my list of things to achieve in Q3. So stay tuned. The data that you're storing in Alaska is exposed through an API that is the main value proposition or is it this UI that is the value proposition because if the API was the value proposition then what. Yeah. Yeah. And that's something we have. Yeah. That's something we've been discussing sort of what would the export method look like for people who want to either have regular reports sent to certain systems either by an API via email like whatever that would look like is something that we're investigating. We do want to be API first as far as just allowing people to easily integrate with our back end and build out whatever they want to build out. And so for the short term if you want a very specific report with certain data fields in it you can do that using Kibana where all of the events that we're capturing are there. You can organize sort them however you want to and create your own visualizations and lists but then we do plan to add that into the product to make it very easy to do. And I think I'm going to take a little bit further. The data, sorry, the value proposition is you'll always have unfettered access to the raw data that we collect. And then what we do is we have our own data processing pipeline where we process that raw data in ways that we anticipate are helpful for you and then we provide visualizations on that data so that you can really quickly and easily understand that. I'm trying to compromise with that processing pipeline. If you're going to make the API that I proposition to split that pipeline would push the data back into Elasticsearch? It does. Okay. Yep. Yeah. And if anybody wants more one-on-one conversation about the product we do have the product suite downstairs from four to five. If you want to kind of tell us about your use case and talk through it, it's room 407. Cool. That's such a wide range because if you're someone who runs like a maintenance run list once a day or if you're someone that runs full suites every 30 minutes, we haven't been able to do that level of capacity analysis yet. Internally we haven't really outgrown our installation yet, which is a moderately sized installation with like 12 or 20 nodes that we have, but we haven't had a chance to do that yet. I'm sure that we'll probably offer some sort of capacity planning because I've built systems like that before. That's a very important part of it. We just haven't had enough time to do that process yet. Cool. Yeah. In the back. How long do you think it's going to take for your 12-ton node to this place? I think we're on an M2 something XL on Amazon with no way, sorry. That's a completely different one. I have no idea. It's actually relatively small. Gigabytes. We are still sub-gigabytes. Yeah. I've been hearing the other. This is the replacement for Analysts. There's one way to put it, but this is obviously a much different product. Yeah. Analytics doesn't have RBAC. So basically this is the next generation of our approach to analytics. As we look at this, how is it significantly the next step for us? The first piece of that is that it allows us to get all of your data in one place, whereas analytics and manage are both one Chef server to one installation of the analytics platform. That's a pretty big deal because you don't want to have segregated data all over the place. So you can get all of your data in one place where you can see all of your data and be able to programmatically understand what is the health of my infrastructure as code. Next, we have basically added significantly more of richer data from the sources that we're collecting it from. Yeah. So instead of old looking, my question is more backwards looking. So yeah, we've gotten analytics from now, we've worked through play, and we've got to see what is the future. So we're going to have to use our education. We don't have this anymore, but it's fine. I mean, the main feature that we currently don't have that analytics has is rules and notifications. And so we have the data tab, which is a fire hose of data, but then actually sorting that out and sending it to a specific endpoint is not there yet. So that is what we are trying to figure out exactly what people want out of that tool. We weren't really happy in retrospect with the rules and notifications engine of analytics. We think it was too difficult to use. And so a lot of people weren't using it to its full potential, and it was really easy to get way too many notifications to the point that they weren't maybe useful anymore. So we are looking at exactly how we want to do rules and notifications and what that looks like. And that kind of ties into an earlier question someone was asking, which was, how do I get the data from here to people who need a report, for example? And then what does that look like? And that's something that we're really looking into. I think that was the API question. But basically, is it an API? Can I just take this data programmatically every now and then and shoot it to another system? And that's one thing we're looking at. We're looking at perhaps a scheduled email report of your own choosing on a regular basis. But we're just sort of playing with different ideas and presenting them to customers and then seeing what will fit that use case in the best possible way, but also in a significantly more user-friendly way. And then the other big thing we have around rules and notifications is the problem with analytics was that everyone could edit the rules and notifications if they had access to analytics. And that means that you might want your developers to be able to write a rule and notification just for the servers they care about, but you don't want them to be able to delete your operations core rules and notifications that you super care about. So those are all use cases that we have in mind as we figure out exactly what we're going to deliver. Yeah. Cool. In the back. Yeah, Tasha at chef.io. Yeah. Yeah. I mean, I am very open to being on customer calls. Brooke and I think that a significant part of our job and Christoph as well is talking to customers, getting feedback before we build stuff and while we're building stuff just to make sure that we're really hitting the nail on the head when we deliver it. Yeah. Yeah. Exactly. In the back. Is there someone? Okay. Stretching. Yeah. Yeah. So delivery. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So delivery to work for it looks like a team of customers. It's like putting in the same back and forth. This looks like a totally different. How do you do about getting the customer in the back end? Yeah. We current. So while elastic search is amazingly elastic and we anticipate that you could do a migration from the back end of analytics into this, we have not productized that right now. Yeah. So we want you to say that you can use the search. No. Cool. Any other questions? Well, as you may know from Barry's presentation earlier, you can go and download a free trial of this and give it a shot on our website, chef.io. And we also have some tutorials that we'll take you through using this if you want to check it out. But it's a new product. We are eagerly awaiting feedback and just kind of seeing where it goes and how people want to use it. So the more communication, the better. We're really interested in what you think. And we will be delivering a lot of new awesome features pretty soon. So yeah, give it a shot. Let us know what you think. Come talk to us in 407. Mark said come talk to us in 407. Cool. Thank you. Thank you. Thank you.
 Okay, we'll get started.  Welcome everyone.  This is a Q&A session focused on Chef Automate focusing on the visibility side of the product  and then the next session at three will be on the workflow side.  So I'd like to introduce you to some of the team.  Starting with Tasha and I think she'll introduce the others.  Thanks.  Awesome.  Hi, I'm Tasha Drew.  I'm the product manager for our visibility and analytics platform.  Hello?  Okay.  Hi, I'm Tasha Drew and I'm the product manager for our visibility and analytics platform.  As you all know, we launched that yesterday as part of Chef Automate.  And yeah, this is Tom.  Hi, I'm Tom.  I was the lead engineer on the visibility side of the Automate product and this is...  Hey, I'm Chris.  I'm mostly responsible for the compliance part in here.  So back to Tasha.  And over here we have Mark who is head of UX.  Hi.  So basically...  So basically today we are briefly going to go over the product just because I don't know  if many of you have seen much of it.  So we're going to quickly show you the product, some mocks, some stuff we're building right  now that we anticipate having included in the product within the next month which we  think is cool, which is our full compliance data integration.  And then we are totally open for questions or whatever you all are interested in talking  more about, probably keep it pretty succinct.  Yeah, I don't know what to say.  So I'm just going to yell.  So what I'm going to show you here is actually a series of mocks that emulate the actual  experience of the product.  But what this has in it that the product today doesn't is where we're going, let's say, in  the next month or so, which specifically is really around compliance.  So if you saw the demo yesterday, then you know that essentially what this is is a...  What the screens that we're going to be looking at are screens that allow you to see sort  of the state of your fleet and your nodes, whether things are passing or failing or missing  or being skipped.  And just the first thing I think I'd point out is that the way that data is getting into  here is right now you're allowed to sort of look by all Chef servers or select individual  Chef servers and that will globally filter what you're seeing, the same as true with  orgs as well.  And if you come down here, you'll see this list, which is showing your nodes and their  status in convergence.  This radial is really just showing you the proportion of success and failure.  You can filter based upon these nav elements here.  So essentially if you wanted to see all of your failed nodes, you'd click here and you'd  be able to scroll through this list and see them.  There's also this trend, which displays a few things.  So one, what you're seeing is sort of hour over hour for the past 12 hours, what's converging  and failing, but you're also seeing based upon this right here where cookbooks were  uploaded or changes were delivered through workflow.  And that's pretty much what we got to show here.  You can also sort of filter by environment and role, but I think probably the most interesting  features are up here in this area right here because we've sort of developed a query language  to allow you to put together more complicated queries based upon, let's see, what do we  have in there?  Resources, attributes, node names, recipes, cookbooks, that's it for now, right?  And the plan is that this would be open so we can be able to add to it so that queries  can become more and more robust over time.  So that's essentially what this is about up here.  There's also, sorry, there's also a few other things.  So the ability to go and execute this, save searches and also share them.  So for instance, imagine you put together a query that then you filtered by environment  or role and then only failed nodes and you wanted to share exactly that with a colleague.  So what we're allowing you to do is sort of click up here, click this and essentially  that link will be copied to your clipboard so you can share in whatever form that you  would like, like for instance for an email or chat or whatever.  And you could also save a search and be able to access it later.  That's what's going on here.  Now the other part of this is compliance.  So this is a similar dashboard to what you saw before, but this is in the context, your  nodes in the context of their compliance state.  So again, you can sort of see here all my own compliance nodes and here's sort of the  scoring.  So for instance, you have a series of checks that are critical.  You can sort on these column matters, et cetera.  And you can also again view when compliance profiles were uploaded or changes were delivered  and then click to view details.  So that's what this is.  And essentially this is showing you a run history over here that you can filter on and  you can click through to see the individual runs and see what do you call it?  Like yeah, exactly.  And view the full logs and the backtrace.  And you can filter on these resources to see what the problem was and see the error for  that.  And that's what we have for now.  We're moving towards building compliance into this as well.  And that's where we're headed next.  Cool.  And just from an integration perspective, basically right now we can collect data from  all of your Chef servers, Chef clients, and Chef solos.  And we gather data from the workflow aspect.  So that's how you're seeing when changes are delivered through workflow.  We do plan to expand on those visualizations in the next quarter or two.  And we also are integrated with InSpec.  And so the data integrations are in the back end and we're working on delivering the front  end visualizations that Mark just showed.  And another aspect that we have is if you already have a dashboarding solution that  you're really happy with and you just want all of the data that we're gathering, you  can export directly to that using a very simple thing that we're calling a data tap, which  is a queue of events that if you grab, you can export to another system.  Do you guys want to go over anything?  Okay.  Cool.  Does anyone have any questions off the top?  Yes.  Green shirt.  So the client before actually brings you a PDF or some form of executive report for  where you could say, hey, here's the physical proof that it has to follow.  That's an excellent question.  I think we had that before in Chef Compliance as well.  And that's especially what we're trying to achieve here.  So visibility is collecting the reports and storing it also about history.  So Chef Compliance is already good in showing you the current state, but the history feature  was not awesome.  So this is like something that we're looking forward to bring it in.  So in specific, this is also coming down to how do we do PDF reports?  How do we share that?  And that is something that we have to consider properly.  And the challenge here is though that normally you need to adapt it to your corporate needs.  So you need to have a template for that.  So it sounds easy, but it's not.  So and that's the challenge to make it right and to make it adaptable to your specific  needs to make it customizable.  And this is currently that we tried to find out how to do it properly.  So reach out to our sales team so that we can find out how we can deliver that.  Just quick show of hands for my own information.  Who in here is currently using Chef Compliance?  And then who is currently using Chef Server?  Okay.  Both?  Oh, cool.  And then anyone use Chef Solo slash zero?  Yeah.  Okay.  Cool.  Awesome.  So, yeah, I mean basically what we have in production today is very focused on the Chef  Server, Chef Solos, and Chef Client data integrations.  And then the compliance piece we anticipate we'll have out probably in the beginning of  August.  But for now, basically what you do is set up Chef Automate and then very easy system  level integration.  And we start collecting data from all of your endpoints and then displaying them the way  you see here.  Yes.  So does this connect these with Chef Server, the data from there, the Chef Solo, where  do we point the data?  So the data collection mechanism is on Automate there is a data collector endpoint and it  takes data being sent to it from your other products.  So for example, in the Chef Client there is a data collector, some data collector configuration  that you can configure and you give it like a token and an endpoint and it will send  the requisite data to that endpoint, the same with the Chef Server.  So Automate isn't grabbing data from things, it's being sent data by things.  So your Chef Server sending data to Automate.  Correct, yes, so you can kind of do whatever networking magic you want to in the middle.  It's just a restful API that all of the data is coming in on.  So there is a 443 restful API on Automate that you send the data to and then it goes  into the back end.  Not yet.  So we anticipate that we will be adding that very soon.  With the attribute data, people love seeing all of their attributes in the system and  I think they will enjoy it for about a week and realize they only need three of them and  they like to save a lot on storage so we anticipate adding that functionality pretty soon.  The saved searches are saved to the back end database so it's not cookie.  So it's connected with your user account.  Yes.  As an export?  Yes, yes, the data tab allows you to export the data to any third party dashboarding tools  that you want to.  Yeah.  In the back.  So that is in the roadmap and probably Q4 target.  Yes.  So I'm having trouble wrapping my head around what's happening because as an engineer I don't  ever look at it really.  I think my life is right.  So I'm having, is this just not your to me or am I, I'm coming at this the wrong way.  Can you help me?  Do you want to talk to me?  Well, okay, so this interface is primarily geared towards ops practitioners rather than  engineers per se.  So this interface is primarily geared towards ops practitioners rather than engineers per  se.  So for instance, a director of operations has a fleet of nodes that they have to monitor  constantly and that person then has a team that has to fix the problems that are already  arise with that.  Well, so this is primarily around where are my fires and then what is the character of  the fire for any series of individual nodes.  So this may not speak to you.  It's primarily not geared towards an application engineer.  It could be both in that sense.  It could be both, but in a lot of ways like it's geared somewhat more towards people  who are managing the work.  Is it currently just on prem or is it, are you guys going off of this in a hope to manage  it?  Yeah, it's currently on prem.  We have made it, out of the gate we do allow you to use Elasticsearch as a service if you  don't want to have Elasticsearch hosted in your data center.  And we are looking at making it very cloud friendly as part of some of our key operational  deliverables.  It's not part of the hosted offering currently.  And we are looking at how to make it more of a cloud first offering.  So if you want it on prem, you can have it on prem.  Everything's on prem.  But then also having a hybrid model for people who want to use a lot of services and don't  want to directly manage databases themselves.  And then having a fully automated solution in the cloud for people who want that.  My timeline for v0 is by the end of the year.  Yeah.  Yes.  So as of right now, the access control is observer, everyone has observer privileges.  There's really no right capabilities with this particular interface.  In the future we've discussed, but I don't know the specifics on timeline, around being  able to limit certain views or certain information to different roles.  But as of right now, there's no role based access.  Is it related to the words that on the number of, like C whatever, whichever word type of  number I see, what's going on, how does it work, and where is it for the word on the  number of, like C.  You can see the sweep of all.  Right now you can see all of the data.  You can see all of the data that's available in the system, not just your, the organizations  that you belong to.  Right now.  The authentication right now is handled through the, what was the delivery authentication,  which is, but is now kind of just the automated authentication, which can sync up with SAML  and LDAP and all of that stuff.  But this interface doesn't have its own role based access or user accounts.  It's the same, automate has its user base and those users get organized in whatever  structure they're organized in at any given time.  And right now you get to see all of it.  Yeah.  That is the default view right now.  Yeah.  But if you want, you can save it within using either searches or user preferences.  If you don't want to see all of it, you can drill down and we'll save those preferences.  If you wanted to limit somebody to seeing only their particular organization, you have  to select the server.  That, for now, yeah, until we have our back, that would be the only way to do it.  I guess I'm kind of curious on understanding a little more about that use case too.  So it sounds like there is some need for only people who are part of an organization should  be able to see data from that organization on the Chef server.  Okay.  Cool.  Yeah, that's good to know.  We've been playing with the idea of RBAC, but because visibility is a read-only system,  whereas the previous products had been write access, people were less worried about it  than who we talked to.  Yeah.  Yeah, I can definitely see that sort of chipping away at the value of having everything in  one place if you have role requirements that keep that from actually being useful without  them.  That's cool.  Okay.  In the back.  We've talked about policy files, but direct integration with that isn't on the roadmap  right now just because not enough people have told us we're using them for it to be at the  top of our hit list for an integration.  So if I have problems with systems that are sub-solo, sub-solo, how do I bring all of  that into the data directory and not to worry about the agent based?  No.  So the solo agent itself knows how to send data to automate, like in the Chef client binary  itself.  So there's no additional agent that you need to install to send the data to automate.  The Chef client, the executable, the binary just knows how to do it out of the box as  of, I think, 12, 11 something.  So we can just go discover things?  It doesn't.  You have to know the, there's like a config.d client, or sorry, like the client.rb, there  is some, you have to point it to the automate endpoint for the data collector and give it  a security token so that it's not just accepting all data from anyone.  And then you just run it and it'll send the data to it.  I'll just push that to all the client.  Yep.  So all of your nodes that are running Chef solo, you just either add the client.d sub-configuration  or modify the client.rb to add the data collector information and it'll start sending that data.  You wouldn't have to bootstrap them, not with the new client.d stuff.  So you can, if you already have like 10,000 nodes, you would just need to run some script  that just iterates through them and drops off a file in a certain location and it'll  just work.  That's how we did it internally.  It's just a knife SSH with a whole bunch of stuff, but you can do an SSH loop of any  kind.  Right.  So anytime the Chef client runs, it sends the data.  So it's kind of like reporting.  So reporting, it would send the data to reporting.  And instead of sending the data to reporting, it's just sending it to visibility.  So it runs when the Chef client runs.  So it depends on how frequently you run it.  Yeah.  Yeah.  So basically incorporate, are you thinking incorporate  system level information into the workflow as you're making builds or where are you thinking?  That is, that's right in the wheelhouse of workflow.  Like that is workflows of bread and butter is catching those errors before they hit your  important production systems.  The automate server is so yes.  As of right now, yes.  We were trying to design it to be more service oriented such that if you wanted to, you could  break them up.  But it comes as an omnibus package with everything that you need.  So you can just install it and go.  And you will offer probably in the future more complex installation options.  But yeah, right now it's everything is on the box and then you can go.  Yes.  No.  Not that I know.  So not yet.  Cool.  I think I saw some questions in the back.  Yeah.  So I'm going to answer, I'm going to provide an answer and you can let me know if it's  the answer, if it answers your question.  Right.  The visibility side of automate can work independently of a chef server.  So it can receive data from any number of chef servers.  Automate itself requires a single chef server in order to operate.  And that chef server is its own standalone instance.  So as part of the automate installation, you need a chef server as of right now.  But visibility can collect data from any number of chef servers.  The automate installer.  Yes.  But you can use an existing chef server as part of that.  It just needs that so that for the workflow side, it uses search and all that kind of  stuff for the back end on the workflow side of things.  Yeah.  Yep.  In the chef server RB configuration, you input the similar data collector configuration  that you're doing for your chef clients.  And the data that the chef server sends to analytics like all of your CRUD operations  and all that kind of stuff then gets sent to visibility.  There is, I don't know it off the top of my head.  If you've been streaming the data to us, then yeah.  I think the specifics of that, there are complexities that come with that amount of  data and that's something that I know we are working on.  So the goal is yes to provide time, be able to view what has happened over a period of  time.  So you just haven't quite nailed that down yet.  Yeah.  So the back end we're using makes it really easy to drop or export data if you don't want  it on the local elastic search cluster.  So basically with elastic search if you want to either ship it to Amazon Glacier or put  it somewhere else when you're done with it because it's out of a time window, it's very  simple to do that.  Or you can just say clean it up.  I don't want it anymore.  Yeah.  At a technical level each day we have elastic search indices for each day.  So you can archive days of like elastic search days or clean them up.  There's nothing built into it right now that will automatically delete it.  So right now that would be an external step but it's really easy to do.  Yeah I think we're going to definitely be adding a lot around data retention policies  both at the what data do I really care about which we anticipate will be different for  different customers and then also the how long do I care about it for.  Yeah.  And there's a lot of complexity in there because we have to support a lot of different  retention policies and different types of retention policies.  So we want to make sure we get that right so that we don't end up forcing you into a  retention policy that you don't or can't follow.  So that is has been put at the top of my list.  Sorry the question was when when will Habitat data be in the system.  And that is on the top of my list of things to achieve in Q3.  So stay tuned.  The data that you're storing in Alaska is exposed through an API that is the main value  proposition or is it this UI that is the value proposition because if the API was the value  proposition then what.  Yeah.  Yeah.  And that's something we have.  Yeah.  That's something we've been discussing sort of what would the export method look like  for people who want to either have regular reports sent to certain systems either by  an API via email like whatever that would look like is something that we're investigating.  We do want to be API first as far as just allowing people to easily integrate with our  back end and build out whatever they want to build out.  And so for the short term if you want a very specific report with certain data fields in  it you can do that using Kibana where all of the events that we're capturing are there.  You can organize sort them however you want to and create your own visualizations and  lists but then we do plan to add that into the product to make it very easy to do.  And I think I'm going to take a little bit further.  The data, sorry, the value proposition is you'll always have unfettered access to the  raw data that we collect.  And then what we do is we have our own data processing pipeline where we process that  raw data in ways that we anticipate are helpful for you and then we provide visualizations  on that data so that you can really quickly and easily understand that.  I'm trying to compromise with that processing pipeline.  If you're going to make the API that I proposition to split that pipeline would push the data  back into Elasticsearch?  It does.  Okay.  Yep.  Yeah.  And if anybody wants more one-on-one conversation about the product we do have the product suite  downstairs from four to five.  If you want to kind of tell us about your use case and talk through it, it's room 407.  Cool.  That's such a wide range because if you're someone who runs like a maintenance run list  once a day or if you're someone that runs full suites every 30 minutes, we haven't been  able to do that level of capacity analysis yet.  Internally we haven't really outgrown our installation yet, which is a moderately sized  installation with like 12 or 20 nodes that we have, but we haven't had a chance to do  that yet.  I'm sure that we'll probably offer some sort of capacity planning because I've built systems  like that before.  That's a very important part of it.  We just haven't had enough time to do that process yet.  Cool.  Yeah.  In the back.  How long do you think it's going to take for your 12-ton node to this place?  I think we're on an M2 something XL on Amazon with no way, sorry.  That's a completely different one.  I have no idea.  It's actually relatively small.  Gigabytes.  We are still sub-gigabytes.  Yeah.  I've been hearing the other.  This is the replacement for Analysts.  There's one way to put it, but this is obviously a much different product.  Yeah.  Analytics doesn't have RBAC.  So basically this is the next generation of our approach to analytics.  As we look at this, how is it significantly the next step for us?  The first piece of that is that it allows us to get all of your data in one place, whereas  analytics and manage are both one Chef server to one installation of the analytics platform.  That's a pretty big deal because you don't want to have segregated data all over the  place.  So you can get all of your data in one place where you can see all of your data and be able  to programmatically understand what is the health of my infrastructure as code.  Next, we have basically added significantly more of richer data from the sources that  we're collecting it from.  Yeah.  So instead of old looking, my question is more backwards looking.  So yeah, we've gotten analytics from now, we've worked through play, and we've got  to see what is the future.  So we're going to have to use our education.  We don't have this anymore, but it's fine.  I mean, the main feature that we currently don't have that analytics has is rules and  notifications.  And so we have the data tab, which is a fire hose of data, but then actually sorting that  out and sending it to a specific endpoint is not there yet.  So that is what we are trying to figure out exactly what people want out of that tool.  We weren't really happy in retrospect with the rules and notifications engine of analytics.  We think it was too difficult to use.  And so a lot of people weren't using it to its full potential, and it was really easy  to get way too many notifications to the point that they weren't maybe useful anymore.  So we are looking at exactly how we want to do rules and notifications and what that looks  like.  And that kind of ties into an earlier question someone was asking, which was, how do I get  the data from here to people who need a report, for example?  And then what does that look like?  And that's something that we're really looking into.  I think that was the API question.  But basically, is it an API?  Can I just take this data programmatically every now and then and shoot it to another  system?  And that's one thing we're looking at.  We're looking at perhaps a scheduled email report of your own choosing on a regular basis.  But we're just sort of playing with different ideas and presenting them to customers and  then seeing what will fit that use case in the best possible way, but also in a significantly  more user-friendly way.  And then the other big thing we have around rules and notifications is the problem with  analytics was that everyone could edit the rules and notifications if they had access  to analytics.  And that means that you might want your developers to be able to write a rule and notification  just for the servers they care about, but you don't want them to be able to delete your  operations core rules and notifications that you super care about.  So those are all use cases that we have in mind as we figure out exactly what we're going  to deliver.  Yeah.  Cool.  In the back.  Yeah, Tasha at chef.io.  Yeah.  Yeah.  I mean, I am very open to being on customer calls.  Brooke and I think that a significant part of our job and Christoph as well is talking  to customers, getting feedback before we build stuff and while we're building stuff just  to make sure that we're really hitting the nail on the head when we deliver it.  Yeah.  Yeah.  Exactly.  In the back.  Is there someone?  Okay.  Stretching.  Yeah.  Yeah.  So delivery.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  So delivery to work for it looks like a team of customers.  It's like putting in the same back and forth.  This looks like a totally different.  How do you do about getting the customer in the back end?  Yeah.  We current.  So while elastic search is amazingly elastic and we anticipate that you could do a migration  from the back end of analytics into this, we have not productized that right now.  Yeah.  So we want you to say that you can use the search.  No.  Cool.  Any other questions?  Well, as you may know from Barry's presentation earlier, you can go and download a free trial  of this and give it a shot on our website, chef.io.  And we also have some tutorials that we'll take you through using this if you want to  check it out.  But it's a new product.  We are eagerly awaiting feedback and just kind of seeing where it goes and how people  want to use it.  So the more communication, the better.  We're really interested in what you think.  And we will be delivering a lot of new awesome features pretty soon.  So yeah, give it a shot.  Let us know what you think.  Come talk to us in 407.  Mark said come talk to us in 407.  Cool.  Thank you.  Thank you.  Thank you.
Chef_Automate_Visibility_Feature_0_131.png Chef_Automate_Visibility_Feature_132_803.png Chef_Automate_Visibility_Feature_804_851.png Chef_Automate_Visibility_Feature_852_923.png Chef_Automate_Visibility_Feature_924_1367.png Chef_Automate_Visibility_Feature_1368_1415.png Chef_Automate_Visibility_Feature_1416_1895.png Chef_Automate_Visibility_Feature_1896_1943.png Chef_Automate_Visibility_Feature_1944_2819.png Chef_Automate_Visibility_Feature_2820_2867.png Chef_Automate_Visibility_Feature_2868_5147.png Chef_Automate_Visibility_Feature_5148_5195.png Chef_Automate_Visibility_Feature_5196_8795.png Chef_Automate_Visibility_Feature_8796_8843.png Chef_Automate_Visibility_Feature_8844_8915.png Chef_Automate_Visibility_Feature_8916_9023.png Chef_Automate_Visibility_Feature_9024_9071.png Chef_Automate_Visibility_Feature_9072_9443.png Chef_Automate_Visibility_Feature_9444_9491.png Chef_Automate_Visibility_Feature_9492_9551.png Chef_Automate_Visibility_Feature_9552_9695.png Chef_Automate_Visibility_Feature_9696_23111.png Chef_Automate_Visibility_Feature_23112_23159.png Chef_Automate_Visibility_Feature_23160_23207.png Chef_Automate_Visibility_Feature_23208_34823.png Chef_Automate_Visibility_Feature_34824_41651.png Chef_Automate_Visibility_Feature_41652_42659.png Chef_Automate_Visibility_Feature_42660_52967.png Chef_Automate_Visibility_Feature_52968_55715.png Chef_Automate_Visibility_Feature_55716_55775.png Chef_Automate_Visibility_Feature_55776_57947.png Chef_Automate_Visibility_Feature_57948_58067.png Chef_Automate_Visibility_Feature_58068_58907.png Chef_Automate_Visibility_Feature_58908_58944.png