SAP's IT Journey into DevOps
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Schygulla, Marin
SAP IT story into DevOps using Chef as our main automation tool in order to achieve large scale configuration management and policy enforcement on servers and user desktops, collaboration with development teams, providing automated CI processes.
 Today, my name is Martin Cugula and I'm working for a company which is called SAP. And in my talk today, I would like you to tell our story, how we went into our DevOps journey and how we choose Chef as our main automation tool to do configuration management and policy enforcement and how we collaborate and work together with our development teams. So before I start, maybe it's good to know how many people of you know SAP? Raise hands? Wow, more than I expected. So but anyway, I give you a small introduction about SAP, who we are and what we are doing. So SAP is creating enterprise applications and SAP is a company which has a history of 44 years. So it was founded in the 70s and since then we are creating applications and innovations for enterprise companies. At the moment SAP serves over 300,000 customers in over 190 countries and we have partners, over 12,000 partners, companies which we are working globally together. SAP has at the moment around 78,000 employees worldwide in over 130 countries. Also we have SAP has development locations in over 100 countries, also worldwide distributed. So we are a globally spread company with development centers worldwide. To give you a little bit an overview about what kind of products SAP provides, I have just added a couple of products to the slide here. If you want to have more details about our products, what we are offering to our customers, please visit our company webpage, SAP.com. So you might have heard or if you are interested in databases, you came across about our in-memory database SAP HANA and of course our SAP HANA platform. Of course we are creating SAP business suite for enterprise customers but we are also doing enterprise analytics applications and also big data for our customers. SAP is creating also mobile applications. I saw a number on our official webpage that we have something like about 130 different mobile applications which we create for our customers and of course we are providing or creating also applications for the Internet of Things and of course for the cloud. So who am I? As I said, my name is Martin Schugler and I started to work in 2001 already with SAP. So I am 15 years in this company and I had of course several roles during that time held in SAP. And one of the last roles I had with SAP was an architect role for a small team which creates applications or solutions for our SAP internal community as well for external properties. So this team created or creates applications or creates our SAP.com website, also our SAP online store and the cool thing what they did is when I joined the team five years ago they did something what I didn't know before, something like what they called continuous integration, continuous delivery and DevOps. So already five years ago there was a small team from SAP what was working with that kind of methodologies and tools. So during that time I learned a lot about DevOps and continuous integration and continuous delivery. So my current position is that I am managing an automation team for the DevOps COE team and I come into this in a few moments. So to give you a little bit an insight about our environment, who we are and what we are doing, this is how our unit with an SAP IT is currently looking like or look like. We have in three teams within our organizations two teams are providing operational support and the one team is the global landscape, global development systems operations team. So this team is providing the operational support for the infrastructure of our development teams. As I mentioned in the beginning SAP has something about 100 development centers and labs around the world. So this team is providing the infrastructure support for those centers. The second team is the training demo consulting team, so this is basically the operations team for our training and demo systems. So they are doing the operations for the real customer trainings, what are given to the customers as well as demos for our sales force that they can do training demos for SAP applications and they are using those systems. There is a third group, the DevOps COE. I will come to that in a few minutes. So our team is distributed worldwide as well since we are an enterprise company. So we have teams sitting in North America, we have teams in Europe, we have teams in Israel, India, Singapore and China. The infrastructure what we have to operate on is it's very mixed. So we have physical hardware which we have to operate, we have virtual machines which we have to operate and we have also recently started to operate on container technology, basically also having the orchestration layer for this kind of technology what we are providing support for. From the type of operating systems we are supporting Windows systems, all kind of flavors or versions better to say, we have different Linux versions what we operate and provide to our development teams. We are providing also macOS operating systems to develop and test our Mac applications on that and of course some UNIX flavors what we have to operate on. So this should give you here a little bit an overview about the landscape and the distribution of our team what we have to handle. To give you some additional numbers that we have for our internal customers which are the development organizations and the training and demo teams, we have over 27,000 servers to manage and all servers or most of the servers are somehow different. So it's not a case that we have all the servers set up in the same way. We have basically a lot of applications what SAP is developing and maintaining. We have all this kind of applications and products of course infrastructure available and ready to test, develop and maintain. Our training organization, I think these are the numbers if I remember right from just from the first half of this year, provided over 4,000 training sessions to customers and also 2,600 demos for our customers. So this is the setup what we have to deal with and in 2015 the leadership of our unit sit together and they started to think about how can we support SAP in the digital transformation. So they decided or they made a decision that we want to integrate practices and tools into the SAP development value chain to enable the development teams to deliver the SAP innovation faster and more reliable. So this was basically the idea what they started in beginning of 2015 to transform our own unit to support SAP. So the leadership team set basically four targets which they wanted to achieve. The first target was to automate everything. So automation from development to the development process, the QA process up to production should be completely automated. So this was one of the targets what we wanted to achieve. The second target was to codify the infrastructure. Means that we have everything under infrastructure as code, everything is versioned that we don't do any manual modifications or changes to our infrastructure. The third target was to share and make it easy for our development teams and to share with other development or other operations organizations, our, let's say, our outcomes what we deliver as infrastructure code to the other teams so that they can easily access it and execute it on their own if they want to have it. The fourth target was that we want to provide our development teams a user-centric feature that they have a self-service to get and manage their infrastructure themselves as needed so that we can, that we are not always involved to provide them infrastructure support for just a single, for a simple request to just set up a machine and install a product for the developer on top of it so that he can test or evaluate and use case or an incident from a customer that he has easy access to such an infrastructure. So the targets were set and we decided, of course, we decided to have everything enter and automated but, of course, you need to define also for the targets some matrix to measure the outcome. So for this, what we decided or the leadership decided that time is that everything what we do with these targets that we achieve is to make it cheaper to put incremental changes and in terms of cost and time at risk to make everything cheaper. Of course, we want to increase the speed and the reliability of our infrastructure and deployments for the development teams and for the operations teams. So with that in 2015, we, the leadership team set the targets and set also how to measure it, the value and the org unit or the unit itself started to execute on those targets. So it was, we created something like a small adoption program for DevOps beginning of 2015 and we choose several, let's say, key team members from our organization who can drive several topics and execute it on them. So during our implementation or adoption of the targets, what we wanted to go to, we created a lot of training sessions for the different teams. We created POCs to evaluate tools if it works for our requirements like we did in evaluation with Chef together, with the Chef server. If we can manage our diverse infrastructure, hardware, physical container was not at that time in focus, but if you can use the Chef server to manage that kind of infrastructure. During the time, the team had an evolution. So the team started from a traditional developer or system administrator to start into the DevOps journey to do things different than what they did before. So before the system administrator did things manually, they logged into the server, replied or worked on a service request or an incident and delivered the result to the customer. The developers had also in the beginning of the journey different tasks. So there was also a shift to have a new language to learn, basically Chef as a new coding language which the team had to adopt and to learn. And during that phase, we saw that in the beginning, the team did not really understand why we are doing the DevOps journey. So it did not really understand the goal, why we have to do that. So even if we described the targets, why we are doing this to help SAP with the transformation and we want to reduce the cycle times, we want to reduce the costs for support and so on, it was really hard and challenging to make the complete understand why we have to do this. Another challenge for us was that not every admin is a good ops and DevOps. So if you are doing system admin tasks in the same way, it is really hard for them to have a mind shift in a very short time to understand what is really the benefit of doing it as infrastructure code than before it was logging in or even having a small batch script which is doing my job. So we had to deal with this in the same way as we had, of course, that not every developer is a good DevOps. So some of the developers did not really understand why we have this whole source control quality check code review thing to do to just install an application. So why not just go to the server and hit execute and install the application. So we had to face also these kind of challenges. And something which is maybe obvious, but it is also what we had as the challenges that not everyone in the team has the same speed of progress to learn new things or to adopt or adopt the new techniques and the new practices what we wanted to do. So here we had our challenges. But we started the journey and what we did is we did the trainings for our teams and at the same time we started also to do some real projects with our customers, our internal customers, the development organizations. So we looked up for some teams which are interested and curious about going the way of DevOps which are open-minded and or even asked for help and support to us at that time to introduce DevOps methods and tools. And we started with them to create this kind of continuous integration, continuous delivery and DevOps integration into their development processes. Their year of executing this and this is basically at the moment the SES situation of this year and we are not yet done with our DevOps journey. So we would say in the middle of it is that we have created an DevOps COE and based off the projects what we did with our development teams and the work together with our operations teams and our development teams. We have created some services where we think or where we are successful with our internal customers, the development teams and to support them. So our services to our development unit and organizations is that we collaborate with the development teams to integrate automation into any continuous integration platform. So within SAP, you know it's a long history of company. We have not only one CI system but we have several CI systems. We have also some other engineering services which are providing a central platform for continuous integration. So we have the development teams which have not the power to have such an operation or such an API tool to operate. We have them to set it up, configure it or to use what we have available with SAP. The second service what we provide is that we automate server system and application provisioning for our development and IT operations. So today we have automated the setup or the deployment of application servers as well as the setup of the application servers with SAP applications or any kind of other development tools that are required, completely automated with Chef and of course with our internal cloud or infrastructure platforms. Another service what we are doing is and this is mainly for our ops team itself is that we apply and enforce configuration management on our production and our development systems. For that we are using here Chef as well and we have the requirement as anyone here in the room I guess and I heard it already many times in the keynote talk is the compliance topic. So we have of course also to be compliant and we have also to enforce several security standards on our servers. Even if they are the development systems we have to be compliant. For that reason we have created also with Chef cookbooks which are implying these kind of configurations and configurations and enforcements and settings for applications, operating systems and we execute them on a frequent or regular basis on those servers. So we have a scheduled task on all servers which execute a Chef client at the moment it's hourly to enforce the configuration on the servers and enforce the policies what we need to apply. So any manual changes what are done to the servers will be reverted within one hour. The next service what we provide is of course to make the life of developer easier is and self-service for the end-to-end process. So many developers just require to have a virtual machine with something installed on it even if it's an SAP product for testing or if they need something to try out. So this is some requirements the development teams have but they do not want to waste or spend the time to create those machines manually, run some kind of command tools to execute a Chef client run or whatever. So for that reason we supported a team and we created an UI which simplifies the whole usage of it so that the development teams do not have to take care about infrastructure topics so that they can focus on their development tasks and can just use our tools for that. Of course we adopt and implement the latest DevOps technologies and tools. We have of course a lot of contacts to our development organizations. They can either themselves with some ideas they want to implement but they don't know how to do it or they are looking for some help and solutions to implement DevOps tools and technologies and we are helping and supporting them here as well. Of course what we have is our main core competencies or services that we provide support for the development systems in all the labs locations. So the DevOps CVE team is if you will something which is at the moment needed for our operations team and for our development teams to glue everything together. So usually you would think about DevOps that you have within the development teams, the operations team embedded and they work together on a product and so on but due to the fact that we have that many development systems to support for us this is at the moment the better model how we can provide value to the development teams here. So how do we do DevOps projects with our development teams today? So we have split our DevOps CVE here as well into two main roles. One role is what we call the solution analysts. So these are some team members which are working very closely together with the LOBs, with our development teams and they are focusing on getting basically the requirements of what the development teams have in regards of continuous integration, continuous delivery, infrastructure needs and development tools what they need to have and require. Those kind of requirements will be then hand over to the automation team which I lead and my team is creating for that Chef cookbooks to install and automate the infrastructure and application installations for these requirements. So how does the solution analysts start is very simple. So they listen to the development teams and try to understand and map out their requirements and their pipeline, what their model is, how they work. So they really map out the complete process from the source code, check in until the build process to the testing phase up to packaging or production deployment. So they map everything out from the complete development stream. During that phase they identify the pain points and the bottlenecks of the development teams. So they try to quantify the process. They measure everything as possible within the build pipeline. So how many times the build is executed, how many times it's tested, how long it takes to build, how long it takes until the tests are completed. So everything what we can measure, we try to measure and we try to find the bottlenecks and potential some spots for improvements. As soon as we have the bottlenecks identified and we have measured the quantity of the builds and we know what we can improve, we of course try to find ways how we can improve it, how we can parallelize processes, how we can make things quicker, where can we improve the situation with automation. And this is how the development teams or this is how the solution analysts map out the development targets for the automation team which we then implement and then hand over to our customers the development teams. So on this slide I will give you just a small selection of tools, of our pick of tools. So it's not the, it's not, I don't want to say this is the tool of choice which you have to choose but this is what we are using here for our end-to-end process. And we are using for the planning, for our projects, we are using for version control Git, we code with Chef for automation, we do some automated testing with Food Critic and Hubocop, we are using our internal cloud platform and infrastructure to do some automated testing of our automation cookbooks as well. We are using here Jenkins to automate that process. We release our cookbooks into our internal shelf, so we have something like a supermarket internally with an SAP which is then also accessible from everyone where we do release our automation cookbooks. And then of course we can deploy those automation to any infrastructure what we have with an IT, either if it is the M-ware based or our internal private cloud solution or if it's Amazon OpenStack, whatever usually the cookbooks are running on all these kind of infrastructures. And of course we do also operation with the tools like Chef Git and our internal incident management system. So we have an internal system which we call IT Direct. So here we are handling incidents or service requests for our infrastructures which we operate on. For the monitoring we are using tools for the project genre of course, for the infrastructure we have in NARCUS in place for that. So what do we do with Chef? As I mentioned, we try to automate everything as far as possible. So any kind of request what we get from development teams to automate or to deploy on infrastructures, to install on infrastructures, to configure on infrastructure if it is a database, if it is an application server or anything else. We try to automate everything end to end. For that reason my team has created meanwhile 177 cookbooks. None of them are available on the supermarket externally because they very have some specific requirements which are only required for us internally SAP. Of course we also look if there is something available on the supermarket which we can reuse, which we then either try to use or if we have to redevelop it. No, they are only internal release. We have an internal shelf, something like a cookbook shelf or supermarket and we release them. We have them only released internally. Although we started to open source our cookbooks as well, I think we have two cookbooks meanwhile open source basically SAP HANA Studio and something else with SAP HANA. You find it also on our public GitHub repository for that. But our goal is to contribute also to the open source community more and more. We started with open sourcing it and we will continue with that. Not only. It is a good question. The question was if the cookbooks are created only by the center of excellence team or also by the infrastructure team. We have also from our operations team selected members which are contributing and helping us to create those cookbooks. Because our developers who are creating those cookbooks are not that deep in infrastructure requirements as our operations team itself. For that reason we have also operations team members which are creating and maintaining cookbooks. So with the combination of operations team and the automation team which are creating and maintaining cookbooks. So in a typical DevOps presentation it is always like you have to have silos slide in the middle. I cannot repeat everything. The complete question I know to the audience. But the short form was don't you create another silo with your DevOps. At the moment we have the feeling that we have to have this kind of team at the moment. Because in such huge enterprise you cannot transform it from day one to complete organization. If you do that I was listening to yesterday also to this leadership summit where they had same questions. And my feeling was many companies are just starting to think how should I create my organization to have DevOps in place. So how do I rearrange, how do I organize my enterprise organization to enable DevOps. So our idea was a little bit different. We are an operations team. We wanted to do DevOps. We wanted to do it with the development teams. But instead of waiting or asking for big reorganizations we just started. We choose four targets which we thought we can achieve during the first year and we executed on that. And we just started. And for that reason we created the DevOps CVE or the outcome was that we have the DevOps CVE which helps and supports that. As I said we are not at the end of our transformation journey into DevOps but we started with that. It fulfills at the moment our needs and requirements and I would be happy if we can get rid of the DevOps CVE in the future and that we have all the development teams and operations team integrated and not having this kind of silos. At the moment for the time being it was our solution to provide this or to go our way into DevOps. So there are other activities that we have to do apart from the management services. So how do you manage those with the dev? Good question. How do we manage not only application or configuration management tasks but also all the other operational tasks? How do we manage them today with the operations team? Questions do we use Chef or that at the moment? This is the question. So we try to do as much as possible with Chef. We cannot do everything with Chef because of course we are doing the support or the operation for our development landscapes but of course there are other landscapes within SAP and there are also some tools already in place which are used. So we have a combination of all at the moment. So we are using also tools from other companies to automate some kind of task but from our perspective we are using at the moment as much as we can Chef and we want also to work on that much more. But at the moment we have also here a mixed situation that we have also some other tools in place where we have to execute and use at the moment. We hope that we can change that also in the future. So you bothijd lign down with this __________. But when you look at people who do not olika they have their own tasks but there is a 곱 like an ADP or an MN, and actually one of those things that they just didn't interact with the other things. They have their own set of rules. Yeah. So even there, we need a set of automations that we can speed up our activities over there. That is bad, I think. But the reason why I'm asking is, those teams are not using any automation at the moment. They are, whatever they have been doing is the kind of point in here. So there's a lot of data for that. So I have to introduce the automation or yes, how do we introduce that? So that's a kind of more capability to the team. Yeah. So to summarize the question a little bit, I hope I get everything together. I think we should hand out mics. It makes it easier. How do we integrate other operation teams, which are providing additional support to basically the operations team, which are already using automation team? Sorry, Chef Automation, for example, or Infrastructure as code. We are in the same situation. So we have also other teams. You can, for example, the network team, which are not using our automation for that. So here we have to use, of course, at the moment, if we can, we can help and show them what we can automate and we can offer them to automate their task as well, at least for our infrastructure and operation components, which we are responsible for. Or we have to live at the moment with the situation where they work in a manner of process. So for that, we have also to create service requests or incident requests to other teams to deliver a service which we rely on. So we have not everything under our control. That's clear. I mean, this is, I think, in the most large enterprises, the case that we do not have the full infrastructure up down to the bare metal in one hand. So we're providing the operational support. We can deploy virtual machines. We, but we cannot go to the data center and put the physical machines into direct. So this is some other service team which is providing that. I need to rush. I get sickness. So let me just quickly finish the slides here because I think here, two numbers, which are also pretty impressive for us at least, is, we have also, or the automation team has created 27 landscape templates. So these templates is something where we are able to deploy a complete landscape setup. If it's multi-tier, if it's a cluster setup, whatever. So with one click, you can deploy basically an automated complete setup of an landscape. And what we also do is we measure the execution of cookbooks. These numbers are also from the first half of the year. So within SAP, we can say that we have automations or individuals using our tools and using our cookbooks. And we have over 32,000 cookbooks executions in the last half year. So 30,000 times one of our 107 cookbooks was executed. And we have also over 6,000 landscapes deployed based on our template landscapes, landscape templates, what we have provided here. The last point, what I have here on the slide here is also that, which I mentioned before, is that we apply and enforce configuration management on our infrastructure. So we have created also cookbooks and configurations to fulfill compliance, but also to fulfill some operational requirements from our operations team, which we apply to our infrastructure. So we make sure that we have several monitoring tools or the monitoring tools installed, that we have an anti-virus solution installed on all the servers, that the security settings are set as they have to be. So this is what we're using also on our infrastructure and which we enforce on a regular basis on all the servers. Just two slides left. I hope the time is enough. I just wanted to give you also a quick, not a demo, but just two screenshots about, which I mentioned before, as one of our services to our developers is. So our team has implemented with the SAP HANA Cloud Platform and User Interface, which interacts with Chef to automate infrastructure. So basically, the developers are able to go to the website, select if they want to have a landscape or a server deployed, can choose on a selection screen. It's really hard to see here, but you can filter, do a search about our 177 cookbooks that we provide. You can search for an application which you need to install. You can check market, and then you can click on the single button and the tool takes care about to deploy either the single server or the landscape and to do the installation based on the automation that we have created for here. So this helps also our development teams a lot to reduce the time to get machines and applications installed for their development. So what are the key takeaways of this talk here and based on the questions, what I already answered is, our goal was to start the DevOps journey with small steps. Yes, you can do it for a complete enterprise if you do a big reorg or doing a top-down to execute DevOps in your organizations. Our approach was more to introduce the DevOps techniques and methods into our own organizations and working together with our customers, our internal customers, and we started small. We started with the teams and the development teams together and collaborated together. Then the second thing is you have to train and educate your teams to understand the reason, which is very important, to understand the value, why we are doing this, and of course you need to learn and train the people about the new tools if you introduce something new to the organization so that they have the knowledge and also the possibility to learn on those new tools. And of course the most important thing is communicate and collaborate with other teams. In a big enterprise like SAP, we are not the only ones doing DevOps. There are a lot of other teams doing DevOps, and we try to identify, to find these teams and to collaborate with them as well. And of course we try to collaborate and communicate with other development teams to have and build up a community with SAP so that we can make, let's say, a transformation in the enterprise out of the community with SAP and not based on an organizational structure or transformation. So this was my talk on my slides. I think I'm on time actually. So thanks for listening. If you have more questions, please feel free to come to me after the talk, and thanks for being here. Thanks for joining us, Martin.
 Today, my name is Martin Cugula and I'm working for a company which is called SAP.  And in my talk today, I would like you to tell our story, how we went into our DevOps  journey and how we choose Chef as our main automation tool to do configuration management  and policy enforcement and how we collaborate and work together with our development teams.  So before I start, maybe it's good to know how many people of you know SAP?  Raise hands?  Wow, more than I expected.  So but anyway, I give you a small introduction about SAP, who we are and what we are doing.  So SAP is creating enterprise applications and SAP is a company which has a history  of 44 years.  So it was founded in the 70s and since then we are creating applications and innovations  for enterprise companies.  At the moment SAP serves over 300,000 customers in over 190 countries and we have partners,  over 12,000 partners, companies which we are working globally together.  SAP has at the moment around 78,000 employees worldwide in over 130 countries.  Also we have SAP has development locations in over 100 countries, also worldwide distributed.  So we are a globally spread company with development centers worldwide.  To give you a little bit an overview about what kind of products SAP provides, I have  just added a couple of products to the slide here.  If you want to have more details about our products, what we are offering to our customers,  please visit our company webpage, SAP.com.  So you might have heard or if you are interested in databases, you came across about our in-memory  database SAP HANA and of course our SAP HANA platform.  Of course we are creating SAP business suite for enterprise customers but we are also doing  enterprise analytics applications and also big data for our customers.  SAP is creating also mobile applications.  I saw a number on our official webpage that we have something like about 130 different  mobile applications which we create for our customers and of course we are providing or  creating also applications for the Internet of Things and of course for the cloud.  So who am I?  As I said, my name is Martin Schugler and I started to work in 2001 already with SAP.  So I am 15 years in this company and I had of course several roles during that time held  in SAP.  And one of the last roles I had with SAP was an architect role for a small team which creates  applications or solutions for our SAP internal community as well for external properties.  So this team created or creates applications or creates our SAP.com website, also our SAP  online store and the cool thing what they did is when I joined the team five years ago  they did something what I didn't know before, something like what they called continuous  integration, continuous delivery and DevOps.  So already five years ago there was a small team from SAP what was working with that kind  of methodologies and tools.  So during that time I learned a lot about DevOps and continuous integration and continuous  delivery.  So my current position is that I am managing an automation team for the DevOps COE team  and I come into this in a few moments.  So to give you a little bit an insight about our environment, who we are and what we are  doing, this is how our unit with an SAP IT is currently looking like or look like.  We have in three teams within our organizations two teams are providing operational support  and the one team is the global landscape, global development systems operations team.  So this team is providing the operational support for the infrastructure of our development  teams.  As I mentioned in the beginning SAP has something about 100 development centers and labs around  the world.  So this team is providing the infrastructure support for those centers.  The second team is the training demo consulting team, so this is basically the operations  team for our training and demo systems.  So they are doing the operations for the real customer trainings, what are given to  the customers as well as demos for our sales force that they can do training demos for  SAP applications and they are using those systems.  There is a third group, the DevOps COE.  I will come to that in a few minutes.  So our team is distributed worldwide as well since we are an enterprise company.  So we have teams sitting in North America, we have teams in Europe, we have teams in  Israel, India, Singapore and China.  The infrastructure what we have to operate on is it's very mixed.  So we have physical hardware which we have to operate, we have virtual machines which  we have to operate and we have also recently started to operate on container technology,  basically also having the orchestration layer for this kind of technology what we are providing  support for.  From the type of operating systems we are supporting Windows systems, all kind of flavors or versions  better to say, we have different Linux versions what we operate and provide to our development  teams.  We are providing also macOS operating systems to develop and test our Mac applications on  that and of course some UNIX flavors what we have to operate on.  So this should give you here a little bit an overview about the landscape and the distribution  of our team what we have to handle.  To give you some additional numbers that we have for our internal customers which are  the development organizations and the training and demo teams, we have over 27,000 servers  to manage and all servers or most of the servers are somehow different.  So it's not a case that we have all the servers set up in the same way.  We have basically a lot of applications what SAP is developing and maintaining.  We have all this kind of applications and products of course infrastructure available  and ready to test, develop and maintain.  Our training organization, I think these are the numbers if I remember right from just  from the first half of this year, provided over 4,000 training sessions to customers  and also 2,600 demos for our customers.  So this is the setup what we have to deal with and in 2015 the leadership of our unit  sit together and they started to think about how can we support SAP in the digital transformation.  So they decided or they made a decision that we want to integrate practices and tools into  the SAP development value chain to enable the development teams to deliver the SAP innovation  faster and more reliable.  So this was basically the idea what they started in beginning of 2015 to transform our own  unit to support SAP.  So the leadership team set basically four targets which they wanted to achieve.  The first target was to automate everything.  So automation from development to the development process, the QA process up to production should  be completely automated.  So this was one of the targets what we wanted to achieve.  The second target was to codify the infrastructure.  Means that we have everything under infrastructure as code, everything is versioned that we don't  do any manual modifications or changes to our infrastructure.  The third target was to share and make it easy for our development teams and to share  with other development or other operations organizations, our, let's say, our outcomes  what we deliver as infrastructure code to the other teams so that they can easily access  it and execute it on their own if they want to have it.  The fourth target was that we want to provide our development teams a user-centric feature  that they have a self-service to get and manage their infrastructure themselves as needed  so that we can, that we are not always involved to provide them infrastructure support for  just a single, for a simple request to just set up a machine and install a product for  the developer on top of it so that he can test or evaluate and use case or an incident  from a customer that he has easy access to such an infrastructure.  So the targets were set and we decided, of course, we decided to have everything enter  and automated but, of course, you need to define also for the targets some matrix to  measure the outcome.  So for this, what we decided or the leadership decided that time is that everything what  we do with these targets that we achieve is to make it cheaper to put incremental changes  and in terms of cost and time at risk to make everything cheaper.  Of course, we want to increase the speed and the reliability of our infrastructure and  deployments for the development teams and for the operations teams.  So with that in 2015, we, the leadership team set the targets and set also how to measure  it, the value and the org unit or the unit itself started to execute on those targets.  So it was, we created something like a small adoption program for DevOps beginning of 2015  and we choose several, let's say, key team members from our organization who can drive  several topics and execute it on them.  So during our implementation or adoption of the targets, what we wanted to go to, we  created a lot of training sessions for the different teams.  We created POCs to evaluate tools if it works for our requirements like we did in evaluation  with Chef together, with the Chef server.  If we can manage our diverse infrastructure, hardware, physical container was not at that  time in focus, but if you can use the Chef server to manage that kind of infrastructure.  During the time, the team had an evolution.  So the team started from a traditional developer or system administrator to start into the  DevOps journey to do things different than what they did before.  So before the system administrator did things manually, they logged into the server, replied  or worked on a service request or an incident and delivered the result to the customer.  The developers had also in the beginning of the journey different tasks.  So there was also a shift to have a new language to learn, basically Chef as a new coding language  which the team had to adopt and to learn.  And during that phase, we saw that in the beginning, the team did not really understand  why we are doing the DevOps journey.  So it did not really understand the goal, why we have to do that.  So even if we described the targets, why we are doing this to help SAP with the transformation  and we want to reduce the cycle times, we want to reduce the costs for support and so  on, it was really hard and challenging to make the complete understand why we have to  do this.  Another challenge for us was that not every admin is a good ops and DevOps.  So if you are doing system admin tasks in the same way, it is really hard for them to  have a mind shift in a very short time to understand what is really the benefit of doing  it as infrastructure code than before it was logging in or even having a small batch script  which is doing my job.  So we had to deal with this in the same way as we had, of course, that not every developer  is a good DevOps.  So some of the developers did not really understand why we have this whole source control quality  check code review thing to do to just install an application.  So why not just go to the server and hit execute and install the application.  So we had to face also these kind of challenges.  And something which is maybe obvious, but it is also what we had as the challenges that  not everyone in the team has the same speed of progress to learn new things or to adopt  or adopt the new techniques and the new practices what we wanted to do.  So here we had our challenges.  But we started the journey and what we did is we did the trainings for our teams and  at the same time we started also to do some real projects with our customers, our internal  customers, the development organizations.  So we looked up for some teams which are interested and curious about going the way of DevOps  which are open-minded and or even asked for help and support to us at that time to introduce  DevOps methods and tools.  And we started with them to create this kind of continuous integration, continuous delivery  and DevOps integration into their development processes.  Their year of executing this and this is basically at the moment the SES situation of this year  and we are not yet done with our DevOps journey.  So we would say in the middle of it is that we have created an DevOps COE and based off  the projects what we did with our development teams and the work together with our operations  teams and our development teams.  We have created some services where we think or where we are successful with our internal  customers, the development teams and to support them.  So our services to our development unit and organizations is that we collaborate with  the development teams to integrate automation into any continuous integration platform.  So within SAP, you know it's a long history of company.  We have not only one CI system but we have several CI systems.  We have also some other engineering services which are providing a central platform for  continuous integration.  So we have the development teams which have not the power to have such an operation or  such an API tool to operate.  We have them to set it up, configure it or to use what we have available with SAP.  The second service what we provide is that we automate server system and application  provisioning for our development and IT operations.  So today we have automated the setup or the deployment of application servers as well  as the setup of the application servers with SAP applications or any kind of other development  tools that are required, completely automated with Chef and of course with our internal  cloud or infrastructure platforms.  Another service what we are doing is and this is mainly for our ops team itself is that  we apply and enforce configuration management on our production and our development systems.  For that we are using here Chef as well and we have the requirement as anyone here in the  room I guess and I heard it already many times in the keynote talk is the compliance  topic.  So we have of course also to be compliant and we have also to enforce several security  standards on our servers.  Even if they are the development systems we have to be compliant.  For that reason we have created also with Chef cookbooks which are implying these kind  of configurations and configurations and enforcements and settings for applications, operating  systems and we execute them on a frequent or regular basis on those servers.  So we have a scheduled task on all servers which execute a Chef client at the moment  it's hourly to enforce the configuration on the servers and enforce the policies what  we need to apply.  So any manual changes what are done to the servers will be reverted within one hour.  The next service what we provide is of course to make the life of developer easier is and  self-service for the end-to-end process.  So many developers just require to have a virtual machine with something installed on  it even if it's an SAP product for testing or if they need something to try out.  So this is some requirements the development teams have but they do not want to waste or  spend the time to create those machines manually, run some kind of command tools to execute a  Chef client run or whatever.  So for that reason we supported a team and we created an UI which simplifies the whole  usage of it so that the development teams do not have to take care about infrastructure  topics so that they can focus on their development tasks and can just use our tools for that.  Of course we adopt and implement the latest DevOps technologies and tools.  We have of course a lot of contacts to our development organizations.  They can either themselves with some ideas they want to implement but they don't know  how to do it or they are looking for some help and solutions to implement DevOps tools  and technologies and we are helping and supporting them here as well.  Of course what we have is our main core competencies or services that we provide support for the  development systems in all the labs locations.  So the DevOps CVE team is if you will something which is at the moment needed for our operations  team and for our development teams to glue everything together.  So usually you would think about DevOps that you have within the development teams, the  operations team embedded and they work together on a product and so on but due to the fact  that we have that many development systems to support for us this is at the moment the  better model how we can provide value to the development teams here.  So how do we do DevOps projects with our development teams today?  So we have split our DevOps CVE here as well into two main roles.  One role is what we call the solution analysts.  So these are some team members which are working very closely together with the LOBs, with  our development teams and they are focusing on getting basically the requirements of what  the development teams have in regards of continuous integration, continuous delivery, infrastructure  needs and development tools what they need to have and require.  Those kind of requirements will be then hand over to the automation team which I lead and  my team is creating for that Chef cookbooks to install and automate the infrastructure  and application installations for these requirements.  So how does the solution analysts start is very simple.  So they listen to the development teams and try to understand and map out their requirements  and their pipeline, what their model is, how they work.  So they really map out the complete process from the source code, check in until the build  process to the testing phase up to packaging or production deployment.  So they map everything out from the complete development stream.  During that phase they identify the pain points and the bottlenecks of the development teams.  So they try to quantify the process.  They measure everything as possible within the build pipeline.  So how many times the build is executed, how many times it's tested, how long it takes  to build, how long it takes until the tests are completed.  So everything what we can measure, we try to measure and we try to find the bottlenecks  and potential some spots for improvements.  As soon as we have the bottlenecks identified and we have measured the quantity of the builds  and we know what we can improve, we of course try to find ways how we can improve it, how  we can parallelize processes, how we can make things quicker, where can we improve the situation  with automation.  And this is how the development teams or this is how the solution analysts map out the development  targets for the automation team which we then implement and then hand over to our customers  the development teams.  So on this slide I will give you just a small selection of tools, of our pick of tools.  So it's not the, it's not, I don't want to say this is the tool of choice which you have  to choose but this is what we are using here for our end-to-end process.  And we are using for the planning, for our projects, we are using for version control  Git, we code with Chef for automation, we do some automated testing with Food Critic  and Hubocop, we are using our internal cloud platform and infrastructure to do some automated  testing of our automation cookbooks as well.  We are using here Jenkins to automate that process.  We release our cookbooks into our internal shelf, so we have something like a supermarket  internally with an SAP which is then also accessible from everyone where we do release  our automation cookbooks.  And then of course we can deploy those automation to any infrastructure what we have with an  IT, either if it is the M-ware based or our internal private cloud solution or if it's  Amazon OpenStack, whatever usually the cookbooks are running on all these kind of infrastructures.  And of course we do also operation with the tools like Chef Git and our internal incident  management system.  So we have an internal system which we call IT Direct.  So here we are handling incidents or service requests for our infrastructures which we  operate on.  For the monitoring we are using tools for the project genre of course, for the infrastructure  we have in NARCUS in place for that.  So what do we do with Chef?  As I mentioned, we try to automate everything as far as possible.  So any kind of request what we get from development teams to automate or to deploy on infrastructures,  to install on infrastructures, to configure on infrastructure if it is a database, if  it is an application server or anything else.  We try to automate everything end to end.  For that reason my team has created meanwhile 177 cookbooks.  None of them are available on the supermarket externally because they very have some specific  requirements which are only required for us internally SAP.  Of course we also look if there is something available on the supermarket which we can reuse,  which we then either try to use or if we have to redevelop it.  No, they are only internal release.  We have an internal shelf, something like a cookbook shelf or supermarket and we release  them.  We have them only released internally.  Although we started to open source our cookbooks as well, I think we have two cookbooks meanwhile  open source basically SAP HANA Studio and something else with SAP HANA.  You find it also on our public GitHub repository for that.  But our goal is to contribute also to the open source community more and more.  We started with open sourcing it and we will continue with that.  Not only.  It is a good question.  The question was if the cookbooks are created only by the center of excellence team or also  by the infrastructure team.  We have also from our operations team selected members which are contributing and helping  us to create those cookbooks.  Because our developers who are creating those cookbooks are not that deep in infrastructure  requirements as our operations team itself.  For that reason we have also operations team members which are creating and maintaining  cookbooks.  So with the combination of operations team and the automation team which are creating  and maintaining cookbooks.  So in a typical DevOps presentation it is always like you have to have silos slide  in the middle.  I cannot repeat everything.  The complete question I know to the audience.  But the short form was don't you create another silo with your DevOps.  At the moment we have the feeling that we have to have this kind of team at the moment.  Because in such huge enterprise you cannot transform it from day one to complete organization.  If you do that I was listening to yesterday also to this leadership summit where they  had same questions.  And my feeling was many companies are just starting to think how should I create my organization  to have DevOps in place.  So how do I rearrange, how do I organize my enterprise organization to enable DevOps.  So our idea was a little bit different.  We are an operations team.  We wanted to do DevOps.  We wanted to do it with the development teams.  But instead of waiting or asking for big reorganizations we just started.  We choose four targets which we thought we can achieve during the first year and we executed  on that.  And we just started.  And for that reason we created the DevOps CVE or the outcome was that we have the DevOps  CVE which helps and supports that.  As I said we are not at the end of our transformation journey into DevOps but we started with that.  It fulfills at the moment our needs and requirements and I would be happy if we can get rid of the  DevOps CVE in the future and that we have all the development teams and operations  team integrated and not having this kind of silos.  At the moment for the time being it was our solution to provide this or to go our way  into DevOps.  So there are other activities that we have to do apart from the management services.  So how do you manage those with the dev?  Good question.  How do we manage not only application or configuration management tasks but also all  the other operational tasks?  How do we manage them today with the operations team?  Questions do we use Chef or that at the moment?  This is the question.  So we try to do as much as possible with Chef.  We cannot do everything with Chef because of course we are doing the support or the  operation for our development landscapes but of course there are other landscapes within  SAP and there are also some tools already in place which are used.  So we have a combination of all at the moment.  So we are using also tools from other companies to automate some kind of task but from our  perspective we are using at the moment as much as we can Chef and we want also to work  on that much more.  But at the moment we have also here a mixed situation that we have also some other tools  in place where we have to execute and use at the moment.  We hope that we can change that also in the future.  So you bothijd lign down with this __________.  But when you look at people who do not olika they have their own tasks but there is a 곱  like an ADP or an MN, and actually one of those things  that they just didn't interact with the other things.  They have their own set of rules.  Yeah.  So even there, we need a set of automations  that we can speed up our activities over there.  That is bad, I think.  But the reason why I'm asking is,  those teams are not using any automation at the moment.  They are, whatever they have been doing  is the kind of point in here.  So there's a lot of data for that.  So I have to introduce the automation  or yes, how do we introduce that?  So that's a kind of more capability to the team.  Yeah.  So to summarize the question a little bit,  I hope I get everything together.  I think we should hand out mics.  It makes it easier.  How do we integrate other operation teams,  which are providing additional support  to basically the operations team,  which are already using automation team?  Sorry, Chef Automation, for example,  or Infrastructure as code.  We are in the same situation.  So we have also other teams.  You can, for example, the network team,  which are not using our automation for that.  So here we have to use, of course, at the moment,  if we can, we can help and show them what we can automate  and we can offer them to automate their task as well,  at least for our infrastructure and operation components,  which we are responsible for.  Or we have to live at the moment with the situation  where they work in a manner of process.  So for that, we have also to create service requests  or incident requests to other teams  to deliver a service which we rely on.  So we have not everything under our control.  That's clear.  I mean, this is, I think, in the most large enterprises,  the case that we do not have the full infrastructure  up down to the bare metal in one hand.  So we're providing the operational support.  We can deploy virtual machines.  We, but we cannot go to the data center  and put the physical machines into direct.  So this is some other service team which is providing that.  I need to rush.  I get sickness.  So let me just quickly finish the slides here  because I think here, two numbers,  which are also pretty impressive for us at least, is,  we have also, or the automation team has created  27 landscape templates.  So these templates is something where we are able  to deploy a complete landscape setup.  If it's multi-tier, if it's a cluster setup, whatever.  So with one click, you can deploy basically  an automated complete setup of an landscape.  And what we also do is we measure the execution of cookbooks.  These numbers are also from the first half of the year.  So within SAP, we can say that we have automations  or individuals using our tools and using our cookbooks.  And we have over 32,000 cookbooks executions  in the last half year.  So 30,000 times one of our 107 cookbooks was executed.  And we have also over 6,000 landscapes deployed  based on our template landscapes,  landscape templates, what we have provided here.  The last point, what I have here on the slide here  is also that, which I mentioned before,  is that we apply and enforce configuration management  on our infrastructure.  So we have created also cookbooks and configurations  to fulfill compliance, but also to fulfill some  operational requirements from our operations team,  which we apply to our infrastructure.  So we make sure that we have several monitoring tools  or the monitoring tools installed,  that we have an anti-virus solution installed  on all the servers, that the security settings are set  as they have to be.  So this is what we're using also on our infrastructure  and which we enforce on a regular basis on all the servers.  Just two slides left.  I hope the time is enough.  I just wanted to give you also a quick,  not a demo, but just two screenshots about,  which I mentioned before,  as one of our services to our developers is.  So our team has implemented with the SAP HANA Cloud Platform  and User Interface, which interacts with Chef  to automate infrastructure.  So basically, the developers are able to go to the website,  select if they want to have a landscape or a server deployed,  can choose on a selection screen.  It's really hard to see here,  but you can filter, do a search about our 177 cookbooks  that we provide.  You can search for an application which you need to install.  You can check market, and then you can click on the single button  and the tool takes care about to deploy either the single server  or the landscape and to do the installation  based on the automation that we have created for here.  So this helps also our development teams a lot  to reduce the time to get machines and applications  installed for their development.  So what are the key takeaways of this talk here  and based on the questions, what I already answered is,  our goal was to start the DevOps journey with small steps.  Yes, you can do it for a complete enterprise  if you do a big reorg or doing a top-down  to execute DevOps in your organizations.  Our approach was more to introduce the DevOps techniques  and methods into our own organizations  and working together with our customers, our internal customers,  and we started small.  We started with the teams and the development teams  together and collaborated together.  Then the second thing is you have to train and educate your teams  to understand the reason, which is very important,  to understand the value, why we are doing this,  and of course you need to learn and train the people  about the new tools if you introduce something new  to the organization so that they have the knowledge  and also the possibility to learn on those new tools.  And of course the most important thing is  communicate and collaborate with other teams.  In a big enterprise like SAP,  we are not the only ones doing DevOps.  There are a lot of other teams doing DevOps,  and we try to identify, to find these teams  and to collaborate with them as well.  And of course we try to collaborate  and communicate with other development teams  to have and build up a community with SAP  so that we can make, let's say,  a transformation in the enterprise out of the community  with SAP and not based on an organizational structure  or transformation.  So this was my talk on my slides.  I think I'm on time actually.  So thanks for listening.  If you have more questions, please feel free to come to me  after the talk, and thanks for being here.  Thanks for joining us, Martin.
SAP_s_IT_Journey_into_DevOps_0_47.png SAP_s_IT_Journey_into_DevOps_48_119.png SAP_s_IT_Journey_into_DevOps_120_1091.png SAP_s_IT_Journey_into_DevOps_1092_2855.png SAP_s_IT_Journey_into_DevOps_2856_4691.png SAP_s_IT_Journey_into_DevOps_4692_6959.png SAP_s_IT_Journey_into_DevOps_6960_12647.png SAP_s_IT_Journey_into_DevOps_12648_19367.png SAP_s_IT_Journey_into_DevOps_19368_24683.png SAP_s_IT_Journey_into_DevOps_24684_27611.png SAP_s_IT_Journey_into_DevOps_27612_27851.png SAP_s_IT_Journey_into_DevOps_27852_28655.png SAP_s_IT_Journey_into_DevOps_28656_29807.png SAP_s_IT_Journey_into_DevOps_29808_29987.png SAP_s_IT_Journey_into_DevOps_29988_30683.png SAP_s_IT_Journey_into_DevOps_30684_30803.png SAP_s_IT_Journey_into_DevOps_30804_30983.png SAP_s_IT_Journey_into_DevOps_30984_31331.png SAP_s_IT_Journey_into_DevOps_31332_32135.png SAP_s_IT_Journey_into_DevOps_32136_32315.png SAP_s_IT_Journey_into_DevOps_32316_32615.png SAP_s_IT_Journey_into_DevOps_32616_34175.png SAP_s_IT_Journey_into_DevOps_34176_37211.png SAP_s_IT_Journey_into_DevOps_37212_40127.png SAP_s_IT_Journey_into_DevOps_40128_55787.png SAP_s_IT_Journey_into_DevOps_55788_57671.png SAP_s_IT_Journey_into_DevOps_57672_61091.png