Automated, modularized and versioned infrastructure with Terraform and Terragrunt
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Karisch, Emily
Learn to automate your infrastructure with Terraform and Terragrunt in a modularized way. Declare your servers as cattle and don't manage them like pets. That's what you are able with fully automated infrastructure. Terraform modules help you to decouple your infrastructure into reusable components and code them together in a flexible way. With Terragrunt you are able to organize and evolve your infrastructure in a versioned and structured way. With both tools together you have whole insight into your infrastructure by just looking at the code.
 Let's begin. I will show you Terraform and Terragrant, two excellent tools for infrastructure automation. So hi at first. I love to have my lectures more interactive. So if anything is unclear, just ask in between or maybe I give you some questions. And I speak German and English. I do this talk in English because you wouldn't understand me in German with my dialect. I'm from Austria. So you can ask any question in English or in German. That's no problem. Good. I'm Emily. Just for my comfortness, I'm a non-binary person. My pronouns are in English, sei and sie or in German, sie or sie, if you talk about me. So just know them. Yeah. I work at Pixelat. We are an digital agency and my job title is called engineering and operations. It's, yeah. My job title is basically I do DevOps, many DevOps system administration, cloud administration, internal development, that's the engineering part. We could have called it DevOps. Yeah. Wayne. So my everyday work is getting up infrastructure, modify them, destroy them, getting up, destroy them, getting up, destroy them. So I don't want to do this by hand. So I have searched for tools, have learned these tools and I always use them. Even if I have to get up one machine, I code it. Because if any developer destroys something on this machine, I just take it down and re-graded it by script. Yeah. A little disclaimer. These code examples are written for Terraform below or equals 0.11 and TerraGrant below or equals 0.18. DevOps tools have released a new version with a new configuration language which is a bit different, has a first class level of variables and so on. So if you take these code examples and use the newest Terraform version, 0.12, you have to adapt them a bit. So the status quo. Who is automating his infrastructure? The rest of you have to do something with infrastructure? Yeah, no. The other person's developers. Okay. Yeah. So there's this famous model, pets versus cattle. Let's look a bit into it as an intro. So the pet service model. When you have a pet at home, what do you do with it? Pets have unique names. Like, my notebook is called, if I remember correctly, one of the four rivers of, yeah, one of the four rivers of Greek mythology. And your pet, they are cared, you like lovingly hand raised and cared for. Like your cat at home, you care for your cat. And if you have a server as a pet, you raise it by hand too. You install your packages by hand on the shell, you create your configuration by hand. Yeah. And when they get ill, you nurse them back to health. Like you have some configuration error, you fix it by hand, you get it running back, all by your hand, like, don't worry, I'm from tech support. On the other side, you have the cattle service model, like all your servers, all your cattle are given numbers, like MGNL, 05, 06, 07, et cetera. They are almost identical to other cattle, of course, because they are not hand raised, you automate them to create them. And when they get ill, you get another one. Like I said before, I destroy them, I regrade it. I just don't give any mind if something is in bad configuration state, if something is destroyed by our developers, I just destroy the whole machine and regrade it. There are a few exceptions, like your magical unicorn pet server, which can still be under configuration management. You have projects, we have only one server because, yeah, enterprise thingy and license costs of 10,000 euros, you have that. So you have this unique pet, but it's still graded by configuration management, infrastructure management, but I can't destroy it, because it's the only production server running, I still have to fix some things by hand. Yeah. So we have infrastructure as code. This is the cattle service model or your special pet. Infrastructure is code. Who has never heard about this term? Good. Just repeat it. Infrastructure is code is reproducible. It's plain defined infrastructure because you have code. It's defined, it has a state, it should have a state, and it's a foundation for DevOps. You can't do DevOps stuff without coded infrastructure. In few servers, all raised by hand, all configured by hand. One is different because one of the system engineers hasn't read the wiki exactly and missed a step. So infrastructure is code is a really important foundation for DevOps. And you gain visibility across your infrastructure because you just look in your infrastructure code, you know, oh, I have 10 servers, or I use this image, I use these packages. And for all of this, I use Terraform. Yeah, it's really Terraform.io. You can download it. Really great documentation. And the documentation is really, really great. I have used tools where the docs are, yeah, in a really bad state, or you don't, or stick overflow of the documentation. No, so Terraform. It's made by HashiCorp, who has heard of HashiCorp? Yeah, they make, they make PECA, console, yeah. The nicest part about Terraform, it has a declarative language. If you compare it to Chef or Ansible, which have a procedural style of infrastructure coding, Terraform is declarative like puppet. You define your resources, and they define the order of execution by themselves. And the HashiCorp configuration language is like a superset of JSON. If you paste JSON in your HashiCorp file, you still have to modify the syntax. But the nice part about it, as a superset of JSON, you can write your Terraform config in JSON too, which I use in a tool to automatically create infrastructure for tenants. I use JSON, scripted by Terraform, and it's done. Projects erased up and down by simple API calls. Yeah. Let's, who of you has never used Terraform? Perfect. So I give you some introduction about Terraform. Terraform has the notation of providers. In this case, I use the provider AWS. Yeah, we use AWS, I'm used to it. So I use this in my examples. In the next talk in this lecture room, you get a Terraform talk which uses Kubernetes. Yeah, cool. So a great defundation for you. Yeah. I have a provider, I use variables, AWS access key and secret key. I can provide the keys on the comment line. Don't use static credentials on the comment line. So I can configure the provider only with the region. Can supply the secrets key by environment variables. Every provider has a good documentation which environment variables are read in instead of the configuration parameters or that's not really better. So it depends on the provider how you can configure it. Like AWS provider, you have your dot AWS credentials file which is also read by Terraform. So I have my access key, I have my secret key and Terraform will read in that file. Standard AWS configuration. We need this has many other configuration modes too. Sorry. Yeah, every provider has its own documentation so you have to read it if you switch providers. Even on, yeah. Using each provider has different resources. It's not exactly exchangeable. If you have scripted your AWS infrastructure, you can't exactly use it for Google Cloud or Azure. But the part about Terraform is you have many, many providers. Azure, you have a HTTP provider. I don't know what that does do. You have a DNS provider to read in. CNames, Bitbucket, Brightbox, InfluxDB if you want. Many cloud providers and service tools. Do we see it? No. Yeah. I was searching for a few DNS providers. I don't see it on the list, but it's nice. Power DNS. If you configure your infrastructure, don't want to use root 53, but you can configure IP addresses into Power DNS. So with Terraform, you're not only configuring one provider, you can configure your infrastructure across many cloud providers, many cloud service tools. Yeah. So let's continue with the introduction. So we have resources. One resource, AWS instance, called MyInstance. I define an AMI image, the instant types, and few tags. You can have variables in Terraform with default, AWS region, AWS AMI. These variables can be read from a TF VAS file or from the comment line or even from environment variables in a special syntax. But to advance in the dog, you don't really need to know this. Yeah. Terraform usage is really simple. You can plan your Terraform run. You can apply your Terraform run and you can destroy it. I don't know. Starting from version 0.10, apply and destroy has the plan integrated. Before that, you always had to run Terraform plan to know what will Terraform do to your infrastructure. But now if you run Terraform apply, it will show you the plan which will be executed and then you can say yes. And it will be applied, it will be destroyed, and so on. The most important part of Terraform is interpolation, which means like I use the variables to interpolate, which is then in 0.12, a first class variable without the quotes and the dollar sign around. I can read in files and many other functions, JSON encoding. No, encoding. Yeah. So I have two resources defined and those two resources, can you see the black lines? Can you see them? Or more hard than them? So yeah. I use these two resources again and this is called interpolation and with this interpolation, Terraform knows which infrastructure has to be graded first. So Terraform will first grade your AWS Keeper and your security group before it is grading your instance because you couldn't grade your instance if the security group doesn't exist. And through this declarative language, Terraform computes which is executed. Some resources are executed in parallel. Some resources like security group IDs can be executed in parallel and you can get an error because you use the same set of IP address and port, for example. But that all depends on your resources you use. So you can have outputs in Terraform. You need the public IP of your instance to find the output and can use it. If you only change your outputs in the Terraform code, you call refresh and then Terraform output, you don't need to apply. And the best part is you can Terraform output public IP and use it in your script, any other script like you need to execute. Yeah. Typically, layout of Terraform, RTF files, yeah, for Terraform. I have a main TF where I define mostly of my infrastructure, local variables, variable state, provider, bit of split up from Terraform infrastructure. So this is how Terraform run looks. I simply run Terraform apply. This is the older recording. It doesn't show you the Terraform plan which is executing. This is from Archimete of fleet configuration where I created this case of smaller Archimete of fleet to execute load testing. So I have called them crew and minions. It's a bit funny. Yeah. Creation complete. Creation complete. And two instances are still grading as its infrastructure, AWS takes its time to create instances. Yeah. And then when I have download testing, I just destroyed this infrastructure. I said old recording, no plan output before. Refreshing the state to see if you already have destroyed your instances by yourself. Could be the case. All this state Terraform knows about is saved in a Terraform TF state file. If you don't configure a backend for the Terraform TF state file, it's saved locally in your Terraform directory where you execute this code. You can configure a remote state too, which is recommendable if you work with teams. Okay. Let's recap that a bit. Terraform is a simply declarative modular tool. It uses many cloud providers and SaaS providers and it's coded infrastructure. Easy peasy, or? Yeah? I said a bit more interact. Okay. Let's introduce the next step because you don't want to grade your infrastructure in one big Terraform configuration. It's not nice. So Terraform has the possibility of using Terraform modules. If Terraform modules, they are like a container for multiple resources. Like the instances I had before, they could be contained in a module or like security group rules could be contained in a model. They create lightweight abstractions because if you create a VPC in AWS, it can get really complicated. So you may want to create or use a module for VPC creation, which abstracts you all the hard bits away. And Terraform modules increase overall usability, which means, okay, you have one VPC, but you may have many security group rules to use them. So Terraform modules can be graded and used in many ways. You can have local modules, like I have in my projects, I have an internal directory, and I have many modules like the data volume, the instance, I have security group rules, this security group rules modules use the security group rule, so you can have modules can use modules. Each module has its own files, main TF, outputs and variables, so input, output and what is executed. Local modules are referenced by the module attributes. You give them any name you want, but use a good name, like SG for security group. Then the source parameter finds which module is referenced. This is a local module, so I go one of directory above, use security group rules, ismp default. Because ismp for AWS has many rules in it, for echo, for fragments, et cetera. Internal modules, like something VPN, you don't want to copy it into every Terraform config. Our VPN configuration is the same for every project. So I have a grid repository, Terraform pixelart VPN, it's versionized, so I can increase, like we have a new subnet in our local office, I just add it to the module, push a new grid tag and every project can use the 1.6 version. But the greatest is the module registry, registry.terraform.io with public modules, verified and unverified modules, so verified modules are verified by Terraform, that they run, that they are good code, still unverified modules are good too. And the best part which is not super for local modules, same version constraints, like for VPC as said, you want to use a module, Terraform, AWS, VPC, AWS, and I want to use every version starting from 2.9, below 3.0. There are simple configs, public subnets, private subnets, database subnets, it's all specific to the module, but it's nice to reuse public modules which are really good. I otherwise I would go insane if I have to write the whole VPC configuration every project. VPCs are not so easy. Yeah. Some recap, Terraform modules are what? Reusable, versatile, repeatable, they provide abstraction and encapsulation, because I can reuse them in every project. It's the same one. The abstraction like the VPC module, it's easy to configure it. So putting them all together, I have to introduce you to new concepts of Terraform, the remote state. As said before, with the Terraform remote state, you can save the state like in S3, like in Azure Block Storage, which is important if you work with team, but you should also use that as a single developer. Because if you split all your modules up, later with TerraGrant, you need the remote state to reference all your outputs of other modules. And I do this with the data Terraform remote state, which is provided by the Terraform provider. The configuration really looks the same. I say the backend and I say the config. And with the Terraform remote state, I have a data usage to read out my outputs of other modules. And these two parts are important for TerraGrant. I like the diken. So grantwork-io-terra-grant, it's on GitHub, it's a simple go tool, a go binary, download it, use it. So what is TerraGrant? It's a thin wrapper for Terraform. So instead of calling Terraform on the console, you call TerraGrant. It keeps your configuration dry. Don't repeat yourself. It allows you to work with multiple modules and, as I said before, it manages your remote state. Let's look how you work with TerraGrant. First, you need an infrastructure modules directory called infrastructure modules or whatever you want to call it. There we have our internal module as seen before. This is not a module, in this internal directory, all my local submodules. And I have many other modules defined. I have a module for RRDB, I have a module for monitoring, a module for Route 53 zone, which can be used multiple times. And in your module, you have the same structure as you would have in the whole Terraform code. Your main TF, which is important, you have your variables, you have your outputs. Because in your module, you need input and output. And the difference to writing a whole big bunch of Terraform code and now split it up in smaller modules, yeah, smaller, encapsulated, lightweight, and reusable. Because if you split it up, I have a RR module, it doesn't have anything to do with your web application node. It's just for your database. So they are lightweight. A difference, the remote state defined in your TerraGrant module and for infrastructure module is just the backend type. The rest is configured by TerraGrant, with the main configuration we will see in the two next steps. So beside your infrastructure module repository, you need a Git repository for that. You can define your infrastructure life directory. This directory structure is how I use it. You are completely free to define your own directory structure. In this case, I use the account name of our AWS account as first level. Then I have the regions. Like in AWS, there are global things without regions like Route 53 zones, IAM, and I have most of my stuff in EU central one. Next level is your environment, dev stage production, and then I have the services. I have decided to use a two level structure. So I have a direct service VPC, I have sorted into services, storage, and below them you have your TerraFarm.TFWars file. Your module is configured. Which module you use, which variables you give into the input. And below then in the directory of the account name, you have your TerraFarm.TFWars for global configuration. Starting from TerraGrant 0.19, it's called TerraFarm.HCL. The global config looks simple. Looks like our remote step before. As kept, we have the TerraGrant global config key, for TerraGrant we now define our remote step there with the backend with the config. Bucket and the dynamic db table locks. It's the same as before, but it's only defined in your one global configuration. So your remote state is already dry. The module config still uses a TerraGrant key too, and the biggest notion is which module you use. You now use the infrastructure modules in your Git and with a reference for your tag. And that's the biggest part. You can versionize your infrastructure. Like this I have a web module at version 1.9.5, which is used in dev stage and production. Now I have any change. Maybe it's a PC breaking change, and I have a version 2.0.0. In dev I can reference this version, run my infrastructure for the web, and stage production is not touched. With this directory structure, you can increase your infrastructure by versions. Yeah, which means you can upgrade your dev. You can later upgrade your stage, as you may be used. But with one look into infrastructure life repository, you know at which version your infrastructure is. You look into the dev web module. Oh, it's a 2.0. You look in your stage web module. Oh, it's still at 1.9. I should upgrade it. You look into your production web module. It's still at 1.9. And luckily I have not upgraded it yet because it's Friday. Maybe you don't want to do a PC break in infrastructure on Friday, but that depends on your workflow. I have include, which is relevant for TerraGround itself, which means finding parent folders finds the root Terra from TF bars, which gets your remote state configuration. That's where you keep your remote state dry. And the coolest part is you can define dependencies. Your web module depends on VPC. It depends on the Aurora storage. It depends on the DB storage, which is later needed if you want to run your whole infrastructure at once. Of course, in your modules, you write your inputs. Like this module has an account and instance type of T3 medium. Yeah. It's pretty simple. And instead of calling Terra from a state, you now call TerraGround. Get, plan, apply, output, every common which Terra from SuperTZ you can call through TerraGround. And you need to because it copies your Terra from infrastructure modules in a temporary cache directory and uses your code and combined with your TF bars. Of course, you want to work locally. You don't want to push every change with attack. That's relatively easy to call TerraGround, apply with TerraGround source. Use your local modules directory. And the most important part is double slash here because infrastructure modules is your directory and this is the sub module in your whole repo. This is needed by Terraform and TerraGround to know exactly which sub module to use and resolve all internally locally used modules correctly. And just to run all the things, you can use TerraGround, apply all, output all, destroy all. In your like, Dev directory, you can go to your stage directory and run all the infrastructure you have defined below this directory. Your web service, your database, your VPC can be graded at once because if I destroy the whole infrastructure because anything is damaged, I don't go into every directory, to every module down and call Terraform destroy. I just use Terraform destroy all, apply all and get maybe one coffee or two coffee, talk a bit with other developers. Depending on the size of your infrastructure, it takes a while. Yeah, keeping your variables dry. If you had noticed, I only have one global config and I hide every module usage. I have a Terraform TFVars, but I have global variables. Like I used the AWS account. I use a PX project name, which I don't want to copy in every configuration. So I move these common variables into the global config file. I create a var file for region specific config and I create a var file for environment specific config. This is excellent supported by TerraGrand. So now I have a configuration for Terraform itself. Like I want to supply extra arguments to every Terraform comment and the comment is called Vars. Which comment, the parameter is called Vars, sorry. And to which comment I want to supply all these comments which need variables like apply, like plan, destroy needs to, but I don't need to notice. TerraGrand knows this and gives it to me. I define few required variable files like the parent Terraform TFVars, which is this same local config, where we are using to and the module Terraform Vars itself. And I set a few optional var variable files like this two for environment and this two for regions. That I have two configuration is just the case because I use a two level folder structure. Like I have to be sealed and I have a services web level. So I have to set them optional and define two levels. TerraGrand is intelligent enough to only use those optional var files which exists. Yeah. This is the resulting layout. So I still have my global config here. Now I have my region and environment variables file. As said before, our services web needs to go two levels up to reach the environment for TFVars and our VPC module has to go only one level up to reach the TFVars. The same for the region. Yeah. It's a really simplistic file layout and now all your variables are dry too because all your variables for the region like which you use and set already put in the region, you put the environment specific variables into NVAR, into NVAR.TFVars file. Now your configuration is dry too. And that should be called TerraGrand. So a recap. You have dry modules, dry Terraform code, a dry remote state and dry CLI flag with it instead of big configurations which you can't really destroy. If you put all your database, all your web server configuration and all your VPC configuration in one big Terraform configuration, you can't call Terraform destroy if you only want to destroy your web nodes. And the biggest part for me is the versionized infrastructure. I can increase my infrastructure code for dev stage and plot separately. And TerraGrand is still just a thin ropper if you don't think about you have to write configuration for TerraGrand. Yeah. Okay. That's all so far. Any questions? I'm sure I missed it. Is TerraGrand also developed by Hashicarp? No. It's developed by Grandwork, which is like a DevOps agency. And they have worked with Terraform and wanted to get their configuration dry, developed some concepts for it, and after they had the concept, they developed TerraGrand as open source project for everyone to use it. Okay. Thanks. Anyone else? Can you give it up, please? So you were basically always playing the changes by hand. Is there also some kind of GitOps way where you basically commit your main config into a repo which is then being applied? I don't like to run Terraform and TerraGrand through CI because then anyone who is allowed to push code to the repo could do terrible things to the infrastructure or just destroy it. It doesn't work. Another thing is all those dependencies. The web service depends on the storage and so on. I have put it into CI, but it always makes problems. So I have done this and I don't like it. I prefer to run it by myself because I have the plan output. I need to know which infrastructure is modified, is destroyed, is graded. There are a few things like in AWS if the aim, I change this for the instance, Terraform wants to regrade it. I don't want to regrade it. I don't want to regrade the instance because it may be the production instance. So it's also a bit of a thing about trust in your developers and other DevOps people. For our organizations, I am the only one which is doing DevOps and all the others are simple developers and it's a matter of trust too. I don't trust them with infrastructure. I have another question. How often do you run into configuration problems? If I see it right, any kind of what I would like to configure on the systems need to be provided by a module or provider in Terraform and how stable is this? Or is it, I want to be dry so I like to steer away from complicated configuration, hotfixes, workarounds, whatever. What's your impression with Terraform over the years you use it? Terraform is really stable. All tough, it uses zero versions. It's really stable. They made a split from getting the providers out of the core so the providers evolve separately like new resources for AWS are added separately, have their own versioning, have their own PC breaks with major versions. Luckily. So it's really stable. New resources are added fast. These problems you mostly get because you just need to know, like if you use AWS, you need to know AWS. Like you can't have the same, can't have two or three buckets with the same name. You just need to know this. If you re-grade the three-bucket, you either use name prefix or you have to first destroy it. The other configuration changes in Terraform is to destroy the resource first and then re-grade it. You can change the life cycle for it like great before destroy. And if you have something like great before destroy for a three-bucket with the same name, it won't work out because Terraform then wants to create a three-bucket with the same name first and AWS says, nay. So the most configuration problems I have because I didn't notice restriction on the AWS side. Anyone else? Okay. It was a bit too quick today. But yeah. We could break for you. Thank you. Thank you.
 Let's begin.  I will show you Terraform and Terragrant, two excellent tools for infrastructure automation.  So hi at first.  I love to have my lectures more interactive.  So if anything is unclear, just ask in between or maybe I give you some questions.  And I speak German and English.  I do this talk in English because you wouldn't understand me in German with my dialect.  I'm from Austria.  So you can ask any question in English or in German.  That's no problem.  Good.  I'm Emily.  Just for my comfortness, I'm a non-binary person.  My pronouns are in English, sei and sie or in German, sie or sie, if you talk about me.  So just know them.  Yeah.  I work at Pixelat.  We are an digital agency and my job title is called engineering and operations.  It's, yeah.  My job title is basically I do DevOps, many DevOps system administration, cloud administration,  internal development, that's the engineering part.  We could have called it DevOps.  Yeah.  Wayne.  So my everyday work is getting up infrastructure, modify them, destroy them, getting up, destroy  them, getting up, destroy them.  So I don't want to do this by hand.  So I have searched for tools, have learned these tools and I always use them.  Even if I have to get up one machine, I code it.  Because if any developer destroys something on this machine, I just take it down and re-graded  it by script.  Yeah.  A little disclaimer.  These code examples are written for Terraform below or equals 0.11 and TerraGrant below  or equals 0.18.  DevOps tools have released a new version with a new configuration language which is a bit  different, has a first class level of variables and so on.  So if you take these code examples and use the newest Terraform version, 0.12, you have  to adapt them a bit.  So the status quo.  Who is automating his infrastructure?  The rest of you have to do something with infrastructure?  Yeah, no.  The other person's developers.  Okay.  Yeah.  So there's this famous model, pets versus cattle.  Let's look a bit into it as an intro.  So the pet service model.  When you have a pet at home, what do you do with it?  Pets have unique names.  Like, my notebook is called, if I remember correctly, one of the four rivers of, yeah,  one of the four rivers of Greek mythology.  And your pet, they are cared, you like lovingly hand raised and cared for.  Like your cat at home, you care for your cat.  And if you have a server as a pet, you raise it by hand too.  You install your packages by hand on the shell, you create your configuration by hand.  Yeah.  And when they get ill, you nurse them back to health.  Like you have some configuration error, you fix it by hand, you get it running back, all  by your hand, like, don't worry, I'm from tech support.  On the other side, you have the cattle service model, like all your servers, all your cattle  are given numbers, like MGNL, 05, 06, 07, et cetera.  They are almost identical to other cattle, of course, because they are not hand raised,  you automate them to create them.  And when they get ill, you get another one.  Like I said before, I destroy them, I regrade it.  I just don't give any mind if something is in bad configuration state, if something is  destroyed by our developers, I just destroy the whole machine and regrade it.  There are a few exceptions, like your magical unicorn pet server, which can still be under  configuration management.  You have projects, we have only one server because, yeah, enterprise thingy and license  costs of 10,000 euros, you have that.  So you have this unique pet, but it's still graded by configuration management, infrastructure  management, but I can't destroy it, because it's the only production server running, I  still have to fix some things by hand.  Yeah.  So we have infrastructure as code.  This is the cattle service model or your special pet.  Infrastructure is code.  Who has never heard about this term?  Good.  Just repeat it.  Infrastructure is code is reproducible.  It's plain defined infrastructure because you have code.  It's defined, it has a state, it should have a state, and it's a foundation for DevOps.  You can't do DevOps stuff without coded infrastructure.  In few servers, all raised by hand, all configured by hand.  One is different because one of the system engineers hasn't read the wiki exactly and  missed a step.  So infrastructure is code is a really important foundation for DevOps.  And you gain visibility across your infrastructure because you just look in your infrastructure  code, you know, oh, I have 10 servers, or I use this image, I use these packages.  And for all of this, I use Terraform.  Yeah, it's really Terraform.io.  You can download it.  Really great documentation.  And the documentation is really, really great.  I have used tools where the docs are, yeah, in a really bad state, or you don't, or stick  overflow of the documentation.  No, so Terraform.  It's made by HashiCorp, who has heard of HashiCorp?  Yeah, they make, they make PECA, console, yeah.  The nicest part about Terraform, it has a declarative language.  If you compare it to Chef or Ansible, which have a procedural style of infrastructure coding,  Terraform is declarative like puppet.  You define your resources, and they define the order of execution by themselves.  And the HashiCorp configuration language is like a superset of JSON.  If you paste JSON in your HashiCorp file, you still have to modify the syntax.  But the nice part about it, as a superset of JSON, you can write your Terraform config  in JSON too, which I use in a tool to automatically create infrastructure for tenants.  I use JSON, scripted by Terraform, and it's done.  Projects erased up and down by simple API calls.  Yeah.  Let's, who of you has never used Terraform?  Perfect.  So I give you some introduction about Terraform.  Terraform has the notation of providers.  In this case, I use the provider AWS.  Yeah, we use AWS, I'm used to it.  So I use this in my examples.  In the next talk in this lecture room, you get a Terraform talk which uses Kubernetes.  Yeah, cool.  So a great defundation for you.  Yeah.  I have a provider, I use variables, AWS access key and secret key.  I can provide the keys on the comment line.  Don't use static credentials on the comment line.  So I can configure the provider only with the region.  Can supply the secrets key by environment variables.  Every provider has a good documentation which environment variables are read in instead  of the configuration parameters or that's not really better.  So it depends on the provider how you can configure it.  Like AWS provider, you have your dot AWS credentials file which is also read by Terraform.  So I have my access key, I have my secret key and Terraform will read in that file.  Standard AWS configuration.  We need this has many other configuration modes too.  Sorry.  Yeah, every provider has its own documentation so you have to read it if you switch providers.  Even on, yeah.  Using each provider has different resources.  It's not exactly exchangeable.  If you have scripted your AWS infrastructure, you can't exactly use it for Google Cloud  or Azure.  But the part about Terraform is you have many, many providers.  Azure, you have a HTTP provider.  I don't know what that does do.  You have a DNS provider to read in.  CNames, Bitbucket, Brightbox, InfluxDB if you want.  Many cloud providers and service tools.  Do we see it?  No.  Yeah.  I was searching for a few DNS providers.  I don't see it on the list, but it's nice.  Power DNS.  If you configure your infrastructure, don't want to use root 53, but you can configure  IP addresses into Power DNS.  So with Terraform, you're not only configuring one provider, you can configure your infrastructure  across many cloud providers, many cloud service tools.  Yeah.  So let's continue with the introduction.  So we have resources.  One resource, AWS instance, called MyInstance.  I define an AMI image, the instant types, and few tags.  You can have variables in Terraform with default, AWS region, AWS AMI.  These variables can be read from a TF VAS file or from the comment line or even from  environment variables in a special syntax.  But to advance in the dog, you don't really need to know this.  Yeah.  Terraform usage is really simple.  You can plan your Terraform run.  You can apply your Terraform run and you can destroy it.  I don't know.  Starting from version 0.10, apply and destroy has the plan integrated.  Before that, you always had to run Terraform plan to know what will Terraform do to your  infrastructure.  But now if you run Terraform apply, it will show you the plan which will be executed and  then you can say yes.  And it will be applied, it will be destroyed, and so on.  The most important part of Terraform is interpolation, which means like I use the variables to interpolate,  which is then in 0.12, a first class variable without the quotes and the dollar sign around.  I can read in files and many other functions, JSON encoding.  No, encoding.  Yeah.  So I have two resources defined and those two resources, can you see the black lines?  Can you see them?  Or more hard than them?  So yeah.  I use these two resources again and this is called interpolation and with this interpolation,  Terraform knows which infrastructure has to be graded first.  So Terraform will first grade your AWS Keeper and your security group before it is grading  your instance because you couldn't grade your instance if the security group doesn't exist.  And through this declarative language, Terraform computes which is executed.  Some resources are executed in parallel.  Some resources like security group IDs can be executed in parallel and you can get an  error because you use the same set of IP address and port, for example.  But that all depends on your resources you use.  So you can have outputs in Terraform.  You need the public IP of your instance to find the output and can use it.  If you only change your outputs in the Terraform code, you call refresh and then Terraform  output, you don't need to apply.  And the best part is you can Terraform output public IP and use it in your script, any other  script like you need to execute.  Yeah.  Typically, layout of Terraform, RTF files, yeah, for Terraform.  I have a main TF where I define mostly of my infrastructure, local variables, variable  state, provider, bit of split up from Terraform infrastructure.  So this is how Terraform run looks.  I simply run Terraform apply.  This is the older recording.  It doesn't show you the Terraform plan which is executing.  This is from Archimete of fleet configuration where I created this case of smaller Archimete  of fleet to execute load testing.  So I have called them crew and minions.  It's a bit funny.  Yeah.  Creation complete.  Creation complete.  And two instances are still grading as its infrastructure, AWS takes its time to create  instances.  Yeah.  And then when I have download testing, I just destroyed this infrastructure.  I said old recording, no plan output before.  Refreshing the state to see if you already have destroyed your instances by yourself.  Could be the case.  All this state Terraform knows about is saved in a Terraform TF state file.  If you don't configure a backend for the Terraform TF state file, it's saved locally  in your Terraform directory where you execute this code.  You can configure a remote state too, which is recommendable if you work with teams.  Okay.  Let's recap that a bit.  Terraform is a simply declarative modular tool.  It uses many cloud providers and SaaS providers and it's coded infrastructure.  Easy peasy, or?  Yeah?  I said a bit more interact.  Okay.  Let's introduce the next step because you don't want to grade your infrastructure in one big  Terraform configuration.  It's not nice.  So Terraform has the possibility of using Terraform modules.  If Terraform modules, they are like a container for multiple resources.  Like the instances I had before, they could be contained in a module or like security  group rules could be contained in a model.  They create lightweight abstractions because if you create a VPC in AWS, it can get really  complicated.  So you may want to create or use a module for VPC creation, which abstracts you all  the hard bits away.  And Terraform modules increase overall usability, which means, okay, you have one VPC, but you  may have many security group rules to use them.  So Terraform modules can be graded and used in many ways.  You can have local modules, like I have in my projects, I have an internal directory,  and I have many modules like the data volume, the instance, I have security group rules,  this security group rules modules use the security group rule, so you can have modules  can use modules.  Each module has its own files, main TF, outputs and variables, so input, output and what is  executed.  Local modules are referenced by the module attributes.  You give them any name you want, but use a good name, like SG for security group.  Then the source parameter finds which module is referenced.  This is a local module, so I go one of directory above, use security group rules, ismp default.  Because ismp for AWS has many rules in it, for echo, for fragments, et cetera.  Internal modules, like something VPN, you don't want to copy it into every Terraform  config.  Our VPN configuration is the same for every project.  So I have a grid repository, Terraform pixelart VPN, it's versionized, so I can increase,  like we have a new subnet in our local office, I just add it to the module, push a new grid  tag and every project can use the 1.6 version.  But the greatest is the module registry, registry.terraform.io with public modules, verified and unverified  modules, so verified modules are verified by Terraform, that they run, that they are  good code, still unverified modules are good too.  And the best part which is not super for local modules, same version constraints, like for  VPC as said, you want to use a module, Terraform, AWS, VPC, AWS, and I want to use every version  starting from 2.9, below 3.0.  There are simple configs, public subnets, private subnets, database subnets, it's all  specific to the module, but it's nice to reuse public modules which are really good.  I otherwise I would go insane if I have to write the whole VPC configuration every project.  VPCs are not so easy.  Yeah.  Some recap, Terraform modules are what?  Reusable, versatile, repeatable, they provide abstraction and encapsulation, because I can  reuse them in every project.  It's the same one.  The abstraction like the VPC module, it's easy to configure it.  So putting them all together, I have to introduce you to new concepts of Terraform, the remote  state.  As said before, with the Terraform remote state, you can save the state like in S3, like in  Azure Block Storage, which is important if you work with team, but you should also use  that as a single developer.  Because if you split all your modules up, later with TerraGrant, you need the remote  state to reference all your outputs of other modules.  And I do this with the data Terraform remote state, which is provided by the Terraform provider.  The configuration really looks the same.  I say the backend and I say the config.  And with the Terraform remote state, I have a data usage to read out my outputs of other  modules.  And these two parts are important for TerraGrant.  I like the diken.  So grantwork-io-terra-grant, it's on GitHub, it's a simple go tool, a go binary, download  it, use it.  So what is TerraGrant?  It's a thin wrapper for Terraform.  So instead of calling Terraform on the console, you call TerraGrant.  It keeps your configuration dry.  Don't repeat yourself.  It allows you to work with multiple modules and, as I said before, it manages your remote  state.  Let's look how you work with TerraGrant.  First, you need an infrastructure modules directory called infrastructure modules or  whatever you want to call it.  There we have our internal module as seen before.  This is not a module, in this internal directory, all my local submodules.  And I have many other modules defined.  I have a module for RRDB, I have a module for monitoring, a module for Route 53 zone,  which can be used multiple times.  And in your module, you have the same structure as you would have in the whole Terraform code.  Your main TF, which is important, you have your variables, you have your outputs.  Because in your module, you need input and output.  And the difference to writing a whole big bunch of Terraform code and now split it up  in smaller modules, yeah, smaller, encapsulated, lightweight, and reusable.  Because if you split it up, I have a RR module, it doesn't have anything to do with your web  application node.  It's just for your database.  So they are lightweight.  A difference, the remote state defined in your TerraGrant module and for infrastructure  module is just the backend type.  The rest is configured by TerraGrant, with the main configuration we will see in the  two next steps.  So beside your infrastructure module repository, you need a Git repository for that.  You can define your infrastructure life directory.  This directory structure is how I use it.  You are completely free to define your own directory structure.  In this case, I use the account name of our AWS account as first level.  Then I have the regions.  Like in AWS, there are global things without regions like Route 53 zones, IAM, and I have  most of my stuff in EU central one.  Next level is your environment, dev stage production, and then I have the services.  I have decided to use a two level structure.  So I have a direct service VPC, I have sorted into services, storage, and below them you  have your TerraFarm.TFWars file.  Your module is configured.  Which module you use, which variables you give into the input.  And below then in the directory of the account name, you have your TerraFarm.TFWars for global  configuration.  Starting from TerraGrant 0.19, it's called TerraFarm.HCL.  The global config looks simple.  Looks like our remote step before.  As kept, we have the TerraGrant global config key, for TerraGrant we now define our remote  step there with the backend with the config.  Bucket and the dynamic db table locks.  It's the same as before, but it's only defined in your one global configuration.  So your remote state is already dry.  The module config still uses a TerraGrant key too, and the biggest notion is which module  you use.  You now use the infrastructure modules in your Git and with a reference for your tag.  And that's the biggest part.  You can versionize your infrastructure.  Like this I have a web module at version 1.9.5, which is used in dev stage and production.  Now I have any change.  Maybe it's a PC breaking change, and I have a version 2.0.0.  In dev I can reference this version, run my infrastructure for the web, and stage production  is not touched.  With this directory structure, you can increase your infrastructure by versions.  Yeah, which means you can upgrade your dev.  You can later upgrade your stage, as you may be used.  But with one look into infrastructure life repository, you know at which version your  infrastructure is.  You look into the dev web module.  Oh, it's a 2.0.  You look in your stage web module.  Oh, it's still at 1.9.  I should upgrade it.  You look into your production web module.  It's still at 1.9.  And luckily I have not upgraded it yet because it's Friday.  Maybe you don't want to do a PC break in infrastructure on Friday, but that depends on your workflow.  I have include, which is relevant for TerraGround itself, which means finding parent folders  finds the root Terra from TF bars, which gets your remote state configuration.  That's where you keep your remote state dry.  And the coolest part is you can define dependencies.  Your web module depends on VPC.  It depends on the Aurora storage.  It depends on the DB storage, which is later needed if you want to run your whole infrastructure  at once.  Of course, in your modules, you write your inputs.  Like this module has an account and instance type of T3 medium.  Yeah.  It's pretty simple.  And instead of calling Terra from a state, you now call TerraGround.  Get, plan, apply, output, every common which Terra from SuperTZ you can call through TerraGround.  And you need to because it copies your Terra from infrastructure modules in a temporary  cache directory and uses your code and combined with your TF bars.  Of course, you want to work locally.  You don't want to push every change with attack.  That's relatively easy to call TerraGround, apply with TerraGround source.  Use your local modules directory.  And the most important part is double slash here because infrastructure modules is your  directory and this is the sub module in your whole repo.  This is needed by Terraform and TerraGround to know exactly which sub module to use and  resolve all internally locally used modules correctly.  And just to run all the things, you can use TerraGround, apply all, output all, destroy  all.  In your like, Dev directory, you can go to your stage directory and run all the infrastructure  you have defined below this directory.  Your web service, your database, your VPC can be graded at once because if I destroy  the whole infrastructure because anything is damaged, I don't go into every directory,  to every module down and call Terraform destroy.  I just use Terraform destroy all, apply all and get maybe one coffee or two coffee, talk  a bit with other developers.  Depending on the size of your infrastructure, it takes a while.  Yeah, keeping your variables dry.  If you had noticed, I only have one global config and I hide every module usage.  I have a Terraform TFVars, but I have global variables.  Like I used the AWS account.  I use a PX project name, which I don't want to copy in every configuration.  So I move these common variables into the global config file.  I create a var file for region specific config and I create a var file for environment specific  config.  This is excellent supported by TerraGrand.  So now I have a configuration for Terraform itself.  Like I want to supply extra arguments to every Terraform comment and the comment is called  Vars.  Which comment, the parameter is called Vars, sorry.  And to which comment I want to supply all these comments which need variables like apply,  like plan, destroy needs to, but I don't need to notice.  TerraGrand knows this and gives it to me.  I define few required variable files like the parent Terraform TFVars, which is this  same local config, where we are using to and the module Terraform Vars itself.  And I set a few optional var variable files like this two for environment and this two  for regions.  That I have two configuration is just the case because I use a two level folder structure.  Like I have to be sealed and I have a services web level.  So I have to set them optional and define two levels.  TerraGrand is intelligent enough to only use those optional var files which exists.  Yeah.  This is the resulting layout.  So I still have my global config here.  Now I have my region and environment variables file.  As said before, our services web needs to go two levels up to reach the environment  for TFVars and our VPC module has to go only one level up to reach the TFVars.  The same for the region.  Yeah.  It's a really simplistic file layout and now all your variables are dry too because all  your variables for the region like which you use and set already put in the region, you  put the environment specific variables into NVAR, into NVAR.TFVars file.  Now your configuration is dry too.  And that should be called TerraGrand.  So a recap.  You have dry modules, dry Terraform code, a dry remote state and dry CLI flag with it  instead of big configurations which you can't really destroy.  If you put all your database, all your web server configuration and all your VPC configuration  in one big Terraform configuration, you can't call Terraform destroy if you only want to  destroy your web nodes.  And the biggest part for me is the versionized infrastructure.  I can increase my infrastructure code for dev stage and plot separately.  And TerraGrand is still just a thin ropper if you don't think about you have to write  configuration for TerraGrand.  Yeah.  Okay.  That's all so far.  Any questions?  I'm sure I missed it.  Is TerraGrand also developed by Hashicarp?  No.  It's developed by Grandwork, which is like a DevOps agency.  And they have worked with Terraform and wanted to get their configuration dry, developed  some concepts for it, and after they had the concept, they developed TerraGrand as open  source project for everyone to use it.  Okay.  Thanks.  Anyone else?  Can you give it up, please?  So you were basically always playing the changes by hand.  Is there also some kind of GitOps way where you basically commit your main config into  a repo which is then being applied?  I don't like to run Terraform and TerraGrand through CI because then anyone who is allowed  to push code to the repo could do terrible things to the infrastructure or just destroy  it.  It doesn't work.  Another thing is all those dependencies.  The web service depends on the storage and so on.  I have put it into CI, but it always makes problems.  So I have done this and I don't like it.  I prefer to run it by myself because I have the plan output.  I need to know which infrastructure is modified, is destroyed, is graded.  There are a few things like in AWS if the aim, I change this for the instance, Terraform  wants to regrade it.  I don't want to regrade it.  I don't want to regrade the instance because it may be the production instance.  So it's also a bit of a thing about trust in your developers and other DevOps people.  For our organizations, I am the only one which is doing DevOps and all the others are simple  developers and it's a matter of trust too.  I don't trust them with infrastructure.  I have another question.  How often do you run into configuration problems?  If I see it right, any kind of what I would like to configure on the systems need to be  provided by a module or provider in Terraform and how stable is this?  Or is it, I want to be dry so I like to steer away from complicated configuration, hotfixes,  workarounds, whatever.  What's your impression with Terraform over the years you use it?  Terraform is really stable.  All tough, it uses zero versions.  It's really stable.  They made a split from getting the providers out of the core so the providers evolve separately  like new resources for AWS are added separately, have their own versioning, have their own  PC breaks with major versions.  Luckily.  So it's really stable.  New resources are added fast.  These problems you mostly get because you just need to know, like if you use AWS, you  need to know AWS.  Like you can't have the same, can't have two or three buckets with the same name.  You just need to know this.  If you re-grade the three-bucket, you either use name prefix or you have to first destroy  it.  The other configuration changes in Terraform is to destroy the resource first and then  re-grade it.  You can change the life cycle for it like great before destroy.  And if you have something like great before destroy for a three-bucket with the same name,  it won't work out because Terraform then wants to create a three-bucket with the same name  first and AWS says, nay.  So the most configuration problems I have because I didn't notice restriction on the  AWS side.  Anyone else?  Okay.  It was a bit too quick today.  But yeah.  We could break for you.  Thank you.  Thank you.
2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_0_179.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_180_2123.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_2124_3815.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_3816_4607.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_4608_6011.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_6012_6131.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_6132_7091.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_7092_8195.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_8196_9683.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_9684_9839.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_9840_9959.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_9960_10247.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_10248_10367.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_10368_12299.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_12300_13703.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_13704_16595.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_16596_21035.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_21036_21455.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_21456_22679.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_22680_22751.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_22752_23051.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_23052_23159.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_23160_23231.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_23232_23807.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_23808_23879.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_23880_24683.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_24684_24803.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_24804_27179.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_27180_27755.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_27756_30131.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_30132_30719.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_30720_32795.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_32796_33335.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_33336_38339.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_38340_39731.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_39732_40607.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_40608_40799.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_40800_41531.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_41532_43319.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_43320_44915.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_44916_44987.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_44988_47507.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_47508_50015.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_50016_50171.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_50172_50507.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_50508_51287.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_51288_51815.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_51816_51923.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_51924_52247.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_52248_53183.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_53184_53531.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_53532_57443.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_57444_57563.png 2019_-_Automated__modularized_and_versioned_infras_sOK-dJ20u04_57564_57945.png