Hacking Terraform for fun and profit
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Babenko, Anton
Using Terraform is often simple, extending it to do what YOU want, can be challenging (or “impossible”). Want to manage unsupported resources? Maintain lots of resources? Integrate non-integrable? The talk is an advanced guide about HOW to extend, integrate and execute Terraform to get things DONE. If you’ve been using Terraform just by following the official documentation, you are not getting all from it. As soon as one cloud provider announces a new service or a feature, you dream that Terraform has zero-day support for it. Well, it is not always like this, and I will show what we can do about it. Are you using Terraform and keep asking yourself why I should copy-paste so much? What if you need to manage more than a dozen resources with Terraform (e.g., hundreds of GitHub repositories with permissions, or hundreds of IAM users and their permissions)? How can I use Terraform to manage absolutely ANY type of resource? What is beyond Terraform modules? What is a really dynamic module and what Terraform 0.12 will help us with? Let's see the advanced and very unusual solutions of how Terraform can be extended, integrated, executed, or merely hacked to get the job done with the help of external open-source services and integrations.
 Tell us all the things you should never do at Terraform. Thank you. Okay. Now you can hear me. Anyway, thank you, Walter, for the introduction. And yes, in fact, there will be some things which you should not do, but at least you should be aware that they exist. So maybe you have this actual situation when you should be doing it. So a couple words. Oh my God. Of course, it never works in this room. So this time without clicker again. What I do, I'm staying through the active in open source community and I'm writing and maintaining a lot of open source tools related to Terraform, AWS. And I became AWS community hero because I actually do something in the community giving talks and trying to figure out how to make usage of AWS and Terraform. First question to all of you. Please raise your hand if you are using Terraform AWS modules. Excellent. Oh my God. That's actually much more. So it means that some people went to previous year talk as well because in previous year I talked about Terraform best practices and how to use it properly. And honestly, I am tired of using it the correct way. So this time we will look into how to not use or how you can use and this should give you some information about possibilities. And yeah, what I do, I actually do everything related to Terraform, consulting and providing different services, running workshops, trainings and I also like mentorship. I mean, I'm exposed to a lot of different codes and solutions which I try to aggregate and sometimes talk about them. You can see my blog where I write exclusively about Terraform and AWS on AntonDabianka.com. And yeah, this is stuff which most of you, as I understand, are using which is pretty cool. There were 7 million downloads so far and I'm sure that amount will be just increasing because come on, so far just 1,000 pull requests and issues resolved. So we need more issues, more pull requests, more users and more sponsors. So there is a link. Another project which I'm involved is CloudCraft which is opinionated and pretty good way of drawing AWS diagrams in the browser. I have 25 minutes and I have 66 slides left so we have to go fast, okay? What I'm working on is that once you visualize something in the browser, you don't have to repeat yourself when you actually implement this as code. So ModulesTF is an open source project which I work on. You can go and use it for free and see what kind of code and structure and tools are actually used. So infrastructure. Infrastructure is all what is necessary to run application. It's all kind of computing, networking and other requirements which we have to maintain. So this is my favorite definition of infrastructure despite the fact that this is not coming from Wikipedia. Terraform. I'm not going to ask who use Terraform because otherwise you would go to another talk. Plan. Disclaimer. Everything what I'm going to talk will probably look, oh my God, why do you even show this? I don't really care because this is disclaimer, okay? And there are a lot of things, a lot of features of Terraform which exist in documentation, in users' code, but it doesn't mean that they are good. For example, Terraform workspaces. And there will be some useful parts as well in the talk. I put them in the beginning so that you actually get something useful first and then you figure out that, oh my God, Terraform can be used as a hammer because everything except, I don't know, is a nail, right? So let's use Terraform for absolutely crazy situations. And then we'll look into how Terraform 012 actually helps and what does it do and how cool is it. So first add, of course, let's make Terraform with AWS faster for free with SMS and registrations, right? Who doesn't like to have it? So please use this one. You will be surprised that if you specify in the provider that you can skip credential validation, check which region you are using, meta, data, API check and all of this, you will have, in most cases, you will have at least one, so you will have one instead of 10 API calls. And if you are working on a slow network like an airplane, this helps a lot. You can work with Terraform offline using these hacks. And this is document, so this is not hack. This is useful, okay? A little bit more practical stuff. Versioning and secrets. I mean, everybody likes to deal with secrets, I know, so we'll talk about that. But first versioning. So versioning in Terraform exists in many places and many flavors. We have Terraform Core, which obviously has to be version 012.samsin providers, many from the community. Now you can get them from the registry and a lot of them are in private because nobody wants to publish them. And also there are modules which are more obvious way and more obvious things to be versioned. Most of them are in registry, some of them are private, and so on. So when we are talking about versioning in Terraform of Terraform Core itself, there are not many tools which are useful and helpful. For example, if you need to run Terraform on your local machine, DFN is one of these tools which you run and switch similar as you do with RubyN, for example. If you have to specify version of Terraform as well as version of providers, that's really good thing that you're starting to look at your code, something that will actually leave longer than you quit, right? So you're worried about future. And the point of versioning of Terraform providers is that when you specify your dependencies and they will be combined later on and you cannot override them from the parent modules which you use. That's why good Terraform modules are never using provider block inside of module itself. They're always instantiated at the highest level. And there are many tools which try to do similar stuff for modules. Honestly, all of them failed, in my opinion, even the one which I'm talking about now, because it worked on October and now it doesn't work. The point is that it is really hard to make versioning correct. For example, as it says here, support ranges for modules from Terraform registry seems to be the case if the latest digit is less than 9. 10 is not more than 9 according to these two. So I was surprised. And overall, the flow looks like that is that we first set up version for the Terraform, then we figure out which version of modules we need to do, and then we use Terraform init to download providers which satisfy modules requirements which we have. And then we use Terraform as normal, and Terraform plan, apply, destroy. If you are into automation and CI CD things, you may look into the PandaBot, which still has ongoing issue related to Terraform 012 syntax. But from what I understand, it will be resolved sooner or later because amount of disappointed people is growing very fast. You can go there and upvote it yourself. Secrets in Terraform. It's still a problem with Terraform 012, not a big surprise. Terraform Core has not done everything what is necessary, but let's see what we can do. So first of all, there are two types of secrets which we are talking about. The first one is inside of plan, apply, and output. And then the second one, which is still no changes at all, is Terraform state. So whatever you put will be in state. First of all, when we specify that we want to output password, we may specify that we want to use sensitive through, then it will be written as sensitive. Pretty good and pretty helpful sometimes. If we want to use actually a random string and we want to generate this random password, and then we use it somewhere else, quite recently there was a new resource which is called random password. And the cool part of it is that it does not output ID of the password itself as console output, which means that we can run it in CI and it will not be available for anyone else. So check out random password resource. And overall, the solutions are very specific to the provider. And if we are talking about AWS, there are a lot of options related to the provider implementation of how to deal with secrets and how to encrypt them somewhere. It is a pretty big topic in general. So if you have some money, let's say from zero to $70, you may get benefit from Terraform Cloud and you will be able to have secrets managed using Terraform Cloud and your state file will be also stored on Terraform Cloud. So you actually will not have so much problem with secrets, but this is only if you have some money. If you don't have money, you can use tool like Secret Hub. I think it is one of the best tool out there. I am surprised that not many people are talking about it, including employees. I don't know why, but I am telling you that this is cool. So check it out. It has support for TF-Rs, TF-States. It is another resource, it is another provider in Terraform, which you have to include and it will be dealing with secrets. So Secret Hub is documentation. So overall, versioning is doable. Secret is still a problem. Secret Hub looks good. That was the end of useful information. Now let's look about what is actually hammer. Hammer looks cool. We know that Terraform is powerful. Let's use for pretty much anything. What means anything? We don't care about resources, but don't care about state. What does infrastructure glue Terraform mean? How to manage any resource on AWS using Terraform? How to do application deployment? How to actually use dynamic Terraform modules? What does actually Sheem means? So let's look in the first problem. How to create resources and don't worry about their state? If you are coming from infrastructure, like infrastructure management, you get used to there is resource, there is configuration or definition, there is state and we have to take care about this TF-State file. We don't care for it because we don't want to lose it. It's really sensitive and it's all these scary things. In most cases, it's right. What about if we want to use Terraform for just write-only operations? Or for example, we have some tooling which allow us to generate Terraform configuration files, and we want to manage data which we populate console key value. Or we just want to upload and overwrite any amount of files on S3 or any other reason. There are a lot of different reasons. I'm putting this as an example. Yes, there is a backend type called inMEM. Unfortunately, I cannot put link to documentation because documentation doesn't exist and nobody wants you to use it. I encourage you to check this out. It's really helpful if you are creating something what you don't want to worry about. For example, it's your test environment which you certainly will never touch anymore. That's a great example of using inMEM. It will make your work with Terraform faster. Of course, there is no state, so every time there will be creation of resources, there is no way to change them, but in many cases it helps. Infrastructure glue Terraform. Well, some time ago we figured out that a lot of APIs exist. Terraform has a unified way to manage all of them. Like, you can manage resources on your file system to create them, to read, create archives, all different scripts, CLIs, REST API, make HTTP calls, and many, many different things. I see not many people are actually taking it seriously because what it gives you is you have a way to interact with these APIs or with these providers in very standardized way. For example, this code shows that we can connect different type of resources and create files, make an archive for them, and upload this archive into S3 bucket. So first of all, we use features of Terraform 0.12 to iterate through lists and make lists of users, for example. And then we make files in specific location on our file system. Then we make an archive from that directory, and we upload this archive into S3 bucket. So all of this is 24 lines, which of course can be updated, and you can use CLI or AWS CLI, or you can use any other tool, or you can write shell script in one line. But the point is that of this talk is not to show that, hey, the same you can do in better way. No, Terraform can do this. So how to manage any type of AWS resource? If you are coming from AWS, you know cloud formation, right? Many people know cloud formation. Okay, cloud formation does not support a lot. It doesn't support everything. Terraform does not support everything either, but in total, they support pretty much everything. If not everything, then you have two options. You have to open pull request to AWS provider, and then wait, and wait, and wait. Meanwhile, you have to use your own fork. And I'm quite sure that there are people who open pull request to Terraform AWS provider, and you are still waiting for pull request to be merged, right? Yeah, I can see some faces. So yes, it takes some time. It takes some time, and meanwhile, you have to use your own fork. But even the worst situation is when you have some stuff developed in cloud formation, because that's defaulting in AWS back then, even still now. And you want to move to something else, but you were told that, oh, you cannot delete this database, or S3 bucket. So what can you do so that you have to use better tools, but you at the same time cannot recreate resources? This is where the requirement is. If you could create and delete, that's easy. So the solution for that one is that you first create cloud formation stack, and you specify deletion policy retain. So you have cloud formation stack where deletion policy retain means that this resource will not be deleted after cloud formation stack is deleted. This means that it's not managed by anything else, literally. Then you delete cloud formation stack, and you import this retained resource to Terraform. It looks like this is that you have to write code describing these resources, like in this case S3 bucket, and you have to import it, and you have to see that plan is happier because you customize the resource configuration, and there is nothing else to change. So what you have now is that resource is managed using Terraform, and it does not have any relation with cloud formation anymore. And if there are some situations which you cannot manage using cloud formation or using Terraform, you can still use null resource and AWS CLI. AWS CLI still supports significant amount of resources, even registration domains, if you want. And yeah, of course, it's never too late. So application deployment. I have a little bit of time, and I want to ask you, what was before? I mean, on this conference, you've heard about immutable infrastructure more than once. You all know what is Packer, Docker, Schmocker. There are many different tools out there. But what was before? Like, who tell, like, how did you do before all of this? Let's say maybe in some places you still do it, like I did, and I still do it for some of projects. So what is, like, anyways, how you do this? Don't mention anything fancy, like immutable. Of course, no Kubernetes, please. No Packer, Docker. What? Yeah, SCP. SCP is pretty cool. Yeah, SCP is supported quite successfully. That's too boring, I would say. What about this one? Terraform and GitPool. Isn't it cool that you can use Terraform to actually use remote execution to connect to an instance, which is not managed, of course, anywhere, and you run GitPool on that machine? Isn't it future? Hopefully not, but still, you can describe this resource. You are kind of one way into 21st century, and then you say, oh, no, now I need to redeploy this. So you run Terraform, you obtain that resource, and you run the play, apply again, and then you run it again. I think it's brilliant because you are almost there. You start to understand that, oh, yeah, there is a gap between this process, but Terraform can bridge them. So, and one of the most complicated feature, or not even feature, but scenario, I would say how to use Terraform for things which kind of was not designed for. And this is called, like in the web browser, it's called Polyfill, where Polyfill is code that implements a feature of the browser that do not support the feature. In terms of Terraform, we all wanted to have something what Terraform, if we go to Terraform, open issues, and we see that there are hundreds of upvotes or even some disappointed customers say, oh, this feature is not supported, we cannot use Terraform because it's too bad. What can we do for them? For example, for rich with modules, like many people wanted to use it, I'm sure, because I go to GitHub issues and I see so many upvotes, or even more like historically, people wanted to have reusable module, but they wanted to have different values inside of ignore changes. For example, an auto scaling group. I still have debate between people who say, oh, we don't have to specify a minimum, maximum and desired capacity as required values because we wanted to have up and down, like scale in and scale out, but at the same time, other group of people say that this has to be fixed. There is no way for me as a maintainer to specify that ignore changes can be parameterized based on what you need because inside of lifecycle block, I cannot use variables. Actually, I cannot use variables in many other places, but in lifecycle ignore changes, this is one of most obvious. Second example is prevent destroy inside of lifecycle. We wanted to have this optional, like now I'm entering pseudo mode, this means that prevent destroy should be false. Okay, fine. So this feature is not supported in Terraform. I think it will be supported in the near future, to be honest, but we'll see. And another example is when we have so-called self-service. This is a feature which I have hardest time to explain and even figure out this myself, to be honest, but I want to have module content to be based on the input to the module itself. Not just values of resources has to be parameterized, but the content of the module itself. If we look into this module block, where we can see that HTTPS, mitraform.tf is actually a website where the source code of the module is generated based on results submitted from the form. This means that inside of ZIP archive, which is going to be produced by mitraform.tf, there will be content which is very unique. For example, I can embed some secret stuff based on request I receive, or I can generate infinite amount of combinations. I can use full features of programming language. I can even use Pulumi to generate terraform code inside of that HTTP response if I want. But the point is that this is a place where we can add so-called sheen to the terraform. If this was not enough, Sean from HashiCorp is a good guy. He published workshop puzzles. I think five of them, which I didn't list here, you can go to that repository and try to solve them yourself, and you will see that terraform is a pretty cool tool to do random stuff. When I say random, I actually mean random. For example, you may want to orchestrate printers remotely using terraform. I mean, yes, it has HTTP interface, so terraform can talk HTTP. You see? It works. I was told by one guy who actually did this for work because he works as a system administrator remotely. He said that sometimes he has to do, like most of his job is with terraform and proper usage of it, but also he has to do some IT stuff without going to work, which is good. So terraform 012, what does it mean for us? Please raise your hand if you are still using 011 and want to move to 012, like immediately, why this part of the room? Really? Okay, I will ask this part of the room. Do you guys know what is 012? Okay, good. So, yeah, talk to them. No, but what does it mean for us? So there are at least two groups of people. Terraform developers and terraform users. And the reason for this separation is quite strict, I would say, is that we don't have to expand the term of full stack developer even more, like you knew HTML, JavaScript, PHP, now Java, or there is DevOps or DevOps as well, and terraform as well. That's too much. There will be no quality. So if we will be constantly expanding this term, that's too bad. The quality will be not so good. What I believe in is that there are a group of people, typically we call them as DevOps engineers. They know AWS, they know provider, they know some company's standard, they know security to some degree, they know where is encryption, and they want to figure out how to connect different components together. So the end result from these guys should be reference architecture, which they publish and maintain inside of organization. And they actually use, happily, all features of Terraform 012. While terraform users are, on another hand, just go into Viki page and see what kind of solutions do we have in our organization, like how do we deploy this stuff. And they pick existing module from the reference architecture published in organization, and they use it just to build whatever they need. They don't have to know all these features about Terraform, Sheem, and crazy other things which exist. They just need to get their job done. And for that, they just need to know basics of HCL2, which is Terraform 012. They already know how to build stuff, and they know the main area. So my last slide here is that Terraform 012 is great, and there is no doubt that that's what we need, and we need this for many reasons. But I don't want to have even more people who are not dealing with Terraform on daily basis to be so excited that, yes, now I will go to work, and now I will make my code work in Terraform 012, and everything will be perfect. No, it will be crap, honestly. It will be crap because you need to understand the thought of internal implementations. You need to know how to connect these things together so that you can actually live with them when Terraform 013 comes out. And, yeah, Terraform is a universal tool. I hope you got this impression. But honestly, please evaluate and pick more carefully than just listening my talk. Thank you. APPLAUSE No questions. One question. OK, yeah. It's not going to work. I'd like to know why you don't like workspaces. I'd like to know why you don't like workspaces. It's so loud. Yeah. It's not going to work. It is so loud. This is for you. Yeah, I knew. This is a common question which people ask too. We've just changed using the workspaces. Sorry? We've just refactored our current use workspaces. Oh, right.
 Tell us all the things you should never do at Terraform.  Thank you.  Okay.  Now you can hear me.  Anyway, thank you, Walter, for the introduction.  And yes, in fact, there will be some things which you should not do, but at least you should  be aware that they exist.  So maybe you have this actual situation when you should be doing it.  So a couple words.  Oh my God.  Of course, it never works in this room.  So this time without clicker again.  What I do, I'm staying through the active in open source community and I'm writing and  maintaining a lot of open source tools related to Terraform, AWS.  And I became AWS community hero because I actually do something in the community giving  talks and trying to figure out how to make usage of AWS and Terraform.  First question to all of you.  Please raise your hand if you are using Terraform AWS modules.  Excellent.  Oh my God.  That's actually much more.  So it means that some people went to previous year talk as well because in previous year  I talked about Terraform best practices and how to use it properly.  And honestly, I am tired of using it the correct way.  So this time we will look into how to not use or how you can use and this should give  you some information about possibilities.  And yeah, what I do, I actually do everything related to Terraform, consulting and providing  different services, running workshops, trainings and I also like mentorship.  I mean, I'm exposed to a lot of different codes and solutions which I try to aggregate  and sometimes talk about them.  You can see my blog where I write exclusively about Terraform and AWS on AntonDabianka.com.  And yeah, this is stuff which most of you, as I understand, are using which is pretty cool.  There were 7 million downloads so far and I'm sure that amount will be just increasing  because come on, so far just 1,000 pull requests and issues resolved.  So we need more issues, more pull requests, more users and more sponsors.  So there is a link.  Another project which I'm involved is CloudCraft which is opinionated and pretty good way of  drawing AWS diagrams in the browser.  I have 25 minutes and I have 66 slides left so we have to go fast, okay?  What I'm working on is that once you visualize something in the browser, you don't have to  repeat yourself when you actually implement this as code.  So ModulesTF is an open source project which I work on.  You can go and use it for free and see what kind of code and structure and tools are actually used.  So infrastructure. Infrastructure is all what is necessary to run application.  It's all kind of computing, networking and other requirements which we have to maintain.  So this is my favorite definition of infrastructure despite the fact that this is not coming from Wikipedia.  Terraform. I'm not going to ask who use Terraform because otherwise you would go to another talk.  Plan. Disclaimer.  Everything what I'm going to talk will probably look, oh my God, why do you even show this?  I don't really care because this is disclaimer, okay?  And there are a lot of things, a lot of features of Terraform which exist in documentation, in users' code,  but it doesn't mean that they are good.  For example, Terraform workspaces.  And there will be some useful parts as well in the talk. I put them in the beginning so that you actually get something useful first  and then you figure out that, oh my God, Terraform can be used as a hammer because everything except, I don't know, is a nail, right?  So let's use Terraform for absolutely crazy situations.  And then we'll look into how Terraform 012 actually helps and what does it do and how cool is it.  So first add, of course, let's make Terraform with AWS faster for free with SMS and registrations, right?  Who doesn't like to have it?  So please use this one.  You will be surprised that if you specify in the provider that you can skip credential validation, check which region you are using,  meta, data, API check and all of this, you will have, in most cases, you will have at least one,  so you will have one instead of 10 API calls.  And if you are working on a slow network like an airplane, this helps a lot.  You can work with Terraform offline using these hacks.  And this is document, so this is not hack. This is useful, okay?  A little bit more practical stuff.  Versioning and secrets.  I mean, everybody likes to deal with secrets, I know, so we'll talk about that.  But first versioning.  So versioning in Terraform exists in many places and many flavors.  We have Terraform Core, which obviously has to be version 012.samsin providers, many from the community.  Now you can get them from the registry and a lot of them are in private because nobody wants to publish them.  And also there are modules which are more obvious way and more obvious things to be versioned.  Most of them are in registry, some of them are private, and so on.  So when we are talking about versioning in Terraform of Terraform Core itself,  there are not many tools which are useful and helpful.  For example, if you need to run Terraform on your local machine,  DFN is one of these tools which you run and switch similar as you do with RubyN, for example.  If you have to specify version of Terraform as well as version of providers,  that's really good thing that you're starting to look at your code,  something that will actually leave longer than you quit, right?  So you're worried about future.  And the point of versioning of Terraform providers is that when you specify your dependencies  and they will be combined later on and you cannot override them from the parent modules which you use.  That's why good Terraform modules are never using provider block inside of module itself.  They're always instantiated at the highest level.  And there are many tools which try to do similar stuff for modules.  Honestly, all of them failed, in my opinion, even the one which I'm talking about now,  because it worked on October and now it doesn't work.  The point is that it is really hard to make versioning correct.  For example, as it says here, support ranges for modules from Terraform registry  seems to be the case if the latest digit is less than 9.  10 is not more than 9 according to these two.  So I was surprised.  And overall, the flow looks like that is that we first set up version for the Terraform,  then we figure out which version of modules we need to do,  and then we use Terraform init to download providers which satisfy modules requirements which we have.  And then we use Terraform as normal, and Terraform plan, apply, destroy.  If you are into automation and CI CD things, you may look into the PandaBot,  which still has ongoing issue related to Terraform 012 syntax.  But from what I understand, it will be resolved sooner or later  because amount of disappointed people is growing very fast.  You can go there and upvote it yourself.  Secrets in Terraform.  It's still a problem with Terraform 012, not a big surprise.  Terraform Core has not done everything what is necessary, but let's see what we can do.  So first of all, there are two types of secrets which we are talking about.  The first one is inside of plan, apply, and output.  And then the second one, which is still no changes at all, is Terraform state.  So whatever you put will be in state.  First of all, when we specify that we want to output password,  we may specify that we want to use sensitive through,  then it will be written as sensitive.  Pretty good and pretty helpful sometimes.  If we want to use actually a random string and we want to generate this random password,  and then we use it somewhere else, quite recently there was a new resource  which is called random password.  And the cool part of it is that it does not output ID of the password itself as console output,  which means that we can run it in CI and it will not be available for anyone else.  So check out random password resource.  And overall, the solutions are very specific to the provider.  And if we are talking about AWS, there are a lot of options related to the provider implementation  of how to deal with secrets and how to encrypt them somewhere.  It is a pretty big topic in general.  So if you have some money, let's say from zero to $70,  you may get benefit from Terraform Cloud and you will be able to have secrets  managed using Terraform Cloud and your state file will be also stored on Terraform Cloud.  So you actually will not have so much problem with secrets,  but this is only if you have some money.  If you don't have money, you can use tool like Secret Hub.  I think it is one of the best tool out there.  I am surprised that not many people are talking about it, including employees.  I don't know why, but I am telling you that this is cool.  So check it out. It has support for TF-Rs, TF-States.  It is another resource, it is another provider in Terraform,  which you have to include and it will be dealing with secrets.  So Secret Hub is documentation.  So overall, versioning is doable.  Secret is still a problem. Secret Hub looks good.  That was the end of useful information.  Now let's look about what is actually hammer.  Hammer looks cool.  We know that Terraform is powerful.  Let's use for pretty much anything.  What means anything?  We don't care about resources, but don't care about state.  What does infrastructure glue Terraform mean?  How to manage any resource on AWS using Terraform?  How to do application deployment?  How to actually use dynamic Terraform modules?  What does actually Sheem means?  So let's look in the first problem.  How to create resources and don't worry about their state?  If you are coming from infrastructure,  like infrastructure management,  you get used to there is resource,  there is configuration or definition,  there is state and we have to take care about this TF-State file.  We don't care for it because we don't want to lose it.  It's really sensitive and it's all these scary things.  In most cases, it's right.  What about if we want to use Terraform for just write-only operations?  Or for example, we have some tooling which allow us to generate Terraform configuration files,  and we want to manage data which we populate console key value.  Or we just want to upload and overwrite any amount of files on S3  or any other reason.  There are a lot of different reasons.  I'm putting this as an example.  Yes, there is a backend type called inMEM.  Unfortunately, I cannot put link to documentation  because documentation doesn't exist  and nobody wants you to use it.  I encourage you to check this out.  It's really helpful if you are creating something  what you don't want to worry about.  For example, it's your test environment  which you certainly will never touch anymore.  That's a great example of using inMEM.  It will make your work with Terraform faster.  Of course, there is no state,  so every time there will be creation of resources,  there is no way to change them,  but in many cases it helps.  Infrastructure glue Terraform.  Well, some time ago we figured out that a lot of APIs exist.  Terraform has a unified way to manage all of them.  Like, you can manage resources on your file system  to create them, to read, create archives,  all different scripts, CLIs, REST API,  make HTTP calls, and many, many different things.  I see not many people are actually taking it seriously  because what it gives you is you have a way  to interact with these APIs or with these providers  in very standardized way.  For example, this code shows that we can connect  different type of resources and create files,  make an archive for them,  and upload this archive into S3 bucket.  So first of all, we use features of Terraform 0.12  to iterate through lists and make lists of users, for example.  And then we make files in specific location on our file system.  Then we make an archive from that directory,  and we upload this archive into S3 bucket.  So all of this is 24 lines,  which of course can be updated,  and you can use CLI or AWS CLI,  or you can use any other tool,  or you can write shell script in one line.  But the point is that of this talk is not to show  that, hey, the same you can do in better way.  No, Terraform can do this.  So how to manage any type of AWS resource?  If you are coming from AWS, you know cloud formation, right?  Many people know cloud formation.  Okay, cloud formation does not support a lot.  It doesn't support everything.  Terraform does not support everything either,  but in total, they support pretty much everything.  If not everything, then you have two options.  You have to open pull request to AWS provider,  and then wait, and wait, and wait.  Meanwhile, you have to use your own fork.  And I'm quite sure that there are people who open pull request  to Terraform AWS provider,  and you are still waiting for pull request to be merged, right?  Yeah, I can see some faces.  So yes, it takes some time.  It takes some time, and meanwhile, you have to use your own fork.  But even the worst situation is when you have some stuff  developed in cloud formation,  because that's defaulting in AWS back then, even still now.  And you want to move to something else,  but you were told that, oh, you cannot delete this database,  or S3 bucket.  So what can you do so that you have to use better tools,  but you at the same time cannot recreate resources?  This is where the requirement is.  If you could create and delete, that's easy.  So the solution for that one is that you first create cloud formation stack,  and you specify deletion policy retain.  So you have cloud formation stack where deletion policy retain  means that this resource will not be deleted  after cloud formation stack is deleted.  This means that it's not managed by anything else, literally.  Then you delete cloud formation stack,  and you import this retained resource to Terraform.  It looks like this is that you have to write code  describing these resources, like in this case S3 bucket,  and you have to import it,  and you have to see that plan is happier  because you customize the resource configuration,  and there is nothing else to change.  So what you have now is that resource is managed using Terraform,  and it does not have any relation with cloud formation anymore.  And if there are some situations which you cannot manage  using cloud formation or using Terraform,  you can still use null resource and AWS CLI.  AWS CLI still supports significant amount of resources,  even registration domains, if you want.  And yeah, of course, it's never too late.  So application deployment.  I have a little bit of time, and I want to ask you,  what was before?  I mean, on this conference,  you've heard about immutable infrastructure more than once.  You all know what is Packer, Docker, Schmocker.  There are many different tools out there.  But what was before?  Like, who tell, like, how did you do before all of this?  Let's say maybe in some places you still do it,  like I did, and I still do it for some of projects.  So what is, like, anyways, how you do this?  Don't mention anything fancy, like immutable.  Of course, no Kubernetes, please.  No Packer, Docker.  What?  Yeah, SCP.  SCP is pretty cool.  Yeah, SCP is supported quite successfully.  That's too boring, I would say.  What about this one?  Terraform and GitPool.  Isn't it cool that you can use Terraform to actually use  remote execution to connect to an instance,  which is not managed, of course, anywhere,  and you run GitPool on that machine?  Isn't it future?  Hopefully not, but still, you can describe this resource.  You are kind of one way into 21st century,  and then you say, oh, no, now I need to redeploy this.  So you run Terraform, you obtain that resource,  and you run the play, apply again,  and then you run it again.  I think it's brilliant because you are almost there.  You start to understand that, oh, yeah,  there is a gap between this process,  but Terraform can bridge them.  So, and one of the most complicated feature,  or not even feature, but scenario,  I would say how to use Terraform for things  which kind of was not designed for.  And this is called, like in the web browser,  it's called Polyfill, where Polyfill is code  that implements a feature of the browser  that do not support the feature.  In terms of Terraform, we all wanted to have something  what Terraform, if we go to Terraform, open issues,  and we see that there are hundreds of upvotes  or even some disappointed customers say,  oh, this feature is not supported,  we cannot use Terraform because it's too bad.  What can we do for them?  For example, for rich with modules,  like many people wanted to use it, I'm sure,  because I go to GitHub issues and I see so many upvotes,  or even more like historically,  people wanted to have reusable module,  but they wanted to have different values  inside of ignore changes.  For example, an auto scaling group.  I still have debate between people who say,  oh, we don't have to specify a minimum,  maximum and desired capacity as required values  because we wanted to have up and down,  like scale in and scale out,  but at the same time, other group of people  say that this has to be fixed.  There is no way for me as a maintainer to specify  that ignore changes can be parameterized  based on what you need because inside of lifecycle block,  I cannot use variables.  Actually, I cannot use variables in many other places,  but in lifecycle ignore changes,  this is one of most obvious.  Second example is prevent destroy inside of lifecycle.  We wanted to have this optional,  like now I'm entering pseudo mode,  this means that prevent destroy should be false.  Okay, fine.  So this feature is not supported in Terraform.  I think it will be supported in the near future, to be honest,  but we'll see.  And another example is when we have so-called self-service.  This is a feature which I have hardest time to explain  and even figure out this myself, to be honest,  but I want to have module content to be based on the input  to the module itself.  Not just values of resources has to be parameterized,  but the content of the module itself.  If we look into this module block,  where we can see that HTTPS,  mitraform.tf is actually a website  where the source code of the module  is generated based on results submitted from the form.  This means that inside of ZIP archive,  which is going to be produced by mitraform.tf,  there will be content which is very unique.  For example, I can embed some secret stuff  based on request I receive,  or I can generate infinite amount of combinations.  I can use full features of programming language.  I can even use Pulumi to generate terraform code  inside of that HTTP response if I want.  But the point is that this is a place  where we can add so-called sheen to the terraform.  If this was not enough,  Sean from HashiCorp is a good guy.  He published workshop puzzles.  I think five of them, which I didn't list here,  you can go to that repository  and try to solve them yourself,  and you will see that terraform is a pretty cool tool  to do random stuff.  When I say random, I actually mean random.  For example, you may want to orchestrate printers  remotely using terraform.  I mean, yes, it has HTTP interface,  so terraform can talk HTTP.  You see? It works.  I was told by one guy who actually did this for work  because he works as a system administrator remotely.  He said that sometimes he has to do,  like most of his job is with terraform  and proper usage of it,  but also he has to do some IT stuff  without going to work, which is good.  So terraform 012, what does it mean for us?  Please raise your hand if you are still using 011  and want to move to 012,  like immediately, why this part of the room?  Really?  Okay, I will ask this part of the room.  Do you guys know what is 012?  Okay, good. So, yeah, talk to them.  No, but what does it mean for us?  So there are at least two groups of people.  Terraform developers and terraform users.  And the reason for this separation is quite strict,  I would say, is that we don't have to expand  the term of full stack developer even more,  like you knew HTML, JavaScript, PHP, now Java,  or there is DevOps or DevOps as well,  and terraform as well. That's too much.  There will be no quality.  So if we will be constantly expanding this term,  that's too bad. The quality will be not so good.  What I believe in is that there are a group of people,  typically we call them as DevOps engineers.  They know AWS, they know provider,  they know some company's standard,  they know security to some degree,  they know where is encryption,  and they want to figure out how to connect  different components together.  So the end result from these guys should be reference architecture,  which they publish and maintain inside of organization.  And they actually use, happily,  all features of Terraform 012.  While terraform users are, on another hand,  just go into Viki page and see what kind of solutions  do we have in our organization,  like how do we deploy this stuff.  And they pick existing module from the reference architecture  published in organization,  and they use it just to build whatever they need.  They don't have to know all these features about Terraform,  Sheem, and crazy other things which exist.  They just need to get their job done.  And for that, they just need to know basics of HCL2,  which is Terraform 012.  They already know how to build stuff,  and they know the main area.  So my last slide here is that Terraform 012 is great,  and there is no doubt that that's what we need,  and we need this for many reasons.  But I don't want to have even more people  who are not dealing with Terraform on daily basis  to be so excited that, yes, now I will go to work,  and now I will make my code work in Terraform 012,  and everything will be perfect.  No, it will be crap, honestly.  It will be crap because you need to understand  the thought of internal implementations.  You need to know how to connect these things together  so that you can actually live with them  when Terraform 013 comes out.  And, yeah, Terraform is a universal tool.  I hope you got this impression.  But honestly, please evaluate and pick more carefully  than just listening my talk.  Thank you.  APPLAUSE  No questions.  One question.  OK, yeah.  It's not going to work.  I'd like to know why you don't like workspaces.  I'd like to know why you don't like workspaces.  It's so loud.  Yeah.  It's not going to work.  It is so loud.  This is for you.  Yeah, I knew.  This is a common question which people ask too.  We've just changed using the workspaces.  Sorry?  We've just refactored our current use workspaces.  Oh, right.
terraform_0_143.png terraform_144_887.png terraform_888_935.png terraform_936_2519.png terraform_2520_3191.png terraform_3192_4199.png terraform_4200_4691.png terraform_4692_5183.png terraform_5184_5375.png terraform_5376_6887.png terraform_6888_7739.png terraform_7740_9359.png terraform_9360_11363.png terraform_11364_11867.png terraform_11868_12419.png terraform_12420_13103.png terraform_13104_13487.png terraform_13488_14183.png terraform_14184_14723.png terraform_14724_15323.png terraform_15324_16211.png terraform_16212_16619.png terraform_16620_16667.png terraform_16668_16715.png terraform_16716_16763.png terraform_16764_16811.png terraform_16812_20891.png terraform_20892_22511.png terraform_22512_24419.png terraform_24420_25163.png terraform_25164_25223.png terraform_25224_25943.png terraform_25944_27995.png terraform_27996_29507.png terraform_29508_37031.png terraform_37032_39623.png terraform_39624_41771.png terraform_41772_42047.png terraform_42048_42131.png terraform_42132_42491.png terraform_42492_42613.png