Elephant Puppets
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Singer, Steve
Deployment automation for PostgreSQL Puppet is a platform for I.T. automation and configuration management that has gained popularity in the devops movement. This talk will provide an introduction to puppet and talk about how Afilias uses puppet to deploy and maintain over one hundred servers running PostgreSQL that support key parts of the DNS infrastructure Puppet allows DBA's to automate the installation and configuration of PostgreSQL across multiple servers. This talk will explain how puppet can be used to install and configure PostgreSQL. Automated configuration of your database servers keeps servers looking the same allowing you to control differences. Managing your database servers through puppet also allows your to easily reproduce your configuration for development and QA purposes. The talk will cover the basics of puppet and best practices for deploying puppet to manage PostgreSQL including: How to install PostgreSQL via puppet and create database clusters How to manage postgresql.conf, pg_hba.conf and .pgpass via puppet Best practices for dealing with configuration differences between different database servers and different environments Afilias is a registry services provider running registries for numerous TLDs including: .INFO, .ORG, .MOBI, .XXX, .IN, .ME, .ASIA, .MOBI. Afilias has been using PostgreSQL since the 7.x days and currently has more 100 production database servers running PostgreSQL.
 nation framework. And before we get into that, a bit about who I am. I've been a member of the Postgres community working with PostgresQL for quite a few years. I am one of the maintainers of the Sloanese Replication System. And I do this while working at Aphilius. Aphilius is a top level domain registry operator. So who in the room knows what a top level registry operator does? A lot of people, good. We run registries, so.info.org, the top level domains that we run. If you wanted to register a name in one of these registries, you would go to a registrar, someone like a GoDaddy or a TwoCouch. You'd go to their website, type in the name that you want. If it's available, you'd say you want to register it. Behind the scenes, they are talking to our servers. They would open up a connection to our servers, check if the domain name is available, and register it on your behalf with us. We use Postgres as the database for storing the system of record for all of the top level domains we run. And we tend to use Postgres in a lot of our back office processing systems for our operational needs, such as billing and reporting. We've been a Postgres user since, I think, 2001 or so, back in these seven X days. Back in those days, Affilius was one of the higher profile, larger Postgres users. These days, there are a lot of other figure of Postgres installations, but we're still involved in the community. We're still heavily involved in Sloney. And we are quite happy with our experiences with Postgres. So a bit about our systems, we have over 100 production Postgres databases. And when I mean databases, I mean different servers running Postgres instances. There's people at the conference who have one database server with tens of thousands of databases inside of it. We run lots of servers with different databases. Many of these databases are wrapped because of each other. Being big Sloney users, we tend to have needs for lots of replicas. So for many of our operational registry databases, we will have a master database and five or six replica databases. These replica databases we use for things like failover, and reporting, and offloading back office functions too. And we have other systems that are not replicated. We have our bug tracker users Postgres. We have billing and data warehouse systems using Postgres. So we have a lot of databases. And we need to manage them. About a year ago, or a year and a half ago, it was time to replace our hardware. We had been running Postgres on AX using PowerServers IBM. These servers were over five years old. The warranties had expired. They were not as fast as a machine you can buy today. It was generally time for a hardware refresh. We went, did an evaluation, and decided we were going to move to Red Hat Linux on VMware virtualized. And at the same time, we decided, hey, we're moving all of our servers. Let's move data centers as well. Our data center contracts were up, and we decided we were going to change our data centers. So we had a lot of new servers. We were going to have to buy, provision, install, and migrate to in a relatively short amount of time. So how could we use automation to help in this? We had already been using CF Engine, which is another configuration management system, deployment system, to deploy our Postgres binaries. But we weren't really using CF Engine. We were a lot more than deploying binaries and maybe deploying front apps. Other departments in affiliates had been using Puppet, particularly our systemins who maintain our registry application, the software we write in-house. They were deploying with Puppet. And we decided, let's standardize on Puppet. We'll use Puppet for the database stuff. And we'll do that as part of this migration. Question? What is the reason why you use Puppet over the Chef? Basically, because the other team had already been using Puppet, the three big players that you hear about are CF Engine, Puppet, and Chef. It seems like CF Engine is losing popularity. You hear about Puppet and Chef all the time. So we picked Puppet because we were using it already. So deployment automation for those who aren't familiar with it is the idea is you want to automate the tasks in setting up a server. Sort of on the non-automated Postgres installation, sort of the extreme case, you have to install a database server. You download the tar file from the Postgres website. You uncompress it. You un-configure. You un-make. You un-make install. You run an IDDB. You copy over a file into the init script there, and so on. This gets very tedious and long, particularly if you're doing this for more than one server. Anything that is more automated than what I just described is a form of deployment automation. Some people have been using shell scripts for this type of stuff for decades. This is deployment automation. Some of the deployment automation packages like Puppet and Chef are frameworks for doing these common tasks easier and with more reuse than if you were writing your own shell script. We had a couple of sort of general goals for our deployment automation project. We wanted servers to look the same, which doesn't mean we wanted them to be identical. Think of a web server where you have 10 web servers for the same web app, all doing exactly the same thing behind a load balancer. There, your servers are pretty much identical. For us, while we have 100 databases, none of them are really identical. Many of them are running different versions of Postgres. Many of them are running different database schemas for completely different products. And then others are configured essentially the same. So we needed a way to keep things that look mostly the same, but manage the differences. So before we did automation back on AX, we hit an instance where the directory we were writing, the Postgres logs, PGLog, on a couple of servers was one level higher in the directory tree in the file system than it was 90% of the places. How did we discover this? It was in the middle of the maintenance. A DBA was doing something, and the log directory was not where he expected it to be. He might have been running a script to check something, and it wasn't where the script expected it. And this is not the type of thing you like discovering in the middle of the maintenance, because, yeah, you can deal with the other directory moving, but it's a pain. We wanted ways of managing this and preventing that type of situation. We also wanted consistency between our production and our development in our QA environments. As we moved to more deployment automation and more automated tasks in general, we wanted to be able to confidently test our scripts in QA and staging and know that they have a good chance of working in production. While we were on AIX, we had, I think, three database servers in our staging environment and maybe five in our QA environment for all of our different products, all our different projects and systems. So out of those 100 databases to test something we might have in QA, we might have five servers to choose from. So QA would install different versions of Postgres on the same server, create multiple backends listening on different ports, and do whatever they needed to do so they could load the data and test the application functionality on the server with lots of other QA activities going on. This was fine for testing application functionality, but it tended to have some problems. Particularly if you wanted to test maintenance type tasks, it didn't work so well. Now I'll admit, while this was one of our goals, we've not been so successful yet in making our QA environments look as close to production as I would like. We're still sort of working on making progress on that. We also wanted to reduce manual commands. Before I gave the example of doing the install manually, if you have to upgrade Postgres on 100 servers, maybe you have to change the version rather quickly because there's a security patch you need to roll out, the less typing you have to do on each of your 100 servers, the better. So we wanted to reduce that, and deployment automation was a good way. As I mentioned, we've picked Puppet. Puppet is an open source project with a commercial addition that you can buy. You can get Puppet from PuppetLabs.com. PuppetLabs does most of the maintenance of Puppet. The open source version or community version of Puppet will probably do everything you want. We initially launched with the open source version when we did our stuff. And then we bought a commercial support contract with PuppetLabs, mostly for access to their professional services. And then we migrated to the enterprise version. It has a bit fancier, gooey, a bit nicer packaging. But most everything I talk about in this talk applies to both the community version and the commercial version. So a bit about how Puppet works. In Puppet, you have a Puppet Master. The Puppet Master is a server that will have all of your Puppet module source code. And a list of what goes on which server. Then you're going to have your servers, which might be database servers, web servers, LDAP servers. These servers do your real work. And they will run a program called the Puppet agent. The Puppet agent contacts the Puppet Master and asks the Puppet Master for instructions on what it should do. And the Puppet Master should stuff down to the agent to make changes on the agent server. Some terminology that I'll refer to. In Puppet, when you tell a server what should be deployed on it, that's the node definition. And the node definitions consist of a set of classes. So for a particular node, you might be saying we're deploying the PGSGrel class and the Sloanee class. The Puppet Master will take this node definition, which is the list of classes, and the actual source code for these classes, and batch them up together and produce a catalog. This catalog is instructions for the agent on what to do and how the agent should make the server look. This catalog gets shipped to the agent. What's a Puppet class? This is a very basic Puppet class. Puppet classes can instantiate other Puppet classes. They can instantiate Puppet resources, and they can have things like variables being set and basic control structures like if statements. A Puppet resource is something in Puppet that you instantiate to do something. Puppet ships with a whole bunch of default pre-provided resource types. One of them is the Package resource. The Package resource installs packages with your operating systems package management functions. So this particular example will create an instance of a package resource to install the Postgres URL server package using YAM. So on a Red Hat system, this will install Postgres. And that's what the class will do. Puppet is also declarative. Another resource type Puppet ships with is the File resource. The File resource manages files in the File system. So the idea is you declare that a file should exist on your server, TemphellerWorld, and you declare what the content of that file should be. So in this example, I'm saying I want to have a file TemphellerWorld, and it should contain the content Welcome to Ottawa. When the Puppet agent runs, if there's no file in Temp called HelloWorld, it'll create it. And same, Welcome to Ottawa. When the Puppet agent runs, and it finds a file in TemphellerWorld, but it says Welcome to Toronto, Puppet will change that file to say Welcome to Ottawa. So you run Puppet once, it makes some changes. You run Puppet again, and you haven't changed anything else. Puppet should really do nothing. It checks to see that everything that you've declared is still true, and it'll do nothing. If something has changed, Puppet will make what you declared true to be true again. And this is the philosophy of Puppet. There's ways you can do things that don't follow this philosophy, but you're going to get into trouble. Puppet is meant to be declarative, and you want to build your Puppet classes to do things in this declarative fashion. So the way we deploy Puppet at Affilius is we have a Puppet master, and we have our servers, database servers, Apache servers, Tomcans servers, et cetera. We write our Puppet classes, and we check them into a Git repository. The Git server we push things to has some hooks that push things onto the Puppet master to make those codes available. When we do things in Git on a particular branch, that branch is then pushed to the Puppet master. And we can have multiple Git branches of Puppet classes available at once. This allows us to have different servers running different versions of our Puppet classes, which is very useful if, let's say, you want to test your Puppet changes in QA before you want to push them to production. You tell Puppet that you want it to use the test 123 branch on a QA server while it's using the prod branch on a production server. Git allows us to do this rather nicely with the hooks. Then, before I talk about a node definition, you can write a file in Puppet that says for server one, it's running classes Postgres and Sloney, server two, it's running classes Tomcat and your Java app. But if you have a lot of servers, a one file with all of this gets a bit unwieldy. We already use LDAP for managing a lot of our infrastructure, what's servers there, what the names are, et cetera. So with Puppet, you can have what's called an external node classifier. And an external node classifier is a program that Puppet will contact during the run that says for server 123, what classes should I install here? And there's a bunch of pre-written external node classifiers, one of them interfaces with LDAP. And that's what we do. So in LDAP for server 123, we list the Puppet classes as attributes in LDAP that should be installed on a server. We also list the branch from Git, the environment, we call it, that we want to run on that those classes of. So a server contacts the Puppet master, Puppet master contacts LDAP. It gets what it should install. The Puppet master builds up the catalog, ships it to our servers. Couple slides ago, I showed you a web page with a management console. That's the Puppet Enterprise Management Console. We can launch Puppet run on a bunch of servers. So I can say all of our servers in Texas, we want to do a Puppet run now that are database servers. I select them on the GUI, launch Puppet. It'll go run Puppet now. And I can view the results in a report screen. If you're not using a fancy GUI, you can launch Puppet on the command line of each of your servers. You could write shell scripts to log on into each of your servers and do this. A lot of people run Puppet on a interval, so they have Puppet automatically runs once an hour or once a day. We're not doing this yet. We might eventually get to that point. That's sort of the other way you can deploy Puppet. So it always runs. Make sure that the things you declare in your manifest are true. We had some specific goals for deploying our databases. We wanted to deploy binaries. We wanted to deploy cont files and manage cont files. We wanted to handle the initdb because as we were rolling out our 100 databases onto these new servers, we wanted to get that done very quickly and easily. We also have a lot of Trontabs that we have to manage. Each of these things I'll talk about in more detail in a couple of slides. And we did some things with Sloney that are easier to manage with Puppet as well. So there is a module for Puppet to manage Postgres. Puppet has these different modules, collections of classes. Many of them are available on GitHub projects and Puppet Labs websites. There's one for Postgres. It does many of the things I talked about in the previous slide, starting, stopping Postgres, handling your cont files. You can get this module at this URL. We don't actually use it. The reason being we had a perceived need to install multiple minor versions of Postgres on the same server. So we thought we wanted to install a 9.2.4 and a 9.2.5 on server at the same time and have them running at the same time. The Postgres RPMs that you get from the community don't really do that. So we had to build our own RPMs. And it seemed to be easier to just write our own class for managing Postgres than it would be to modify the community classes here to do that. But I would recommend you take this as a starting point and see if it will meet your needs. So what we instead do is we have a resource called the PGSQL cluster resource. It manages a single data deer. So it will launch, create the data deer, perform the NITDB, install the cont files we want here. And then we have a whole bunch of infrastructure that sits above this layer for managing actually what cont files and what data deers and what users go to a particular server. So one of the more interesting things and more common things you're going to want to do is manage files like your Postgres cont file. Puppet has a templating language. It's the Ruby templating language. If you've ever used Ruby on Rails and done web apps, HTML, it's the same templating language that that uses. Puppet is actually written in Ruby. And there are many places in Puppet where you can break out in your classes and basically write pure Ruby code to do things. And often you will need to do that when there's no easy way to accomplish what you want in Puppet. So the way these templates works is you might have a Postgres cont and you want shared buffers to be variable. So you put that in and instead of a specific value for shared buffers like 32 megs, you would say, within these square brackets or triangle brackets, shared buffers. And shared buffers is a variable and that will get evaluated out of runtime. So when you're deploying your catalog, that variable will be substituted with, let's say, 300 megs. So the template, the ERB, gets turned into a specific confile with the variable substitution applied. The way you would use that template is using the same file resource we discussed before. You set the variables in your Puppet classes. So shared buffers might be 300 megs and we set the template we want to use. And then we're saying this file is for the Postgres conf, which should be in the data tier. And the content of this file, instead of being a static string like my previous slide, it should use the template and we pass it the name of the template. Puppet then goes, evaluates the template in the context of the variables you've set in the class and gives you your result. You can also do ordering on Puppet resources. So here we're saying that we also want to make sure we run this file after we've done the initdb. I have another package called exec, which execs the initdb command, or another resource that execs the initdb command. And we're just saying, in order to run the Postgres file conf, we require the initdb to have been completed. If you install your template before the initdb, it'll get overrided. So in Puppet, each resource instance has to be uniquely named. So if I had a different Puppet class that was installed on my server that had this string here with data tier being the same value, it would be a conflict. Puppet would say, oh, this file or resource in this class wants Postgres conf. And this file, also this other class is also defining Postgres conf as the same thing. Which do I choose? I don't know. I'm confused. I'm going to error out. So every resource and every class has a name. In this case, the name of the file resource I'm instantiating, the instance name, is the path to my data tier slash Postgres conf. The class that I'm including this to also has a name. And the $name evaluates to the name of the current instance you're running at for the class or resource you're inside of. Does that answer your question? So I have no idea. That's probably how I named it. There's no good reason. The $name will give you the uniqueness. It could even be a typo on the slide or a typo in my code. So some parameters in your Postgres conf file, you know, are going to want to be based on information on the server you're actually deploying to. A good example is the IP address. Often you will do listen address equals star to listen on all your Ethernet devices. But you might only want to listen on the first Ethernet device. Postgres conf requires you to type in the IP address of that first device. But this is something you can automate. Puppet has a program called Factor that runs as part of the Puppet agent run that collects facts about your system and ships them back to the Puppet master to make these facts available as variables during the Puppet run. For example, the IP address of the first Ethernet device and the second Ethernet device and all your other Ethernet devices are available as facts. The amount of memory on your machine is available as a fact. The operating system you're running is available as a fact. And you can also write custom facts. You can write a bit of Ruby code that will run on the server, the agent server, as part of the agent run to do whatever you want to compute a value that gets shipped back to the Puppet master as a variable. Then inside of your templates, you look up these variables and use them as values. Some variables you're using are coming from your agent servers. But other variables you're going to want to control and set as configuration to manage your configuration and get a lot of usability between your Puppet classes. So you have your Puppet classes and you have 100 database servers. Ideally, you would have one Puppet class for all your database servers to maximize code reuse. But if you're doing this, you need to take the configuration parts out of your class and put them somewhere else. Puppet has a library or facility called Hira, which is a data lookup library for looking up data in a hierarchical fashion. What it allows you to do is move your data out of your classes into these YAML files. And you can then do lookups and vary your configuration hierarchically. So the way it works is you might have a DB server class and you might want to find out what ports should Postgres be running on. And it might be that for different products you're selling or doing, you have different ports you want your Postgres to be using. So you'd say my application is Postgres, my product is org, and the Hira, what is my Postgres port? And Hira will come back with the port it should use. How does Hira do this? First, you have to define a hierarchy. This is sort of an example of a hierarchy where what I'm saying is for a particular fully qualified domain name for my application, which is Postgres, I want to be able to set things on a particular server. But other things for Postgres, I want to set for the entire domain. So for all of my tor.affilias.info servers, I want to set some stuff there. And then texas.affilias.info, I want to set other values. But some environments, some settings might not vary based on where your server is located or the domain name of your server. Some of our settings vary based on what we call environment type, which is whether it's a QA server, a production server, or a staging server. So we've made that another variable. And then other things, like maybe the port, varies based on the product. Everything in yellow is a variable that we've defined. And we've decided these are the switches, the control things we want to use to build our configuration trees. You can pick other things that work for you. The important thing is, if it's a variable in your Hira configuration, this variable should be set either by factor or explicitly as we did in our slide here. So how does this then work? You then write some YAML files. YAML is a language for writing data. It's kind of like JSON. Hira, its default is YAML, but there are other backends you could use. And you write these files that will contain your configuration in a tree structure on your file system. So this tree structure matches the structure of my previous slide. So some values, we might have a server called db1.tor.aphilius. And we set some values in Postgres here. So when I go look up my port, Hira will first check this file, this Postgres file, for db1, because that's the fully qualified domain name. It doesn't find the value. It moves on, oh, this is a production server. Have I defined it at the production level? No, I don't see it there. Oh, this server is, the product was org, because that's the variable value I set to the variable two slides ago. Yes, I see a port value. Hira will return that value. If it found the port value at the top, the first file it looked, that's what it would return. Hira will walk through your hierarchy in the order you define it, looking for the values you've set. What do one of these YAML files look like? They're pretty simple. You have a key and a value. So the key in this case is Postgres port. The value is 5 or 3, 2. You can also build more complicated structures. So at the bottom here, we have our dbList, which is a hash. So the key is Postgres dbList. The value is a hash. The hash, the key of a hash is PgCon, which is, let's say, the database name. And the value that points to is the owner of the database. There's another value in the hash for a different conference with a different owner. We then have some puppet code that walks through this hash creating databases with create database, or with the create database SQL command. For every database we've defined in the dbList, and setting the role in Postgres that owns the database to be the role we've set here. We have some databases where, or some clusters, I guess I should say, where the databases in the cluster vary. For a lot of miscellaneous things, we'll have one database with four or five different databases in it, and another cluster will have three of those four, but not all of them. We use this to manage what databases go on a server, and it works pretty well. As you manage your Postgres configuration, you're going to need to come up with a strategy for differentiating different confiles and grouping things. I'm going to talk about two strategies. This is the one we actually use mostly. For every major version of Postgres, we start off, and we do the initDB, and we get the default confile for that version of Postgres. And then we start to change it. There are some things that we change in the confile that apply to all of our database servers. And then there are some things that differ based on the class of database. Some of the databases are small. They're not running on big systems. They don't have a lot of activity. So we created a grouping of servers called small. And we have confiles for 9.0, 9.1, 9.2, et cetera, that are small. Other databases do batch processing systems. These systems typically have few connections, use lots of memory. So we set up a set of files for these batch systems. Max connections are low, workman is high, min duration for logging long running queries is rather high, things are appropriate for batch jobs. Then our OLTP systems, we tend to do a lot of very high max connections, much smaller workman, because max connections are so high, and we have a lot of connections. And our min duration, we consider a period running long, if it's 100 milliseconds on these systems. So we log it. While on our batch processing systems, the query of a minute is not long at all. So we built things into groups, and we try to manage things on these groups. And even though different applications are using the same group, we try to keep it together. Yes? Do you get a lot of value out of the type of files that are major works? You probably could. The question for the audio was, do we get a lot of value about having different version templates for each major version? And I like the fact that you can see the values that we've not changed. And sometimes the default value is changed in Postgres. And we sort of find that's helpful. But it might work well for others not to do this. The overhead of how there's not that many major versions of Postgres, so the overhead of this is not a big deal. And there are settings we've changed on different versions of Postgres. Sometimes there are things that are appropriate for 8.3, that are not appropriate for 9.2, as an example. And you could conditionalize some of this stuff, which sort of brings me to my next slide. Instead of having one file for class of systems, you could have one file for each major version of Postgres, make everything that you might change a variable, and then set up a higher level or use higher to manage the different groupings. This is probably a better way of doing it. And I'd like to move from having one confile per class to one confile per version. What Peter suggested is, I guess, another level we could go to of just one confire for all versions. I'm hoping to go to this level. So we would say then that our shared buffers and work mem are all variables, and at some higher level, we would say, for big, it would be certain values, and for small, it would be other values, and some systems we would override. Another thing we do a fair amount of is crontabs. We have a lot of crontabs for things like backups, purging jobs, both purging things in the database, purging log files and the file systems, et cetera. And so while it wasn't a big deal for us to rewrite that module for installing Postgres and doing the NITDB, because probably like 75% of our puppet classes in terms of volume of code deals with all this other stuff that is not reusable between other places with respect to managing our crontabs and what files we go. We actually run most of our crontabs. We have database servers, and we have what we call database application servers. A database application server does not run the Postgres daemon. It has Postgres installed, and it runs crontabs and our slawns and other things that connect to a database and do maintenance. We do this to move load off of our database servers onto other servers, because our backups between the BZIP and some of our backups we then encrypt and send places. It's actually a fair amount of CPU load, and getting this stuff off of a database is helpful we found. So managing these crontabs is a big part of what we do. Question? Some questions say capital management doesn't even use these words for it. That is probably open office misbehaving. Yeah, so to do the cron, to manage crons, Puppet has a cron resource type, and the cron resource type manages your crontab. So for a particular line in your crontab, you will declare a instance of the cron resource type, saying I want a cron for a backup under bar name. This is the name I was talking about again, where because we have so many different servers, and you cannot have duplicate instances of a cron. So name might be the product name in this case. Our class might be named DappServerOrg. And we would, so cron would then create a crontab with a name, which will be a comment in the crontab file, DappServerBackupUnderBarOrg. And it will put the command we've specified here. And the command can have a variable. So this way, we have the same Puppet class or Puppet resource that lets us back up all of our different products. And some of our products, we might be running the backup at one time, and others, we might make the minute be a variable, instead of 0, so we can control the time. We have a lot of different products and crontabs that do different things. And we've been managing those crontabs with Puppet. And we have switches to say, which crontab should run on which products? And we have switches in our higher files to control what options we pass to them. And we have other switches to control what servers a backup should be launched from. So it gets kind of complicated, but higher has turned out to be a pretty good way of managing it. And it definitely is working better than trying to manage all of these crontabs directly with CVS or SVN, pushing them everywhere. Another thing we use Puppet to help us do is manage some of our Sloney stuff. For those of you who don't know what Sloney is, it's a replication system. And to configure Sloney, Sloney has a command line language called Slonic. Slonic is how you tell Sloney what it is you want Sloney to do. And this is an example of a typical Sloney script. They start out, you name the cluster, and these different clusters in your environment will have different names. Then you tell Sloney how to contact each of the nodes in this cluster. And you have to provide the host name and users, et cetera, in the Slonic file. Then you actually do interesting things like your Sloney commands. This file will not work in both the QA and production environments, because at the top, we're specifying host names. And the host names are not the same in QA and in production. So how do we solve this problem with Puppet? We have a template in Puppet that generates Sloney through evils. These are files basically contain that top part of our Slonic commands that are the same for any command. We go, oh, the cluster name should be from the variable cluster name. Then we do some more advanced templating stuff. We can do loops in the templates. We loop over all of our nodes. Slonconf is a hash that comes from Hira. And we loop over it. We get the node ID, which is the key, and some values. And the values is another hash. And the values might have the port. It might have the host name. We could put users and passwords in this hash as well. And we write out that admin con info line from the previous slide using our Puppet template. Sloney also has the idea of the variable that you can then reference later. In addition to the con info, this same information is available as a Slonic variable. So we can reference that path to a particular host later in a Sloney store path command. This is an example of how you can build up things in Puppet to separate the stuff that is the same on all servers, let's say the set add table from stuff that varies. So then when we run our Slonic commands on QA or production, we first include this Purembo file. Then we do our set add table or whatever it is we actually want to do. So how do our YAML files in Hira look for something like this? That hash I showed you that we were looping over will look like this. The key might be put to a Slon nodes. And we'll do a Hira lookup and assign that to a variable Slonconf. And then we have the node ID 9181. That's the first level of the hash. That contains another hash that has host and port. And then we have another entry that's a different host and a different port for a different Sloney node ID. This might be our production conf. And then for QA, we would have a different YAML file that had different values for these things that would generate a different Purembo on the server when we ran Puppet. So I'll talk a bit about what has been a bit challenging for us. One issue is removing stuff. You declare these Puppet classes. You run the Puppet agent. It installs a bunch of things. Then you decide, oh, I no longer want to be running this database on a particular server. Maybe you decide to move it. Or maybe you're decommissioning it. You don't want to throw away the whole server. And if you were doing this for web applications, for a web server, often what people do is they will just build a new web server and replace the old web server with the new one behind their load balancer and throw away the old one. With virtualization, this is very easy if you're doing web servers. Because you can just throw it away and replace it with another one if you're using a load balancer. This doesn't work so well with databases, because you can't really, unless you're throwing away the entire server, you have to keep that data. If you're only throwing away one of the databases on your server and keeping the other one, you can't throw away the data of the one you're keeping by reprovisioning. So what we've found we needed to do is for every class that does something in Puppet, we've had to write a second class that undoes it. So we have these removal classes and removal resources. So we'll install things with the install one. And then when we decide, oh, we want to actually remove this from a server, instead of having that server list, the install class, the post-dress class on the server, it might list the post-dress remove class in the manifest, in the node list, I should say. So it undoes things. Testing this is a bit of a pain, because you have to install things, make sure things are how they look, right, deploy your removal class, and then look for any changes, any leftover things. And usually, when you're installing something, you want to get it out there. You're not worried about removal when you're writing the Puppet class initially. You just get it out there. And it's only when you uninstall something that you tend to bother about writing these removal classes. No, yep. That's normally handled by attributes and reports, but file reports are not necessarily to ensure that the Puppet is not absent. Yeah, so Puppet, for example, the file resource has an insurer present and insurer absent. So just like we were ensuring that the file was there in the examples I gave, you can also add an attribute, insurer absent, where Puppet will make sure the file is not there. And for your resources, that's how you remove things. And one way of doing this, instead of having a separate class for removals, you can have an insurer absent mode that will undo it. We've mostly not gone that route. Maybe that is a better route to go, but I've been doing separate classes. I don't have, off the top of my head, a good reason why. But either case, you have to actually think about what should happen in a removal and spend the time writing the code to remove that. We also have had some challenges. Yes? Could you also just stop the full-fledged learning in the technical distribution? Well, the issue is, so let's say we have Postgres, and we have two databases in Postgres, Database 1 and Database 2. And if you want to remove Database 1, but not Database 2, you can't do that. But even if we have a class that removes Postgres, and it stops Postgres, it deletes the DataDeer. Because often, if we're deleting something, we want that space free. We don't want that DataDeer laying around, because we're likely going to deploy something else on that server. So yeah, you can just implement half the problem, or half the solution by just stopping Postgres, and not completely cleaning it up. But eventually, you're going to want to completely clean things up. So password management is also a challenge. If you put your passwords in your higher files, and you put these into Git, and lots of people have access to your Git branches, you just exposed your passwords to lots of people, that's not so good. So when we first tackled this, we actually weren't using Hira. And I wrote some custom Ruby functions that ran on the Puppet Master to look up passwords. It was sort of like Hira, but custom stuff that just look up passwords in the passwords were only on our Puppet Master, that only our systemins had access to. And that worked OK, except adding and removing passwords to Hydraksis and Min to do that. Then Hira, when we moved to Hira, we discovered that there were different back ends for Hira. Hira has, in addition to the YAML back end, it has a back end that works on GPG encrypted files. So you write up your YAML files, and you encrypt them, and you can check them into maybe a different source repository, and get them somehow deployed onto the Puppet Master. And then you deploy decryption keys on the Puppet Master that you control fairly tightly. And then the Puppet Master can decrypt these YAML files to look up passwords and push them down to into the catalog so they get deployed on your server. You can do this for writing things like writing VG pass files, et cetera. Changing passwords is also a bit of a challenge in that if you want to change your database user password, your super user password, you might need both the old and the new password available at once. You need the old password to have your PSGrelSmart login, and you need a new password to set it to something different. This is something we've not yet solved. One way is to have other, one way of doing this is to have the password available as two variables. Another way is to have a different user that you're using for changing these passwords. But we haven't actually implemented a solution. There are some next steps we're looking at doing. One, I spoke about how we're managing databases as we define the list of databases we create. We're not doing this for our typical database login users. Right now, we're still manually logging into our server as issuing a created role or a dropped role. And it's getting kind of tedious. I'd like to sort of have Puppet manage the login users, the roles for our databases. I think that's quite doable. Also, when we push a compile change, we are not restarting Postgres. So we change, let's say, shared buffers in our template or in Hira. We push this down. It gets, the compile has changed on our database server, but it has no effect because we've not restarted Postgres. You could tell Puppet to just restart Postgres, but that means Postgres is going to be shut down when Puppet runs. If you're running Puppet every hour, you probably don't want to have your database go down and planned if it's an important database. So I still have to figure out some good techniques for managing, restarting servers that can be restarted, sort of at whim, because a lot of our replicas actually can be restarted at whim, because they're not yet being used for things, but not restarting servers that can't be restarted at whim. And then getting an idea about when we've pushed something to a server that doesn't take effect until a restart, because changing the compile, but not having it take effect, is a good way to confuse yourself. So that's sort of on my next list of 2D stuff for Puppet. Any questions? You mentioned the tools for pushing changes from Git to your public master, because they were branch-aligned. Yes. Are they kernel tools or are they open source? No, we use, I think it's Git to Lite as our Git server. And then we've written custom hooks. We've written something that involves RabbitMQ, I believe. I think. Those hooks are custom. Yeah, those hooks are custom. We might be able to release them or open source them if there's interest. I can talk to the guy who wrote them and see what they look like and maybe push them to GitHub. Any other questions? Okay, thank you. Thank you. Thank you. Thank you.
 nation framework.  And before we get into that, a bit about who I am.  I've been a member of the Postgres community  working with PostgresQL for quite a few years.  I am one of the maintainers of the Sloanese Replication  System.  And I do this while working at Aphilius.  Aphilius is a top level domain registry operator.  So who in the room knows what a top level registry operator  does?  A lot of people, good.  We run registries, so.info.org, the top level domains  that we run.  If you wanted to register a name in one of these registries,  you would go to a registrar, someone like a GoDaddy  or a TwoCouch.  You'd go to their website, type in the name that you want.  If it's available, you'd say you want to register it.  Behind the scenes, they are talking to our servers.  They would open up a connection to our servers,  check if the domain name is available,  and register it on your behalf with us.  We use Postgres as the database for storing the system of record  for all of the top level domains we run.  And we tend to use Postgres in a lot of our back office  processing systems for our operational needs,  such as billing and reporting.  We've been a Postgres user since, I think, 2001 or so,  back in these seven X days.  Back in those days, Affilius was one of the higher profile,  larger Postgres users.  These days, there are a lot of other figure of Postgres  installations, but we're still involved in the community.  We're still heavily involved in Sloney.  And we are quite happy with our experiences with Postgres.  So a bit about our systems, we have over 100 production  Postgres databases.  And when I mean databases, I mean different servers  running Postgres instances.  There's people at the conference who have one database server  with tens of thousands of databases inside of it.  We run lots of servers with different databases.  Many of these databases are wrapped because of each other.  Being big Sloney users, we tend to have needs  for lots of replicas.  So for many of our operational registry databases,  we will have a master database and five or six replica  databases.  These replica databases we use for things like failover,  and reporting, and offloading back office functions too.  And we have other systems that are not replicated.  We have our bug tracker users Postgres.  We have billing and data warehouse systems using Postgres.  So we have a lot of databases.  And we need to manage them.  About a year ago, or a year and a half ago,  it was time to replace our hardware.  We had been running Postgres on AX using PowerServers IBM.  These servers were over five years old.  The warranties had expired.  They were not as fast as a machine you can buy today.  It was generally time for a hardware refresh.  We went, did an evaluation, and decided  we were going to move to Red Hat Linux on VMware  virtualized.  And at the same time, we decided, hey,  we're moving all of our servers.  Let's move data centers as well.  Our data center contracts were up, and we decided  we were going to change our data centers.  So we had a lot of new servers.  We were going to have to buy, provision, install,  and migrate to in a relatively short amount of time.  So how could we use automation to help in this?  We had already been using CF Engine, which is another  configuration management system, deployment system,  to deploy our Postgres binaries.  But we weren't really using CF Engine.  We were a lot more than deploying binaries and maybe  deploying front apps.  Other departments in affiliates had been using Puppet,  particularly our systemins who maintain our registry  application, the software we write in-house.  They were deploying with Puppet.  And we decided, let's standardize on Puppet.  We'll use Puppet for the database stuff.  And we'll do that as part of this migration.  Question?  What is the reason why you use Puppet over the Chef?  Basically, because the other team had already been using  Puppet, the three big players that you hear about are CF  Engine, Puppet, and Chef.  It seems like CF Engine is losing popularity.  You hear about Puppet and Chef all the time.  So we picked Puppet because we were using it already.  So deployment automation for those who aren't familiar  with it is the idea is you want to automate the tasks in  setting up a server.  Sort of on the non-automated Postgres installation, sort  of the extreme case, you have to install a database server.  You download the tar file from the Postgres website.  You uncompress it.  You un-configure.  You un-make.  You un-make install.  You run an IDDB.  You copy over a file into the init script there, and so on.  This gets very tedious and long, particularly if you're  doing this for more than one server.  Anything that is more automated than what I just  described is a form of deployment automation.  Some people have been using shell scripts for this type of  stuff for decades.  This is deployment automation.  Some of the deployment automation packages like  Puppet and Chef are frameworks for doing these common tasks  easier and with more reuse than if you were writing your  own shell script.  We had a couple of sort of general goals for our  deployment automation project.  We wanted servers to look the same, which doesn't mean we  wanted them to be identical.  Think of a web server where you have 10 web servers for the  same web app, all doing exactly the same thing behind a  load balancer.  There, your servers are pretty much identical.  For us, while we have 100 databases, none of them are  really identical.  Many of them are running different versions of  Postgres.  Many of them are running different database schemas for  completely different products.  And then others are configured essentially the same.  So we needed a way to keep things that look mostly the  same, but manage the differences.  So before we did automation back on AX, we hit an instance  where the directory we were writing, the Postgres logs,  PGLog, on a couple of servers was one level higher in the  directory tree in the file system than it was 90% of  the places.  How did we discover this?  It was in the middle of the maintenance.  A DBA was doing something, and the log directory was not  where he expected it to be.  He might have been running a script to check something, and  it wasn't where the script expected it.  And this is not the type of thing you like discovering in  the middle of the maintenance, because, yeah, you can deal  with the other directory moving, but it's a pain.  We wanted ways of managing this and preventing that type  of situation.  We also wanted consistency between our production and  our development in our QA environments.  As we moved to more deployment automation and more  automated tasks in general, we wanted to be able to  confidently test our scripts in QA and staging and know that  they have a good chance of working in production.  While we were on AIX, we had, I think, three database  servers in our staging environment and maybe five in  our QA environment for all of our different products, all  our different projects and systems.  So out of those 100 databases to test something we might  have in QA, we might have five servers to choose from.  So QA would install different versions of Postgres on the  same server, create multiple backends listening on different  ports, and do whatever they needed to do so they could  load the data and test the application functionality on  the server with lots of other QA activities going on.  This was fine for testing application functionality, but  it tended to have some problems.  Particularly if you wanted to test maintenance type tasks,  it didn't work so well.  Now I'll admit, while this was one of our goals, we've not  been so successful yet in making our QA environments look  as close to production as I would like.  We're still sort of working on making progress on that.  We also wanted to reduce manual commands.  Before I gave the example of doing the install manually, if  you have to upgrade Postgres on 100 servers, maybe you have  to change the version rather quickly because there's a  security patch you need to roll out, the less typing you  have to do on each of your 100 servers, the better.  So we wanted to reduce that, and deployment  automation was a good way.  As I mentioned, we've picked Puppet.  Puppet is an open source project with a commercial  addition that you can buy.  You can get Puppet from PuppetLabs.com.  PuppetLabs does most of the maintenance of Puppet.  The open source version or community version of Puppet  will probably do everything you want.  We initially launched with the open source version when we  did our stuff.  And then we bought a commercial support contract with  PuppetLabs, mostly for access to their  professional services.  And then we migrated to the enterprise version.  It has a bit fancier, gooey, a bit nicer packaging.  But most everything I talk about in this talk applies to  both the community version and the commercial version.  So a bit about how Puppet works.  In Puppet, you have a Puppet Master.  The Puppet Master is a server that will have all of your  Puppet module source code.  And a list of what goes on which server.  Then you're going to have your servers, which might be  database servers, web servers, LDAP servers.  These servers do your real work.  And they will run a program called the Puppet agent.  The Puppet agent contacts the Puppet Master and asks the  Puppet Master for instructions on what it should do.  And the Puppet Master should stuff down to the agent to  make changes on the agent server.  Some terminology that I'll refer to.  In Puppet, when you tell a server what should be deployed  on it, that's the node definition.  And the node definitions consist of a set of classes.  So for a particular node, you might be saying we're  deploying the PGSGrel class and the Sloanee class.  The Puppet Master will take this node definition, which is  the list of classes, and the actual source code for these  classes, and batch them up together and produce a  catalog.  This catalog is instructions for the agent on what to do and  how the agent should make the server look.  This catalog gets shipped to the agent.  What's a Puppet class?  This is a very basic Puppet class.  Puppet classes can instantiate other Puppet classes.  They can instantiate Puppet resources, and they can have  things like variables being set and basic control  structures like if statements.  A Puppet resource is something in Puppet that you  instantiate to do something.  Puppet ships with a whole bunch of default pre-provided  resource types.  One of them is the Package resource.  The Package resource installs packages with your  operating systems package management functions.  So this particular example will create an instance of a  package resource to install the Postgres URL server  package using YAM.  So on a Red Hat system, this will install Postgres.  And that's what the class will do.  Puppet is also declarative.  Another resource type Puppet ships with is the File  resource.  The File resource manages files in the File system.  So the idea is you declare that a file should exist on  your server, TemphellerWorld, and you declare what the  content of that file should be.  So in this example, I'm saying I want to have a file  TemphellerWorld, and it should contain the content  Welcome to Ottawa.  When the Puppet agent runs, if there's no file in Temp  called HelloWorld, it'll create it.  And same, Welcome to Ottawa.  When the Puppet agent runs, and it finds a file in  TemphellerWorld, but it says Welcome to Toronto, Puppet  will change that file to say Welcome to Ottawa.  So you run Puppet once, it makes some changes.  You run Puppet again, and you haven't changed anything  else.  Puppet should really do nothing.  It checks to see that everything that you've declared  is still true, and it'll do nothing.  If something has changed, Puppet will make what you  declared true to be true again.  And this is the philosophy of Puppet.  There's ways you can do things that don't follow this  philosophy, but you're going to get into trouble.  Puppet is meant to be declarative, and you want to  build your Puppet classes to do things in this  declarative fashion.  So the way we deploy Puppet at Affilius is we have a Puppet  master, and we have our servers, database servers,  Apache servers, Tomcans servers, et cetera.  We write our Puppet classes, and we check them into a  Git repository.  The Git server we push things to has some hooks that push  things onto the Puppet master to make those  codes available.  When we do things in Git on a particular branch, that  branch is then pushed to the Puppet master.  And we can have multiple Git branches of Puppet classes  available at once.  This allows us to have different servers running  different versions of our Puppet classes, which is very  useful if, let's say, you want to test your Puppet  changes in QA before you want to push them to production.  You tell Puppet that you want it to use the test 123 branch  on a QA server while it's using the prod branch on a  production server.  Git allows us to do this rather nicely with the hooks.  Then, before I talk about a node definition, you can write  a file in Puppet that says for server one, it's running  classes Postgres and Sloney, server two, it's running  classes Tomcat and your Java app.  But if you have a lot of servers, a one file with all  of this gets a bit unwieldy.  We already use LDAP for managing a lot of our  infrastructure, what's servers there, what the  names are, et cetera.  So with Puppet, you can have what's called an  external node classifier.  And an external node classifier is a program that  Puppet will contact during the run that says for server  123, what classes should I install here?  And there's a bunch of pre-written external node  classifiers, one of them interfaces with LDAP.  And that's what we do.  So in LDAP for server 123, we list the Puppet classes  as attributes in LDAP that should be installed on a  server.  We also list the branch from Git, the environment, we  call it, that we want to run on that those classes of.  So a server contacts the Puppet master, Puppet master  contacts LDAP.  It gets what it should install.  The Puppet master builds up the catalog, ships it to our  servers.  Couple slides ago, I showed you a web page with a  management console.  That's the Puppet Enterprise Management Console.  We can launch Puppet run on a bunch of servers.  So I can say all of our servers in Texas, we want to do a  Puppet run now that are database servers.  I select them on the GUI, launch Puppet.  It'll go run Puppet now.  And I can view the results in a report screen.  If you're not using a fancy GUI, you can launch Puppet on  the command line of each of your servers.  You could write shell scripts to log on into each of your  servers and do this.  A lot of people run Puppet on a interval, so they have Puppet  automatically runs once an hour or once a day.  We're not doing this yet.  We might eventually get to that point.  That's sort of the other way you can deploy Puppet.  So it always runs.  Make sure that the things you declare in your  manifest are true.  We had some specific goals for deploying our databases.  We wanted to deploy binaries.  We wanted to deploy cont files and manage cont files.  We wanted to handle the initdb because as we were rolling  out our 100 databases onto these new servers, we wanted to  get that done very quickly and easily.  We also have a lot of Trontabs that we have to manage.  Each of these things I'll talk about in more detail in a  couple of slides.  And we did some things with Sloney that are easier to  manage with Puppet as well.  So there is a module for Puppet to manage Postgres.  Puppet has these different modules,  collections of classes.  Many of them are available on GitHub projects and Puppet  Labs websites.  There's one for Postgres.  It does many of the things I talked about in the previous  slide, starting, stopping Postgres, handling your  cont files.  You can get this module at this URL.  We don't actually use it.  The reason being we had a perceived need to install  multiple minor versions of Postgres on the same server.  So we thought we wanted to install a 9.2.4 and a 9.2.5 on  server at the same time and have them running at the same  time.  The Postgres RPMs that you get from the community don't  really do that.  So we had to build our own RPMs.  And it seemed to be easier to just write our own class for  managing Postgres than it would be to modify the  community classes here to do that.  But I would recommend you take this as a starting point and  see if it will meet your needs.  So what we instead do is we have a resource called the  PGSQL cluster resource.  It manages a single data deer.  So it will launch, create the data deer, perform the NITDB,  install the cont files we want here.  And then we have a whole bunch of infrastructure that sits  above this layer for managing actually what cont files and  what data deers and what users go to a particular server.  So one of the more interesting things and more common  things you're going to want to do is manage files like your  Postgres cont file.  Puppet has a templating language.  It's the Ruby templating language.  If you've ever used Ruby on Rails and done web apps, HTML,  it's the same templating language that that uses.  Puppet is actually written in Ruby.  And there are many places in Puppet where you can break out  in your classes and basically write pure Ruby code to do  things.  And often you will need to do that when there's no easy way  to accomplish what you want in Puppet.  So the way these templates works is you might have a  Postgres cont and you want shared  buffers to be variable.  So you put that in and instead of a specific value for  shared buffers like 32 megs, you would say,  within these square brackets or triangle brackets,  shared buffers.  And shared buffers is a variable and that will get  evaluated out of runtime.  So when you're deploying your catalog, that variable will be  substituted with, let's say, 300 megs.  So the template, the ERB, gets turned into a specific  confile with the variable substitution applied.  The way you would use that template is using the same  file resource we discussed before.  You set the variables in your Puppet classes.  So shared buffers might be 300 megs and we set the  template we want to use.  And then we're saying this file is for the Postgres conf,  which should be in the data tier.  And the content of this file, instead of being a static  string like my previous slide, it should use the  template and we pass it the name of the template.  Puppet then goes, evaluates the template in the context of  the variables you've set in the class and gives you your  result.  You can also do ordering on Puppet resources.  So here we're saying that we also want to make sure we run  this file after we've done the initdb.  I have another package called exec, which execs the initdb  command, or another resource that execs the initdb command.  And we're just saying, in order to run the Postgres file  conf, we require the initdb to have been completed.  If you install your template before the initdb, it'll  get overrided.  So in Puppet, each resource instance has to be  uniquely named.  So if I had a different Puppet class that was installed on  my server that had this string here with data tier being  the same value, it would be a conflict.  Puppet would say, oh, this file or resource in this class  wants Postgres conf.  And this file, also this other class is also defining  Postgres conf as the same thing.  Which do I choose?  I don't know.  I'm confused.  I'm going to error out.  So every resource and every class has a name.  In this case, the name of the file resource I'm instantiating,  the instance name, is the path to my data tier  slash Postgres conf.  The class that I'm including this to also has a name.  And the $name evaluates to the name of the current instance  you're running at for the class or resource you're inside of.  Does that answer your question?  So I have no idea.  That's probably how I named it.  There's no good reason.  The $name will give you the uniqueness.  It could even be a typo on the slide or a typo in my code.  So some parameters in your Postgres conf file, you know,  are going to want to be based on information on the server  you're actually deploying to.  A good example is the IP address.  Often you will do listen address equals  star to listen on all your Ethernet devices.  But you might only want to listen on the first Ethernet device.  Postgres conf requires you to type in the IP address  of that first device.  But this is something you can automate.  Puppet has a program called Factor that  runs as part of the Puppet agent run that  collects facts about your system and ships them back  to the Puppet master to make these facts available as  variables during the Puppet run.  For example, the IP address of the first Ethernet device  and the second Ethernet device and all your other Ethernet  devices are available as facts.  The amount of memory on your machine is available as a fact.  The operating system you're running is available as a fact.  And you can also write custom facts.  You can write a bit of Ruby code that  will run on the server, the agent server,  as part of the agent run to do whatever you want to compute  a value that gets shipped back to the Puppet master  as a variable.  Then inside of your templates, you  look up these variables and use them as values.  Some variables you're using are coming from your agent servers.  But other variables you're going to want to control and set  as configuration to manage your configuration  and get a lot of usability between your Puppet classes.  So you have your Puppet classes and you  have 100 database servers.  Ideally, you would have one Puppet class for all your database  servers to maximize code reuse.  But if you're doing this, you need  to take the configuration parts out of your class  and put them somewhere else.  Puppet has a library or facility called  Hira, which is a data lookup library for looking up  data in a hierarchical fashion.  What it allows you to do is move your data out of your classes  into these YAML files.  And you can then do lookups and vary your configuration  hierarchically.  So the way it works is you might have a DB server class  and you might want to find out what ports should Postgres  be running on.  And it might be that for different products  you're selling or doing, you have different ports  you want your Postgres to be using.  So you'd say my application is Postgres, my product is org,  and the Hira, what is my Postgres port?  And Hira will come back with the port it should use.  How does Hira do this?  First, you have to define a hierarchy.  This is sort of an example of a hierarchy  where what I'm saying is for a particular fully qualified  domain name for my application, which is Postgres,  I want to be able to set things on a particular server.  But other things for Postgres, I want  to set for the entire domain.  So for all of my tor.affilias.info servers,  I want to set some stuff there.  And then texas.affilias.info, I want to set other values.  But some environments, some settings  might not vary based on where your server is located  or the domain name of your server.  Some of our settings vary based on what we call environment  type, which is whether it's a QA server, a production server,  or a staging server.  So we've made that another variable.  And then other things, like maybe the port,  varies based on the product.  Everything in yellow is a variable that we've defined.  And we've decided these are the switches, the control things  we want to use to build our configuration trees.  You can pick other things that work for you.  The important thing is, if it's a variable  in your Hira configuration, this variable  should be set either by factor or explicitly  as we did in our slide here.  So how does this then work?  You then write some YAML files.  YAML is a language for writing data.  It's kind of like JSON.  Hira, its default is YAML, but there are other backends  you could use.  And you write these files that will contain your configuration  in a tree structure on your file system.  So this tree structure matches the structure  of my previous slide.  So some values, we might have a server called db1.tor.aphilius.  And we set some values in Postgres here.  So when I go look up my port, Hira  will first check this file, this Postgres file, for db1,  because that's the fully qualified domain name.  It doesn't find the value.  It moves on, oh, this is a production server.  Have I defined it at the production level?  No, I don't see it there.  Oh, this server is, the product was org,  because that's the variable value I set to the variable  two slides ago.  Yes, I see a port value.  Hira will return that value.  If it found the port value at the top,  the first file it looked, that's what it would return.  Hira will walk through your hierarchy in the order  you define it, looking for the values you've set.  What do one of these YAML files look like?  They're pretty simple.  You have a key and a value.  So the key in this case is Postgres port.  The value is 5 or 3, 2.  You can also build more complicated structures.  So at the bottom here, we have our dbList, which is a hash.  So the key is Postgres dbList.  The value is a hash.  The hash, the key of a hash is PgCon,  which is, let's say, the database name.  And the value that points to is the owner of the database.  There's another value in the hash for a different conference  with a different owner.  We then have some puppet code that  walks through this hash creating databases with create database,  or with the create database SQL command.  For every database we've defined in the dbList,  and setting the role in Postgres that  owns the database to be the role we've set here.  We have some databases where, or some clusters,  I guess I should say, where the databases in the cluster vary.  For a lot of miscellaneous things,  we'll have one database with four or five different databases  in it, and another cluster will have three of those four,  but not all of them.  We use this to manage what databases go on a server,  and it works pretty well.  As you manage your Postgres configuration,  you're going to need to come up with a strategy  for differentiating different confiles and grouping things.  I'm going to talk about two strategies.  This is the one we actually use mostly.  For every major version of Postgres,  we start off, and we do the initDB,  and we get the default confile for that version of Postgres.  And then we start to change it.  There are some things that we change in the confile  that apply to all of our database servers.  And then there are some things that differ  based on the class of database.  Some of the databases are small.  They're not running on big systems.  They don't have a lot of activity.  So we created a grouping of servers called small.  And we have confiles for 9.0, 9.1, 9.2, et cetera, that are small.  Other databases do batch processing systems.  These systems typically have few connections,  use lots of memory.  So we set up a set of files for these batch systems.  Max connections are low, workman is high,  min duration for logging long running queries is rather high,  things are appropriate for batch jobs.  Then our OLTP systems, we tend to do a lot of very high max  connections, much smaller workman,  because max connections are so high,  and we have a lot of connections.  And our min duration, we consider a period running long,  if it's 100 milliseconds on these systems.  So we log it.  While on our batch processing systems,  the query of a minute is not long at all.  So we built things into groups, and we  try to manage things on these groups.  And even though different applications  are using the same group, we try to keep it together.  Yes?  Do you get a lot of value out of the type of files  that are major works?  You probably could.  The question for the audio was, do we  get a lot of value about having different version templates  for each major version?  And I like the fact that you can see the values  that we've not changed.  And sometimes the default value is changed in Postgres.  And we sort of find that's helpful.  But it might work well for others not to do this.  The overhead of how there's not that many major versions  of Postgres, so the overhead of this is not a big deal.  And there are settings we've changed  on different versions of Postgres.  Sometimes there are things that are appropriate for 8.3,  that are not appropriate for 9.2, as an example.  And you could conditionalize some of this stuff, which  sort of brings me to my next slide.  Instead of having one file for class of systems,  you could have one file for each major version of Postgres,  make everything that you might change a variable,  and then set up a higher level or use higher  to manage the different groupings.  This is probably a better way of doing it.  And I'd like to move from having one confile per class  to one confile per version.  What Peter suggested is, I guess, another level  we could go to of just one confire for all versions.  I'm hoping to go to this level.  So we would say then that our shared buffers and work  mem are all variables, and at some higher level, we would say,  for big, it would be certain values,  and for small, it would be other values,  and some systems we would override.  Another thing we do a fair amount of is crontabs.  We have a lot of crontabs for things like backups, purging  jobs, both purging things in the database, purging log files  and the file systems, et cetera.  And so while it wasn't a big deal for us  to rewrite that module for installing Postgres  and doing the NITDB, because probably like 75% of our puppet  classes in terms of volume of code  deals with all this other stuff that is not  reusable between other places with respect  to managing our crontabs and what files we go.  We actually run most of our crontabs.  We have database servers, and we have what we call database  application servers.  A database application server does not  run the Postgres daemon.  It has Postgres installed, and it runs crontabs and our slawns  and other things that connect to a database and do maintenance.  We do this to move load off of our database servers  onto other servers, because our backups  between the BZIP and some of our backups we then encrypt  and send places.  It's actually a fair amount of CPU load,  and getting this stuff off of a database is helpful we found.  So managing these crontabs is a big part of what we do.  Question?  Some questions say capital management  doesn't even use these words for it.  That is probably open office misbehaving.  Yeah, so to do the cron, to manage crons,  Puppet has a cron resource type, and the cron resource type  manages your crontab.  So for a particular line in your crontab,  you will declare a instance of the cron resource type,  saying I want a cron for a backup under bar name.  This is the name I was talking about again,  where because we have so many different servers,  and you cannot have duplicate instances of a cron.  So name might be the product name in this case.  Our class might be named DappServerOrg.  And we would, so cron would then create a crontab with a name,  which will be a comment in the crontab file,  DappServerBackupUnderBarOrg.  And it will put the command we've specified here.  And the command can have a variable.  So this way, we have the same Puppet class or Puppet resource  that lets us back up all of our different products.  And some of our products, we might  be running the backup at one time, and others,  we might make the minute be a variable,  instead of 0, so we can control the time.  We have a lot of different products and crontabs  that do different things.  And we've been managing those crontabs with Puppet.  And we have switches to say, which crontab  should run on which products?  And we have switches in our higher files  to control what options we pass to them.  And we have other switches to control what servers a backup  should be launched from.  So it gets kind of complicated, but higher  has turned out to be a pretty good way of managing it.  And it definitely is working better  than trying to manage all of these crontabs  directly with CVS or SVN, pushing them everywhere.  Another thing we use Puppet to help us do  is manage some of our Sloney stuff.  For those of you who don't know what Sloney is,  it's a replication system.  And to configure Sloney, Sloney has a command line language  called Slonic.  Slonic is how you tell Sloney what it is you want  Sloney to do.  And this is an example of a typical Sloney script.  They start out, you name the cluster,  and these different clusters in your environment  will have different names.  Then you tell Sloney how to contact each of the nodes  in this cluster.  And you have to provide the host name and users, et cetera,  in the Slonic file.  Then you actually do interesting things  like your Sloney commands.  This file will not work in both the QA and production  environments, because at the top, we're specifying host names.  And the host names are not the same in QA and in production.  So how do we solve this problem with Puppet?  We have a template in Puppet that generates Sloney  through evils.  These are files basically contain that top part  of our Slonic commands that are the same for any command.  We go, oh, the cluster name should be from the variable  cluster name.  Then we do some more advanced templating stuff.  We can do loops in the templates.  We loop over all of our nodes.  Slonconf is a hash that comes from Hira.  And we loop over it.  We get the node ID, which is the key, and some values.  And the values is another hash.  And the values might have the port.  It might have the host name.  We could put users and passwords in this hash as well.  And we write out that admin con info line  from the previous slide using our Puppet template.  Sloney also has the idea of the variable  that you can then reference later.  In addition to the con info, this same information  is available as a Slonic variable.  So we can reference that path to a particular host  later in a Sloney store path command.  This is an example of how you can build up things in Puppet  to separate the stuff that is the same on all servers,  let's say the set add table from stuff that varies.  So then when we run our Slonic commands on QA or production,  we first include this Purembo file.  Then we do our set add table or whatever it is  we actually want to do.  So how do our YAML files in Hira look for something like this?  That hash I showed you that we were looping over  will look like this.  The key might be put to a Slon nodes.  And we'll do a Hira lookup and assign that  to a variable Slonconf.  And then we have the node ID 9181.  That's the first level of the hash.  That contains another hash that has host and port.  And then we have another entry that's a different host  and a different port for a different Sloney node ID.  This might be our production conf.  And then for QA, we would have a different YAML file  that had different values for these things  that would generate a different Purembo  on the server when we ran Puppet.  So I'll talk a bit about what has been a bit challenging for us.  One issue is removing stuff.  You declare these Puppet classes.  You run the Puppet agent.  It installs a bunch of things.  Then you decide, oh, I no longer want  to be running this database on a particular server.  Maybe you decide to move it.  Or maybe you're decommissioning it.  You don't want to throw away the whole server.  And if you were doing this for web applications,  for a web server, often what people do  is they will just build a new web server  and replace the old web server with the new one  behind their load balancer and throw away the old one.  With virtualization, this is very easy  if you're doing web servers.  Because you can just throw it away and replace it  with another one if you're using a load balancer.  This doesn't work so well with databases,  because you can't really, unless you're  throwing away the entire server,  you have to keep that data.  If you're only throwing away one of the databases  on your server and keeping the other one,  you can't throw away the data of the one you're  keeping by reprovisioning.  So what we've found we needed to do  is for every class that does something in Puppet,  we've had to write a second class that undoes it.  So we have these removal classes and removal resources.  So we'll install things with the install one.  And then when we decide, oh, we want to actually remove this  from a server, instead of having that server list,  the install class, the post-dress class on the server,  it might list the post-dress remove class in the manifest,  in the node list, I should say.  So it undoes things.  Testing this is a bit of a pain, because you  have to install things, make sure things are how they look,  right, deploy your removal class,  and then look for any changes, any leftover things.  And usually, when you're installing something,  you want to get it out there.  You're not worried about removal  when you're writing the Puppet class initially.  You just get it out there.  And it's only when you uninstall something  that you tend to bother about writing these removal classes.  No, yep.  That's normally handled by attributes and reports, but file  reports are not necessarily to ensure that the Puppet is  not absent.  Yeah, so Puppet, for example, the file resource  has an insurer present and insurer absent.  So just like we were ensuring that the file was there  in the examples I gave, you can also add an attribute,  insurer absent, where Puppet will make sure the file is not  there.  And for your resources, that's how you remove things.  And one way of doing this, instead  of having a separate class for removals,  you can have an insurer absent mode that will undo it.  We've mostly not gone that route.  Maybe that is a better route to go,  but I've been doing separate classes.  I don't have, off the top of my head, a good reason why.  But either case, you have to actually think about what  should happen in a removal and spend the time writing  the code to remove that.  We also have had some challenges.  Yes?  Could you also just stop the full-fledged learning  in the technical distribution?  Well, the issue is, so let's say we have Postgres,  and we have two databases in Postgres,  Database 1 and Database 2.  And if you want to remove Database 1, but not Database 2,  you can't do that.  But even if we have a class that removes Postgres,  and it stops Postgres, it deletes the DataDeer.  Because often, if we're deleting something,  we want that space free.  We don't want that DataDeer laying around,  because we're likely going to deploy something else  on that server.  So yeah, you can just implement half the problem,  or half the solution by just stopping Postgres,  and not completely cleaning it up.  But eventually, you're going to want to completely clean things  up.  So password management is also a challenge.  If you put your passwords in your higher files,  and you put these into Git, and lots of people  have access to your Git branches,  you just exposed your passwords to lots of people,  that's not so good.  So when we first tackled this, we actually weren't using  Hira.  And I wrote some custom Ruby functions that ran on the Puppet  Master to look up passwords.  It was sort of like Hira, but custom stuff that just  look up passwords in the passwords were only on our Puppet  Master, that only our systemins had access to.  And that worked OK, except adding and removing passwords  to Hydraksis and Min to do that.  Then Hira, when we moved to Hira,  we discovered that there were different back ends for Hira.  Hira has, in addition to the YAML back end,  it has a back end that works on GPG encrypted files.  So you write up your YAML files, and you encrypt them,  and you can check them into maybe a different source  repository, and get them somehow deployed onto the Puppet Master.  And then you deploy decryption keys on the Puppet Master  that you control fairly tightly.  And then the Puppet Master can decrypt these YAML files  to look up passwords and push them down to into the catalog  so they get deployed on your server.  You can do this for writing things like writing  VG pass files, et cetera.  Changing passwords is also a bit of a challenge  in that if you want to change your database user  password, your super user password,  you might need both the old and the new password  available at once.  You need the old password to have your PSGrelSmart login,  and you need a new password to set it to something different.  This is something we've not yet solved.  One way is to have other, one way of doing this  is to have the password available as two variables.  Another way is to have a different user  that you're using for changing these passwords.  But we haven't actually implemented a solution.  There are some next steps we're looking at doing.  One, I spoke about how we're managing databases  as we define the list of databases we create.  We're not doing this for our typical database login users.  Right now, we're still manually logging into our server  as issuing a created role or a dropped role.  And it's getting kind of tedious.  I'd like to sort of have Puppet manage the login users,  the roles for our databases.  I think that's quite doable.  Also, when we push a compile change,  we are not restarting Postgres.  So we change, let's say, shared buffers  in our template or in Hira.  We push this down.  It gets, the compile has changed on our database server,  but it has no effect because we've not restarted Postgres.  You could tell Puppet to just restart Postgres,  but that means Postgres is going to be shut down  when Puppet runs.  If you're running Puppet every hour,  you probably don't want to have your database go down  and planned if it's an important database.  So I still have to figure out some good techniques  for managing, restarting servers that can be restarted,  sort of at whim, because a lot of our replicas  actually can be restarted at whim,  because they're not yet being used for things,  but not restarting servers that can't be restarted at whim.  And then getting an idea about when we've pushed something  to a server that doesn't take effect until a restart,  because changing the compile, but not having it take effect,  is a good way to confuse yourself.  So that's sort of on my next list of 2D stuff for Puppet.  Any questions?  You mentioned the tools for pushing changes from Git  to your public master,  because they were branch-aligned.  Yes.  Are they kernel tools or are they open source?  No, we use, I think it's Git to Lite as our Git server.  And then we've written custom hooks.  We've written something that involves RabbitMQ, I believe.  I think.  Those hooks are custom.  Yeah, those hooks are custom.  We might be able to release them or open source them  if there's interest.  I can talk to the guy who wrote them  and see what they look like  and maybe push them to GitHub.  Any other questions?  Okay, thank you.  Thank you.  Thank you.  Thank you.
Elephant_Puppets_2_0_47.png Elephant_Puppets_2_48_95.png Elephant_Puppets_2_96_155.png Elephant_Puppets_2_156_203.png Elephant_Puppets_2_204_251.png Elephant_Puppets_2_252_299.png Elephant_Puppets_2_300_347.png Elephant_Puppets_2_348_395.png Elephant_Puppets_2_396_443.png Elephant_Puppets_2_444_491.png Elephant_Puppets_2_492_539.png Elephant_Puppets_2_540_587.png Elephant_Puppets_2_588_2747.png Elephant_Puppets_2_2748_4487.png Elephant_Puppets_2_4488_4535.png Elephant_Puppets_2_4536_4583.png Elephant_Puppets_2_4584_4631.png Elephant_Puppets_2_4632_4739.png Elephant_Puppets_2_4740_4823.png Elephant_Puppets_2_4824_4871.png Elephant_Puppets_2_4872_5051.png Elephant_Puppets_2_5052_5111.png Elephant_Puppets_2_5112_5327.png Elephant_Puppets_2_5328_5519.png Elephant_Puppets_2_5520_5663.png Elephant_Puppets_2_5664_5723.png Elephant_Puppets_2_5724_5807.png Elephant_Puppets_2_5808_5891.png Elephant_Puppets_2_5892_6011.png Elephant_Puppets_2_6012_6071.png Elephant_Puppets_2_6072_6155.png Elephant_Puppets_2_6156_6251.png Elephant_Puppets_2_6252_6503.png Elephant_Puppets_2_6504_6575.png Elephant_Puppets_2_6576_6707.png Elephant_Puppets_2_6708_6851.png Elephant_Puppets_2_6852_6983.png Elephant_Puppets_2_6984_7031.png Elephant_Puppets_2_7032_7199.png Elephant_Puppets_2_7200_7259.png Elephant_Puppets_2_7260_7343.png Elephant_Puppets_2_7344_8963.png Elephant_Puppets_2_8964_11231.png Elephant_Puppets_2_11232_13403.png Elephant_Puppets_2_13404_14063.png Elephant_Puppets_2_14064_15407.png Elephant_Puppets_2_15408_16451.png Elephant_Puppets_2_16452_17603.png Elephant_Puppets_2_17604_19031.png Elephant_Puppets_2_19032_21167.png Elephant_Puppets_2_21168_21923.png Elephant_Puppets_2_21924_24071.png Elephant_Puppets_2_24072_26495.png Elephant_Puppets_2_26496_27395.png Elephant_Puppets_2_27396_29315.png Elephant_Puppets_2_29316_30083.png Elephant_Puppets_2_30084_31763.png Elephant_Puppets_2_31764_32087.png Elephant_Puppets_2_32088_36251.png Elephant_Puppets_2_36252_38423.png Elephant_Puppets_2_38424_40043.png Elephant_Puppets_2_40044_40871.png Elephant_Puppets_2_40872_42959.png Elephant_Puppets_2_42960_43055.png Elephant_Puppets_2_43056_45047.png Elephant_Puppets_2_45048_47375.png Elephant_Puppets_2_47376_48239.png Elephant_Puppets_2_48240_50051.png Elephant_Puppets_2_50052_51935.png Elephant_Puppets_2_51936_53507.png Elephant_Puppets_2_53508_59111.png Elephant_Puppets_2_59112_60779.png Elephant_Puppets_2_60780_63251.png Elephant_Puppets_2_63252_64451.png Elephant_Puppets_2_64452_74171.png Elephant_Puppets_2_74172_76523.png Elephant_Puppets_2_76524_76967.png Elephant_Puppets_2_76968_78272.png