Planning for the worst
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Jacob, Alexys
Gelin, Guillaume
Alexys Jacob/Guillaume Gelin - Planning for the worst Sharing our worst production experiences and the tricks, good practices and code we developed to address them. - This talk is about sharing our experience about how we handled production problems on all levels of our applications. We'll begin with common problems, errors and failures and dig on to more obscure ones while sharing concrete tips, good practices and code to address them ! This talk will make you feel the warmth of not being alone facing a problem :)
 So I guess I'll have to ask someone I know to pass around the mic. Because as you may know or remember, this is an interactive talk. So we prepared some stuff to follow around, but we are really hoping to have a discussion. So it's a sharing experience. I guess it should almost be time to start. I don't know. All right. I guess you will agree that we'll now proceed. Okay. So the topic of this talk, as I just told earlier, is an interactive talk. So we're really hoping to share our experience. Why did we want to make this talk? This is Ramnes, I'm Ultrabug, and we work at Numberly. That's where you can find us if you want later to discuss things with us. But to get back on the title of this talk, it's about what happens when. She happens. The main thing in our daily job, we run some pretty heavy throughput. We have web services that gather data for our customers. And we can never be down. Down time is not acceptable. And losing data, which is another story, is not acceptable either. So we've developed over the years some kind of practical reactions. And we have learned to develop and design our infrastructure a bit differently. And we are still learning. That's why it's an interactive talk. Because we don't claim we have the answer for every use case. So we wanted to start with the basic stuff, which will lead, maybe, I hope, to the conversation we'll have. Let's take a simple example, which Guillaume will introduce you to. This is a very basic application like what you could have when you start a company or anything. So you have Enginix, who service GDP requests. You have a Flask application, who handles all the logic stuff. And you put all your data in MongoDB database, for example. And it could be any database. So first example is what happens when your database is down? So in our cases, we have multiple solutions. So, for example, the server could be burning. But if you have that, you can have just a replica set of databases. So if one burns, well, there are still two or three other databases that can take the lead. And, okay, you continue to serve requests. Something else that could happen is that you miss some resources. For example, you don't have RAM anymore. So if you don't have RAM anymore, well, you could trigger some automatic kills. Like with UisGi, for example, do that. You can just say in UisGi, okay, if that process takes more than, I don't know, like one gigabyte, okay, kill it. You could use C groups, like with dockers or anything, just to say, okay, this process just adds that amount of memory. If you don't have any disk anymore, like if the disk burns off, if you have big failures, what could help is a red1, red10, anything. Basically, never run in production, a web application on something that doesn't have a redvill. Another good thing you could have is a distributed file system, like NFS or anything. There is a lot of things you could have. This is a good idea for some use case. For some use case, sometimes it can add some other risk, but that's a choice to take. If you have a server overload, like the database can't handle any more requests because it's already like this full-sloads pool. There's not much you can do except monitoring it, so you know when it happens, and scaling it recently. You can add more servers, so you can handle more requests. If you have some other ideas or some remarks about that, don't hesitate to tell us about it. Like Alex said, it's really an interactive thing. While you get the microphone, I'd like you to raise your hand if your backend database server already crashed your web service applications. I guess you all have experience in these fields. Like I said, we prepared basic stuff like this. We'll get deeper and deeper in between the talk. Hi. I think we all shared the experience. My question would be why don't you use or didn't use any of the standard tools or solutions for those kind of problems? For example, for the list here, Mezos seems to be a good solution. I can answer that. For complexity sake, who doesn't know Mezos? Okay. We lost already a lot of the audience. Just to get back about what it is, and correct me if I'm saying it wrong, it's a cluster service-oriented solution with resource management. So that it can spawn resource somewhere and spawn it somewhere else if the given server was running accidentally dies. But setting up Mezos and managing it is an overhead that you may or may not want to have. Kubernetes is also the same kind of thing by Google. Google platform runs on Kubernetes and it's also maybe a good solution. It depends on the architecture. Here, we took a basic example with no automation whatsoever. Because also we believe that sometimes simplicity is and built-in features of the technology we use are based on a response to making a bigger infrastructure and adding again complexity. Maybe you can save complexity by using right technologies or technologies who under failure in the right way. Also, we won't talk about Mezos or Kubernetes in this talk, but this is really the first example. In the next example, we'll go on bigger architectures. Yeah, so it is my experience that I heard a lot of similar responses from different teams. The thing is sooner or later, they end with a lot of moving parts. Sometimes it's perhaps cheaper to just use something and invest like a week or two instead of having to answer the phone at 3 AM. Yeah, like I said, it really depends on your team and the size of your team or your company. But I really agree with you. I just wanted to say, please don't call plain NFS a distributed file system. You're talking like Gloucester or it will burn you. Yeah, you're right. When we wrote a distributed file system, we had more in mind HDFS, which we use intensively. Okay, in my experience, it's not very hard to avoid hardware failures. We have replication, we have master-slave, we could backup our data. But it's very hard to recover logical failures when we logically corrupt our database or corrupt our MongoDB database and how we could avoid this. Yeah, we'll cover maybe deeper examples that relate to the problem you're talking about. I agree with you. That's only pure hardware failure. Any other hardware failure experience? Hi, so I forgot to mention that mostly with this kind of home-grown solutions, I noticed that they end up with a much more complicated architecture. For example, if you would want to somehow make out of this technology stack some failsafe architecture, then in my experience, teams have ended with multi-master, highly complex MariaDB clusters and whatever. And the solution is simply use salary, use blah framework, just do it. Yeah, we'll get some of those afterwards. You're right. Let's continue. I don't know if I'll be contributing much, but just an anecdote about hardware failures. On this one project that I was only briefly, we had this big data center in Verizon or Amazon or something, but it was in one place in the world and tsunami hit. We'll talk about this later. No, no, no, keep on. Then we thought, yeah, we have to have another one on the other coast of US. Sure. Of course, we'll get to that as well. You just want to see me walking. Yeah. That's because you said you were tired earlier. No, actually, I was... No, actually, I was a little bit late because I was stuck in the EPS meeting. Sorry. These are all server things, but if you have servers, just servers, you're not reachable, so the network is missing. And network is a big problem as well. Thank you. Also, hardware can fail there. Yeah. Very barely. Yeah. That's another possibility. Unreachable backends. Indeed. That's maybe what occurs most than a server burning, a burning server, actually. The first thing that comes to my mind with unreachable backends is a CIS admin guy who tripped over the cables. True story. I'm sorry. Not me, man. Anyway. The first thing is you have to make him remember. That's human behavior. So maybe find a forefeet for it, an Hello Kitty keyboard for one week, whatever you want, but you have to make him remember. On the hardware side, you can handle also switching and switch failure. The easy answer to this on Linux, for instance, but it also works on Windows, is use network bonding. Now, when you buy a server, they have at least one network card with two ports, use those two ports to and plug them to two different switches. It's really easy to do. When you have a really, really network people, you can do LACP, which is a higher but more resilient and more robust way to do the same thing, aggregating two ports and adding up their bandwidth while adding full tolerance to your networking. That's the principle. Do you have any sharing knowledge about switch or unreachable things? Yeah. Yeah, hi. Is anybody using hardware anymore? Is not everyone running in the cloud or using virtual machines? And you're running it yourself? Yep. Okay, just asking. So, yeah, we do it ourselves. So, yes, we buy everything, we host everything ourselves. And so we have to take care of these kind of problems. And we use GEN2 in production. Yeah, we use GEN2 Linux in production, which maybe a lot of you haven't heard about. We are some kind of crazy people. When I say we're used to shit handling, maybe it's part true. Any other thing to share about network resiliency? Okay. Now let's get a bit deeper in the stack. Having a fail-proof stack can also help when it's not about only the hardware part. On NGINX, there are two things I like to use mostly, is that in NGINX, you can handle back-end HTTP errors. Your upstream gets back to you with a 500 error. What do you do? Do you pass back this 500 error to your client? Or do you try to handle it nicely? I'll show an example of this. If you don't know about this, it's called name location in NGINX. We use this a lot. So when something bad happens, you can see on the bottom error page, whatever it is. We will change the error code to 200 to mask it for the user, while still serving some kind of pixel, because this is a pixel service. And we can even handle if there was a redirect get parameter in the URL, we can still redirect the user to the correct page, even if our back-end didn't or made something terrible. So that's a kind of little trick, location and error page handling. It can really save you from facing, hey, 500 error calls from your clients. We use it quite a lot. You can also serve from cache. So NGINX has cache capabilities. You can say, okay, if I get an error code from my back-end, I will just serve a stale cache response. It's pretty handy as well. On your Flask application, usually you can also use stale caching, which can be handy if your database is down as well. You can have some answers in cache and serve from stale cache. It's better to answer something than an error code. And then you can have multiple techniques to not lose data. This is more focused on not losing data. Spooling and TaskDefero in the basic way is the way that you get some data from your HTTP call. And this data is very important to you. You don't want to be asking your client to send this data twice. Even more when, in our case, it's navigation data. So it's a browser and user browsing a website data. We can have this data back. Spooling it means that whenever we have it, we're not forced to immediately insert it in database. We can take this data, write it somewhere on disk, and have another process be fitted with this data and insert it in a safe way. So if your backend is down, it just can try and try over and over inserting this data, while it was a long time ago since you responded to the client. That's the feral. There are also message queuing technologies such as, maybe you heard about it already here, 0MQ, RabbitMQ, which is more resilient, and stuff like that that can help you get data and make him into a task. That's also the salary philosophy, which is using RabbitMQ as a message broker. The important thing here to me and to us is don't send back error codes to your clients, even if you have, unless you really have to, depends on what you're doing, but you can handle them even on higher levels of your infrastructure. And don't lose data. Don't ask your clients to send again this data. You have ways and means to handle these kind of failures as well and to not ask for it. Do any of you use any of those techniques? Two, three, four, what techniques do you use? Hi, I used to work for a WordPress hosting company and a lot of what we did was basically rely on the reverse HTTP cache to... A lot of the content being served is actually just static content in a way. Think of a lot of people running basically websites. Those glorified blogs are basically just static content after a while. And then the back ends could fail all the time and customers would never notice if you serve from cache. Everyone's happy, the front page is up, the main articles are up. A lot of things are available, especially when your website is basically a content publishing platform, because that content doesn't actually change that much, it's not very dynamic. It works very well. You don't have to wake up every five minutes in the middle of the night drinking out as you can sleep through it and everything's fine. No one will notice except the people trying to publish an article. If it's something really urgent, then they will complain. Any other users who want to share their experience or what they are using it for? Yes, to complete this thing, even on websites like e-commerce, you can use similar techniques, but you need a database to insert the orders or stuff like this because 95% of the content is static, so you can have something like varnish, serve the static content, then use some tiny JavaScript to get the little tiny parts specific to the user, like the username, the name, the basket, etc. I've seen it used to lighten a lot the charge on the back ends. It's very effective and even if you have one or two minutes of downtime for your back ends, your user can still navigate the websites, see all the products, and maybe by the time they add two cards, the back end will be back up and you won't lose any money. Yeah, yeah, yeah. Anyone else? Yeah, I guess the conclusion here is it's better to run even a degraded version of your website or whatever service it is you run than having it fully down. It depends on the use cases. Yeah, it can be argued. You want to argue it? Come on, we are here for it. I want to hear the counterpoint. For example, if you charge money from clients, it's better to say, I cannot then take money after several hours. I guess even that can be argued. So the next thing you can do is of course clustering your application. So if one of your back ends is down or one of your database is down, well, it's still working. So the bad thing is even with a lot of balancers, there's still a single point of failure. So you can always get more redundancy. Even if you have two load balancers and two absences, then the whole data center can go down. So you have to get another data center. So it's kind of an affin loop, but redundancy is good. Okay, so now we can get to your point where your data center burns. Yeah, this photo looks pretty bad. I don't know if it was photoshopped or if it's an actual photo, but I was like, oh my god. I don't want to be the C-Subs coming back after the fire alarm in the data center room. On the upside, actually, it's pretty simple. Have multiple data centers if you run them yourself. If you use the cloud, like it's been suggested, in Amazon you have this notion of availability zone that you should use. Make sure you do remote backups, whatever you do, and test them. In France, we had a recent story where a big company lost its customers' data and they found that they had backups because they were using backups and remote backups, and when they tried to get them back up, it could be said, it failed. There again, I don't want to be the C-Subs over there, and you don't want to, I guess. On the IP routine and connectivity stuff, you have the GP, anycast stuff for having a single IP address accessible all over the world. Something I appreciate also is DNS health checking. For this, we use Route 53 on AWS. Who knows about Route 53? Okay, not so much. It's DNS service from AWS where basically you can have geo-distribution-based DNS responses and add to those DNS records the health checking. So if your data center or whatever happens is down, one of your IP to your web services is down, it will not be answered from DNS queries anymore. It's pretty handy and cheap as well. On the application design, you have to think about geo-distributed applications. Who runs at least one geo-distributed service here? Okay, so I'm not talking about too much people, but still, it's a very interesting thing to do. As a developer, it's a real challenge, as an ops, it's a real challenge. Even when you want this service or this kind of, when I say service, it can be a database service available all around the world. It's also a nice thing to try and achieve. Anyone had this kind of problem already? Where they were relying on everything in one place? Yeah? What happened to you? On the whole data center? Yeah, so obviously I'm not an administrator of the network of some kind, but I was, I seen this all. So my main service was located in one data center and it failed power. And it ended up just in four hours of outage, complete, nothing more. Crucial infrastructure was located there. So we just dialed up our clients and said we're sorry. Afterwards, we apparently distributed. Yeah. What time did it take to distribute the whole thing? I would have to ask my administrators, but I know that certain steps were carried out. Yeah. Just add a little bit because just how easy terrible stuff can happen to a data center, especially if it's not like a big company, the small data center or service provider having a small hosting area, because I used to work in the kind of the same environment. And basically so many things killed the world. We had a story, I mean, I won't name the company, but basically it happened overnight. And the night shift who was monitoring the object just everyone fell asleep suddenly. And they missed all the alarms. And basically when the morning shift came, like all the temperature in the server room where we had a lot of our customers hosting their services was like 70 degrees. We opened all windows and started just like, you know, to try to get somewhere there. But basically a lot of things can go horribly wrong. So choose your data centers carefully and try to really get more of them if it's possible. Yeah. Contracts with your providers. One more? Yeah, yeah, yeah. But I'm just really into the contracts to your providers are not enough usually. And I even provide us say like 99.9999% but not 100%. Yeah, yeah. It's a luckily this was a data center that was only used for the development. But we had an air conditioning that was running really hard and it leaked water into the power outlet that was behind the UPS. So no more uninterruptible power supply and what proved that it was interruptible. It was down for two days. Yeah, it was major, major problem. Yeah, you have to call your clients in the end. So I guess this is this must be very hard to explain. I don't want to be in the sales department at this time. The problem with geo separated distributed locations is not when it goes down. When things come up again. That's right. I've had a few times where services came back up and we had both of them active because they couldn't see each other. But the rest of the world could either see one or the other. Yeah. And then people start using it and when they when they see each other again, then one of them has to decide to be slave again. And weird things happen. Yeah, that's called the split brain situation where your brain doesn't know anymore because you had usually two peers. That's why in clustering in general and in everything you should do is that always be at uneven numbers. And you already know about the voting strategy. Okay. If I am in a disconnectic situation, who is down? I am or is my peer down? If you have only two peers, you have no way to know. You have at least to have three peers to be able to know. If you can't reach any of the two other peers, you're down. That's solid, pretty solid. It's not always solid, but it's pretty solid. At least always thinking uneven numbers, always, whatever you do. Yeah. Okay. So, the co-terror is great, but sometimes real problems are a bit more complicated. And it's not always like dev app stuff. It can be like really coming from your code. That's what we are going to see. So, one day I was walking like normally doing my stuff. And one of our market guys came and told me, hey, run as the client says you can authenticate on the server, on the website. Something's wrong. I was like, okay, I'm going to check the logs. This happened like maybe 10 times per day. So, okay, let's see maybe something's wrong. So, I set the machine. I look at the log and everything's okay. So, well, the client is wrong. What? What did I? So, yeah, the client must be wrong. So, he goes away and I'm happy. Something like one hour later, I'm still walking and the guy come back and tell me that it's still not working for the client. So, I'm exhausted. All right, I'll check the code. Something's wrong. Then I look at my application and I see that. Does anything see something wrong? So, after 30 seconds, I notice that the same email function can fail. So, if the email function fail, well, it returns. Okay, it works. So, yeah, my conclusion to that story is that you have to know your code. Interest situation is great, but code can fail too. Even if you don't like the guy who wrote the code, even if you don't understand the code, if you're a maintainer of something, you have to understand what you're doing and you have to re-factorize when needed. So, the code should never pass suddenly. That's from the Zenf Python. Well, yeah, don't always blame the guy. Sometimes it's easy, like, okay, that's not my fault. It might be another server thing. So, that's why the devil thing is great. So, you can really understand what's happening on your server, even if you're just a developer at the origin. And the other way I run this is true too. So, do any of you have similar situation? What kind of really weird things happen? Okay, now it's going to be brave for developers to raise their hands. I know. I had a city situation where a similar thing where they're saying, oh, this isn't working for the client. She's trying to do all these things. She had a really odd workflow. So, I was thinking, this is all working, all the tests are passing. I go into the website and I'm looking and thinking, this is all working fine. And I ran all the tests and it's all working fine. And what I didn't really realize, it took me like a week to realize where she kept on coming back. The point where I'm like, I use no script. So, I'm happy using the HTML back end and everything so I can find. What I didn't realize was if you enable the JavaScript, JavaScript uses a different API and that's the thing causing the problem. So, make sure to eat your own dog food and use your own API. That's like that got, yeah, it looked like it was working, but I didn't write the code, so it's fine. Yeah, but in the end, you were responsible. Yeah, that's, yeah. So, in Python, we get used to the libraries we're using, raising exceptions. A really common one that doesn't is Memcache. Pretty much every Memcache library will return zero instead of raising exception. So, you need to wrap it or do something like that, but there's four or five places I can think of in different projects that we've been working on where we trace back something to like, why isn't anything working? It's because we think Memcache is working when it's not. Yeah, I tend to like the Memcache Python library because of this, but sometimes it can be in the nightmare, yeah. So, you have always to check about the, it's like the goal. It's like a goal. You have to check the year, you return of the, of the, of the operations you do. Any brave, any other brave developer want to share about this? Oh, yes. We have one here. That's it. That's it through. Yeah, so my example is not related to Python really, but to PHP. Yeah, and now. Who thinks it happens more in PHP? That's brave developers. For my defense, I am not the root decode, but there is a very nasty thing when you try to auto load some file class and you have a syntax error. Then if you do not handle this properly, then PHP dies, returns, and the web server returns blank page with 200. Okay. And, okay, no other way to debug this issue. That's a nice one. We ended up at the WordPress hosting company when I work, we ended up writing some code in the reverse proxy that would detect these sort of situations, the white pages and alert us to it just because it's such a stupid default. Why would you return it to 100 when something's wrong? And yeah, it's horrible to monitor for that. I have to mention Python break, break itself this rule. For example, he's out of height, but exception. And sometimes it's can make very strange things. Switch to Python street. One thing that's not related to Python or any programming language really. I was having a server with a pretty large disk in it. And there were very, very, very many files on that. And then suddenly a developer called in and said, hey, I think the disk is full. So I go look, do a DF. And he says, well, I can't write any files anymore. Okay, touch fall. This full. Yeah, it ran out of our notes. That's something. Yeah, that's a nasty one. That's a nasty one. We often overlook here. Absolutely right. And there are five systems who don't rely on I-nodes. So when you know that your application might spawn a lot of files, think about them. Indeed. I have another story that I forgot to put in the presentation. So I'm going to tell it right now. Basically in my old company, like it was a small startup before I worked on Numboli. So we were trying to get things fast. So basically our web server was running inside a T-max. And sometime when we looked at logs, we were like just crawling on T-max. And one day we were like, oh my God, the web server is not running anymore. And actually it was just T-max. When you scroll, it sent a pose, it sent pose to your application. So the application was down just because T-max was trying, we were trying to take the log with T-max. Don't run T-max in production. And about DevOps philosophy, I don't know what kind of objective we can have. Who works as a DevOps or in DevOps-minded company? I see, I don't know if you're waving to say hello or... Okay. So it's amazing that you work in a DevOps company. Just wait, I come with a microphone. Because we can understand you in the back. Yeah. Why is not the delivery of that? Yeah, we should have thought about that probably a little bit early. The DevOps question is hard because when your managers and everybody talks a lot about DevOps, but they hire a guy who is a DevOps. As a DevOps position, then it gets tricky. So you get back to the silo. So we are developers and the DevOps. So yeah, back to developers and Sysadmins. I was actually a developer who had to run back to our admins to check up why the fuck is Docker not working again. Oh, the elastic search containers clustered with each other. Oh, interesting. So in that sense, I was a DevOps because I needed to worry about the code and about infrastructure mess ups. So it's a tricky word. Yeah, it has a different depth depending on where you stand. Which lead me to a question, who runs Docker in production? And can you share some experience with it? I'm interested. And I'm more interested in when it's failing, obviously. Just one thing, we were actually doing just this new project. So it was more of a proof of concept, but we've already started to get it out to customers. And we were working together with this consultancy who told us how to do Docker and Cloud Foundry if someone knows Cloud Foundry. So our whole infrastructure, we would provide services for Cloud Foundry based on Docker. So we know like spawn, radices, elastic searches, stuff like that. But the Docker cluster was actually one machine with all the containers for all services. So don't do that. Oh, yeah. Okay. Thank you. So I can share a funny story where the Docker demon crashed on the CI server. So you can imagine having like 15 super highly paid developers who are just mixing the air. And it was also fun to debug because, you know, who would have thought of it? Yeah, but to get to the previous point, how much effort would it take to implement something like Supervisor or whatever process that would monitor the demon? Probably not much. Yeah, you're right. You're right. Sometimes we're on own barriers. And we, yeah, I really agree with you. Just about the DevOps thing. Yeah, it's kind of a buzzword, except especially for recruiters. But all we see it's that normally is really like not a single person being in DevOps, but really a team where you have, yeah, people who develops, people who make labs. That's the thing here. But just working together and understanding what the other is doing, it's just very important. And it's also a great time also to a developer to acquire and helping him understand what is not used to do. Another real world problem. Go ahead. Yeah. So we have our statistic on Grafana. So it's a very nice board. One day I was looking at Grafana and I saw the statistics. So it wasn't really important because it was really just the maximum processing time of one of our services. Basically, the average processing time was still very low. So we didn't really investigate that thing. It stayed there for like, I don't know, maybe two or three weeks or even more, maybe, I don't remember. But we never understood really what was happening with why the maximum processing time was so high. And then one day, boom, like what? So my first idea when I saw that the graph was going so low is, oh my God, the service is down. Actually, no, it was still running. But what was happening is, so I searched for like one or two understand what was, what happened. I ended talking with one of the most ops guy in the team and told me, hmm, that's strange because at that moment I deployed an Ansible playbook on one of those servers. And so we looked at the playbook. What was the difference? And the only difference was in the ETC host file. Basically, the NS server was like all the time queried at each database insertion. So sometimes it was just overloaded. So just putting the IPs of the database in the ETC host file of each machine fixed the trick. So, yeah, that was pretty weird. Go ahead. Paul. Sorry. Paul. Here. Paul. Sometimes some other stuff I have to sort of like. And it's not just resolving the database server. You'll be surprised to see how many code is reversing the DNS. So you have to not just forward, but also reverse is happening quite a lot. Yeah. That looks like this problem. To be honest, we felt pretty stupid with this one as well. And this one is pretty interesting because two days ago you made a presentation about using console. Yeah. Sounds very contradictory. Yeah, yeah. And the thing, the question is have you tried to put a local GNS cache, I don't know, a bind 9, something like that, and a TTL around 30 seconds or minutes, something like that. Would it make the trick? Yes, absolutely. What's embarrassing with this is that we also lacked consistency in what we do. You know, on another type of infrastructure, we have local GNS cache, but there we didn't have it. And when Guillaume says the people were working on the Uncivil Playbook is also to stop normalizing all of these. So yeah. So maybe we think that we have something in production and it's running for so long that nothing can happen to it. And we tend maybe sometimes to forget about its resiliency or performance or just applying the latest of your knowledge, just for the sake that it's running, I don't care, or I don't need to bother so much unless something weird happens. In this case, it was good news. You know, we were satisfied with this shitty processing time. But on another type of application, it might be not so. I think that one good trick is to always profile your applications at least once. And I recently used VMPro from PyPyGuys and it actually just slowed down my service at around 5%. So it's actually viable to just switch one instance and check what actually your code is doing. Actually, in that situation, I profiled the code and I didn't have the same results as Synod's Garfan as others. That's why I was like, what the fuck? It's not working as it should. And that's why it wasn't really important. So like I said, we just let it go. Well, it was a good surprise when it was fixed. We came up only with embarrassing examples, so you feel more comfortable sharing this. I have a comment regarding performance monitoring tools. You really have to configure it properly. We had a situation that response time was between 30 and 60 seconds and it was caused by uploading files. For example, 5GB files was uploaded by a few people and it increased the average response time. It was the same kind of problem. Sometimes our metrics ever goes down and then we think, oh my god, my application is done. But it's running. Yeah. Who is using, who is not using a metric system? Who doesn't do metrics on their applications? Who doesn't have this kind of graph? Nobody. You all have it. Yeah, I see you again waving people. Question. Yeah, yeah. You can't just stand in front of the camera and hold your hands. Question, I was precisely going to ask this tension of this kind of question. Have you managed, Nien, to put in the graph, in Grafana some kind of percentile graphing so you can know 90%, 95% percentile and 99% percentile of the response time? So you all know that kind of problem. We have been trying, we have a hard problem with this. We are deploying Promethines plus Grafana and using that with elastic surgeon, we are having a lot of problems. We really calculate the 99% percentile, the 95% percentile. But have you managed to do this? Basically, we just show what we use in general is a comparison between the current day and the day seven days ago. So it gives a good idea to, is it normal or is it weird? And in using carbon and graphana for the visualization, you also have the annotation feature, which is good, where you can have a bar on your graph saying, okay, and you can plug it to your deployment or continuous delivery stuff. So you can have a bar on your graph saying, okay, from this point on, this is version 2.1. And then you can do also matrix comparison related to code deployment. It's pretty good. In disaster recovery, it's also a good thing to have. So when you know them, you broke something and you can do the same with server provisioning and deployment. At this time, I added a new server. Maybe it has some weird side effects. For the percentiles, if you mentioned you've already got elastic search, if you don't have aggregates but have the actual requests logged there, you can just use Kibana because it has a really nice visualization. If I recall, it gives you the percentages, the percentiles as well. You should get that for free from Kibana. The answer is the problem is to combine it with graphana for the remote audience. Did anyone come with a question? I guess we have four minutes left. So open discussion now. Question. Okay. This is tricky. I'm looking to know if Edmai has some experience with trying to deploy a new version of your backend and only deploy it to, let's say, 5% of your users, try it out, see how it handles, and then go for 100%. Especially in GenX, if you do in GenX, that's really good. So progressive deployment. Anyone? Thank you. Yes, for the same company I was talking about a bit sooner about the e-commerce, we were always rolling up the traffic, but it was with H-Epoxy in front of... I think it was H-Epoxy, then Varnish, then Nginx in that order. And the new servers were rolling up 10% traffic on the new version. Then we had the software error monitoring and all the metrics on this server. We were checking that the response time was not doubling, et cetera. And after a few hours, a few more servers were joining. And at the end of the day, all the traffic was rolled up from the... That's one sort of question. Depending on your stack, we do a lower level of this. We run our Python using UwSGI. And in UwSGI, you have this feature where you have... It's called TouchChain Reload, where your workers are reloaded one by one after... And UwSGI will make sure that the one that is reloaded correctly before reloading the others. So it's a good fail-safe, low-level deployment trick. And on the side note, if you are really, really committed to trying canary releases, which is usually called canary releases when you put the bird and try the mine, if it's pushed through IQVIRNets that resolve this problem in a very reliable way. But it's obviously very... It may be too much complicated for your case, but it has exactly this kind of procedure. When you say, I have a rolling deploy strategy, when I want to keep an number of pots, which is your application deployed, and in an hour, would it begin to increase the number? Yeah. Kubernetes... This feature in Kubernetes is very nice. But Kubernetes still doesn't have Elf Check, right? Kubernetes still doesn't have Elf Check, right? Yeah. Elf Check. Yeah. Oh. Okay. So yes, it has. Okay. Thank you. So that was one of the really bad things that I would add us to the table. Okay. Great. Readiness check. That's right. Thank you, Paul. Yes. I want, especially thank you, because this is like an interactive format, and it's a little experiment, just like to have, not to have only like one-sided talks, just like to have a, on the direction, I want to thank you for like taking a leap of faith in the first year we tried this. So I think please give these guys an extra hand, please. Thank you. Thank you very much. Yeah. Yeah. Lightning talks up at five. Thank you. Thank you.
 So I guess I'll have to ask someone I know to pass around the mic.  Because as you may know or remember, this is an interactive talk.  So we prepared some stuff to follow around, but we are really hoping to have a discussion.  So it's a sharing experience.  I guess it should almost be time to start. I don't know.  All right. I guess you will agree that we'll now proceed.  Okay. So the topic of this talk, as I just told earlier, is an interactive talk.  So we're really hoping to share our experience.  Why did we want to make this talk?  This is Ramnes, I'm Ultrabug, and we work at Numberly.  That's where you can find us if you want later to discuss things with us.  But to get back on the title of this talk, it's about what happens when.  She happens.  The main thing in our daily job, we run some pretty heavy throughput.  We have web services that gather data for our customers.  And we can never be down.  Down time is not acceptable.  And losing data, which is another story, is not acceptable either.  So we've developed over the years some kind of practical reactions.  And we have learned to develop and design our infrastructure a bit differently.  And we are still learning.  That's why it's an interactive talk.  Because we don't claim we have the answer for every use case.  So we wanted to start with the basic stuff, which will lead, maybe, I hope,  to the conversation we'll have.  Let's take a simple example, which Guillaume will introduce you to.  This is a very basic application like what you could have when you start a company or anything.  So you have Enginix, who service GDP requests.  You have a Flask application, who handles all the logic stuff.  And you put all your data in MongoDB database, for example.  And it could be any database.  So first example is what happens when your database is down?  So in our cases, we have multiple solutions.  So, for example, the server could be burning.  But if you have that, you can have just a replica set of databases.  So if one burns, well, there are still two or three other databases that can take the lead.  And, okay, you continue to serve requests.  Something else that could happen is that you miss some resources.  For example, you don't have RAM anymore.  So if you don't have RAM anymore, well, you could trigger some automatic kills.  Like with UisGi, for example, do that.  You can just say in UisGi, okay, if that process takes more than, I don't know, like one gigabyte, okay, kill it.  You could use C groups, like with dockers or anything, just to say, okay, this process just adds that amount of memory.  If you don't have any disk anymore, like if the disk burns off, if you have big failures, what could help is a red1, red10, anything.  Basically, never run in production, a web application on something that doesn't have a redvill.  Another good thing you could have is a distributed file system, like NFS or anything.  There is a lot of things you could have.  This is a good idea for some use case.  For some use case, sometimes it can add some other risk, but that's a choice to take.  If you have a server overload, like the database can't handle any more requests because it's already like this full-sloads pool.  There's not much you can do except monitoring it, so you know when it happens, and scaling it recently.  You can add more servers, so you can handle more requests.  If you have some other ideas or some remarks about that, don't hesitate to tell us about it.  Like Alex said, it's really an interactive thing.  While you get the microphone, I'd like you to raise your hand if your backend database server already crashed your web service applications.  I guess you all have experience in these fields. Like I said, we prepared basic stuff like this.  We'll get deeper and deeper in between the talk.  Hi. I think we all shared the experience. My question would be why don't you use or didn't use any of the standard tools or solutions for those kind of problems?  For example, for the list here, Mezos seems to be a good solution.  I can answer that. For complexity sake, who doesn't know Mezos?  Okay. We lost already a lot of the audience.  Just to get back about what it is, and correct me if I'm saying it wrong, it's a cluster service-oriented solution with resource management.  So that it can spawn resource somewhere and spawn it somewhere else if the given server was running accidentally dies.  But setting up Mezos and managing it is an overhead that you may or may not want to have.  Kubernetes is also the same kind of thing by Google.  Google platform runs on Kubernetes and it's also maybe a good solution.  It depends on the architecture. Here, we took a basic example with no automation whatsoever.  Because also we believe that sometimes simplicity is and built-in features of the technology we use are based on a response to making a bigger infrastructure and adding again complexity.  Maybe you can save complexity by using right technologies or technologies who under failure in the right way.  Also, we won't talk about Mezos or Kubernetes in this talk, but this is really the first example.  In the next example, we'll go on bigger architectures.  Yeah, so it is my experience that I heard a lot of similar responses from different teams.  The thing is sooner or later, they end with a lot of moving parts.  Sometimes it's perhaps cheaper to just use something and invest like a week or two instead of having to answer the phone at 3 AM.  Yeah, like I said, it really depends on your team and the size of your team or your company.  But I really agree with you.  I just wanted to say, please don't call plain NFS a distributed file system.  You're talking like Gloucester or it will burn you.  Yeah, you're right.  When we wrote a distributed file system, we had more in mind HDFS, which we use intensively.  Okay, in my experience, it's not very hard to avoid hardware failures.  We have replication, we have master-slave, we could backup our data.  But it's very hard to recover logical failures when we logically corrupt our database or corrupt our MongoDB database  and how we could avoid this.  Yeah, we'll cover maybe deeper examples that relate to the problem you're talking about.  I agree with you. That's only pure hardware failure.  Any other hardware failure experience?  Hi, so I forgot to mention that mostly with this kind of home-grown solutions,  I noticed that they end up with a much more complicated architecture.  For example, if you would want to somehow make out of this technology stack some failsafe architecture,  then in my experience, teams have ended with multi-master, highly complex MariaDB clusters and whatever.  And the solution is simply use salary, use blah framework, just do it.  Yeah, we'll get some of those afterwards. You're right.  Let's continue.  I don't know if I'll be contributing much, but just an anecdote about hardware failures.  On this one project that I was only briefly, we had this big data center in Verizon or Amazon or something,  but it was in one place in the world and tsunami hit.  We'll talk about this later.  No, no, no, keep on.  Then we thought, yeah, we have to have another one on the other coast of US.  Sure.  Of course, we'll get to that as well.  You just want to see me walking.  Yeah. That's because you said you were tired earlier.  No, actually, I was...  No, actually, I was a little bit late because I was stuck in the EPS meeting. Sorry.  These are all server things, but if you have servers, just servers, you're not reachable, so the network is missing.  And network is a big problem as well.  Thank you.  Also, hardware can fail there.  Yeah.  Very barely.  Yeah. That's another possibility.  Unreachable backends.  Indeed.  That's maybe what occurs most than a server burning, a burning server, actually.  The first thing that comes to my mind with unreachable backends is a CIS admin guy who tripped over the cables.  True story.  I'm sorry.  Not me, man.  Anyway.  The first thing is you have to make him remember.  That's human behavior.  So maybe find a forefeet for it, an Hello Kitty keyboard for one week, whatever you want, but you have to make him remember.  On the hardware side, you can handle also switching and switch failure.  The easy answer to this on Linux, for instance, but it also works on Windows, is use network bonding.  Now, when you buy a server, they have at least one network card with two ports, use those two ports to and plug them to two different switches.  It's really easy to do.  When you have a really, really network people, you can do LACP, which is a higher but more resilient and more robust way to do the same thing,  aggregating two ports and adding up their bandwidth while adding full tolerance to your networking.  That's the principle.  Do you have any sharing knowledge about switch or unreachable things?  Yeah.  Yeah, hi.  Is anybody using hardware anymore?  Is not everyone running in the cloud or using virtual machines?  And you're running it yourself?  Yep.  Okay, just asking.  So, yeah, we do it ourselves.  So, yes, we buy everything, we host everything ourselves.  And so we have to take care of these kind of problems.  And we use GEN2 in production.  Yeah, we use GEN2 Linux in production, which maybe a lot of you haven't heard about.  We are some kind of crazy people.  When I say we're used to shit handling, maybe it's part true.  Any other thing to share about network resiliency?  Okay.  Now let's get a bit deeper in the stack.  Having a fail-proof stack can also help when it's not about only the hardware part.  On NGINX, there are two things I like to use mostly,  is that in NGINX, you can handle back-end HTTP errors.  Your upstream gets back to you with a 500 error.  What do you do?  Do you pass back this 500 error to your client?  Or do you try to handle it nicely?  I'll show an example of this.  If you don't know about this, it's called name location in NGINX.  We use this a lot.  So when something bad happens, you can see on the bottom error page, whatever it is.  We will change the error code to 200 to mask it for the user,  while still serving some kind of pixel, because this is a pixel service.  And we can even handle if there was a redirect get parameter in the URL,  we can still redirect the user to the correct page,  even if our back-end didn't or made something terrible.  So that's a kind of little trick, location and error page handling.  It can really save you from facing, hey, 500 error calls from your clients.  We use it quite a lot.  You can also serve from cache.  So NGINX has cache capabilities.  You can say, okay, if I get an error code from my back-end,  I will just serve a stale cache response.  It's pretty handy as well.  On your Flask application, usually you can also use stale caching,  which can be handy if your database is down as well.  You can have some answers in cache and serve from stale cache.  It's better to answer something than an error code.  And then you can have multiple techniques to not lose data.  This is more focused on not losing data.  Spooling and TaskDefero in the basic way is the way that you get some data from your HTTP call.  And this data is very important to you.  You don't want to be asking your client to send this data twice.  Even more when, in our case, it's navigation data.  So it's a browser and user browsing a website data.  We can have this data back.  Spooling it means that whenever we have it,  we're not forced to immediately insert it in database.  We can take this data, write it somewhere on disk,  and have another process be fitted with this data and insert it in a safe way.  So if your backend is down, it just can try and try over and over inserting this data,  while it was a long time ago since you responded to the client.  That's the feral.  There are also message queuing technologies such as, maybe you heard about it already here,  0MQ, RabbitMQ, which is more resilient,  and stuff like that that can help you get data and make him into a task.  That's also the salary philosophy, which is using RabbitMQ as a message broker.  The important thing here to me and to us is don't send back error codes to your clients,  even if you have, unless you really have to, depends on what you're doing,  but you can handle them even on higher levels of your infrastructure.  And don't lose data.  Don't ask your clients to send again this data.  You have ways and means to handle these kind of failures as well and to not ask for it.  Do any of you use any of those techniques?  Two, three, four, what techniques do you use?  Hi, I used to work for a WordPress hosting company and a lot of what we did was basically rely on the reverse HTTP cache to...  A lot of the content being served is actually just static content in a way.  Think of a lot of people running basically websites.  Those glorified blogs are basically just static content after a while.  And then the back ends could fail all the time and customers would never notice if you serve from cache.  Everyone's happy, the front page is up, the main articles are up.  A lot of things are available, especially when your website is basically a content publishing platform,  because that content doesn't actually change that much, it's not very dynamic.  It works very well.  You don't have to wake up every five minutes in the middle of the night drinking out as you can sleep through it and everything's fine.  No one will notice except the people trying to publish an article.  If it's something really urgent, then they will complain.  Any other users who want to share their experience or what they are using it for?  Yes, to complete this thing, even on websites like e-commerce, you can use similar techniques,  but you need a database to insert the orders or stuff like this because 95% of the content is static,  so you can have something like varnish, serve the static content,  then use some tiny JavaScript to get the little tiny parts specific to the user,  like the username, the name, the basket, etc.  I've seen it used to lighten a lot the charge on the back ends.  It's very effective and even if you have one or two minutes of downtime for your back ends,  your user can still navigate the websites, see all the products,  and maybe by the time they add two cards, the back end will be back up and you won't lose any money.  Yeah, yeah, yeah.  Anyone else?  Yeah, I guess the conclusion here is it's better to run even a degraded version of your website  or whatever service it is you run than having it fully down.  It depends on the use cases.  Yeah, it can be argued.  You want to argue it?  Come on, we are here for it.  I want to hear the counterpoint.  For example, if you charge money from clients, it's better to say,  I cannot then take money after several hours.  I guess even that can be argued.  So the next thing you can do is of course clustering your application.  So if one of your back ends is down or one of your database is down, well, it's still working.  So the bad thing is even with a lot of balancers, there's still a single point of failure.  So you can always get more redundancy.  Even if you have two load balancers and two absences, then the whole data center can go down.  So you have to get another data center.  So it's kind of an affin loop, but redundancy is good.  Okay, so now we can get to your point where your data center burns.  Yeah, this photo looks pretty bad.  I don't know if it was photoshopped or if it's an actual photo, but I was like, oh my god.  I don't want to be the C-Subs coming back after the fire alarm in the data center room.  On the upside, actually, it's pretty simple.  Have multiple data centers if you run them yourself.  If you use the cloud, like it's been suggested, in Amazon you have this notion of availability zone that you should use.  Make sure you do remote backups, whatever you do, and test them.  In France, we had a recent story where a big company lost its customers' data and they found that they had backups  because they were using backups and remote backups, and when they tried to get them back up, it could be said, it failed.  There again, I don't want to be the C-Subs over there, and you don't want to, I guess.  On the IP routine and connectivity stuff, you have the GP, anycast stuff for having a single IP address accessible all over the world.  Something I appreciate also is DNS health checking.  For this, we use Route 53 on AWS.  Who knows about Route 53?  Okay, not so much.  It's DNS service from AWS where basically you can have geo-distribution-based DNS responses  and add to those DNS records the health checking.  So if your data center or whatever happens is down, one of your IP to your web services is down,  it will not be answered from DNS queries anymore.  It's pretty handy and cheap as well.  On the application design, you have to think about geo-distributed applications.  Who runs at least one geo-distributed service here?  Okay, so I'm not talking about too much people, but still, it's a very interesting thing to do.  As a developer, it's a real challenge, as an ops, it's a real challenge.  Even when you want this service or this kind of, when I say service, it can be a database service  available all around the world.  It's also a nice thing to try and achieve.  Anyone had this kind of problem already?  Where they were relying on everything in one place?  Yeah?  What happened to you?  On the whole data center?  Yeah, so obviously I'm not an administrator of the network of some kind, but I was, I seen this all.  So my main service was located in one data center and it failed power.  And it ended up just in four hours of outage, complete, nothing more.  Crucial infrastructure was located there.  So we just dialed up our clients and said we're sorry.  Afterwards, we apparently distributed.  Yeah.  What time did it take to distribute the whole thing?  I would have to ask my administrators, but I know that certain steps were carried out.  Yeah.  Just add a little bit because just how easy terrible stuff can happen to a data center,  especially if it's not like a big company, the small data center or service provider having a small hosting area,  because I used to work in the kind of the same environment.  And basically so many things killed the world.  We had a story, I mean, I won't name the company, but basically it happened overnight.  And the night shift who was monitoring the object just everyone fell asleep suddenly.  And they missed all the alarms.  And basically when the morning shift came, like all the temperature in the server room where we had a lot of our customers hosting their services was like 70 degrees.  We opened all windows and started just like, you know, to try to get somewhere there.  But basically a lot of things can go horribly wrong. So choose your data centers carefully and try to really get more of them if it's possible.  Yeah.  Contracts with your providers.  One more?  Yeah, yeah, yeah.  But I'm just really into the contracts to your providers are not enough usually.  And I even provide us say like 99.9999% but not 100%.  Yeah, yeah.  It's a luckily this was a data center that was only used for the development.  But we had an air conditioning that was running really hard and it leaked water into the power outlet that was behind the UPS.  So no more uninterruptible power supply and what proved that it was interruptible.  It was down for two days.  Yeah, it was major, major problem.  Yeah, you have to call your clients in the end.  So I guess this is this must be very hard to explain.  I don't want to be in the sales department at this time.  The problem with geo separated distributed locations is not when it goes down.  When things come up again.  That's right.  I've had a few times where services came back up and we had both of them active because they couldn't see each other.  But the rest of the world could either see one or the other.  Yeah.  And then people start using it and when they when they see each other again, then one of them has to decide to be slave again.  And weird things happen.  Yeah, that's called the split brain situation where your brain doesn't know anymore because you had usually two peers.  That's why in clustering in general and in everything you should do is that always be at uneven numbers.  And you already know about the voting strategy.  Okay.  If I am in a disconnectic situation, who is down?  I am or is my peer down?  If you have only two peers, you have no way to know.  You have at least to have three peers to be able to know.  If you can't reach any of the two other peers, you're down.  That's solid, pretty solid.  It's not always solid, but it's pretty solid.  At least always thinking uneven numbers, always, whatever you do.  Yeah.  Okay.  So, the co-terror is great, but sometimes real problems are a bit more complicated.  And it's not always like dev app stuff.  It can be like really coming from your code.  That's what we are going to see.  So, one day I was walking like normally doing my stuff.  And one of our market guys came and told me, hey, run as the client says you can authenticate on the server, on the website.  Something's wrong.  I was like, okay, I'm going to check the logs.  This happened like maybe 10 times per day.  So, okay, let's see maybe something's wrong.  So, I set the machine.  I look at the log and everything's okay.  So, well, the client is wrong.  What?  What did I?  So, yeah, the client must be wrong.  So, he goes away and I'm happy.  Something like one hour later, I'm still walking and the guy come back and tell me that it's still not working for the client.  So, I'm exhausted.  All right, I'll check the code.  Something's wrong.  Then I look at my application and I see that.  Does anything see something wrong?  So, after 30 seconds, I notice that the same email function can fail.  So, if the email function fail, well, it returns.  Okay, it works.  So, yeah, my conclusion to that story is that you have to know your code.  Interest situation is great, but code can fail too.  Even if you don't like the guy who wrote the code, even if you don't understand the code,  if you're a maintainer of something, you have to understand what you're doing and you have to re-factorize when needed.  So, the code should never pass suddenly.  That's from the Zenf Python.  Well, yeah, don't always blame the guy.  Sometimes it's easy, like, okay, that's not my fault.  It might be another server thing.  So, that's why the devil thing is great.  So, you can really understand what's happening on your server, even if you're just a developer at the origin.  And the other way I run this is true too.  So, do any of you have similar situation?  What kind of really weird things happen?  Okay, now it's going to be brave for developers to raise their hands.  I know.  I had a city situation where a similar thing where they're saying, oh, this isn't working for the client.  She's trying to do all these things.  She had a really odd workflow.  So, I was thinking, this is all working, all the tests are passing.  I go into the website and I'm looking and thinking, this is all working fine.  And I ran all the tests and it's all working fine.  And what I didn't really realize, it took me like a week to realize where she kept on coming back.  The point where I'm like, I use no script.  So, I'm happy using the HTML back end and everything so I can find.  What I didn't realize was if you enable the JavaScript, JavaScript uses a different API and that's the thing causing the problem.  So, make sure to eat your own dog food and use your own API.  That's like that got, yeah, it looked like it was working, but I didn't write the code, so it's fine.  Yeah, but in the end, you were responsible.  Yeah, that's, yeah.  So, in Python, we get used to the libraries we're using, raising exceptions.  A really common one that doesn't is Memcache.  Pretty much every Memcache library will return zero instead of raising exception.  So, you need to wrap it or do something like that, but there's four or five places I can think of in different projects that we've been working on where we trace back something to like, why isn't anything working?  It's because we think Memcache is working when it's not.  Yeah, I tend to like the Memcache Python library because of this, but sometimes it can be in the nightmare, yeah.  So, you have always to check about the, it's like the goal.  It's like a goal.  You have to check the year, you return of the, of the, of the operations you do.  Any brave, any other brave developer want to share about this?  Oh, yes.  We have one here.  That's it.  That's it through.  Yeah, so my example is not related to Python really, but to PHP.  Yeah, and now.  Who thinks it happens more in PHP?  That's brave developers.  For my defense, I am not the root decode, but there is a very nasty thing when you try to auto load some file class and you have a syntax error.  Then if you do not handle this properly, then PHP dies, returns, and the web server returns blank page with 200.  Okay.  And, okay, no other way to debug this issue.  That's a nice one.  We ended up at the WordPress hosting company when I work, we ended up writing some code in the reverse proxy that would detect these sort of situations, the white pages and alert us to it just because it's such a stupid default.  Why would you return it to 100 when something's wrong?  And yeah, it's horrible to monitor for that.  I have to mention Python break, break itself this rule.  For example, he's out of height, but exception.  And sometimes it's can make very strange things.  Switch to Python street.  One thing that's not related to Python or any programming language really.  I was having a server with a pretty large disk in it.  And there were very, very, very many files on that.  And then suddenly a developer called in and said, hey, I think the disk is full.  So I go look, do a DF.  And he says, well, I can't write any files anymore.  Okay, touch fall.  This full.  Yeah, it ran out of our notes.  That's something.  Yeah, that's a nasty one.  That's a nasty one.  We often overlook here.  Absolutely right.  And there are five systems who don't rely on I-nodes.  So when you know that your application might spawn a lot of files, think about them.  Indeed.  I have another story that I forgot to put in the presentation.  So I'm going to tell it right now.  Basically in my old company, like it was a small startup before I worked on Numboli.  So we were trying to get things fast.  So basically our web server was running inside a T-max.  And sometime when we looked at logs, we were like just crawling on T-max.  And one day we were like, oh my God, the web server is not running anymore.  And actually it was just T-max.  When you scroll, it sent a pose, it sent pose to your application.  So the application was down just because T-max was trying,  we were trying to take the log with T-max.  Don't run T-max in production.  And about DevOps philosophy, I don't know what kind of objective we can have.  Who works as a DevOps or in DevOps-minded company?  I see, I don't know if you're waving to say hello or...  Okay.  So it's amazing that you work in a DevOps company.  Just wait, I come with a microphone.  Because we can understand you in the back.  Yeah.  Why is not the delivery of that?  Yeah, we should have thought about that probably a little bit early.  The DevOps question is hard because when your managers and everybody talks a lot about DevOps,  but they hire a guy who is a DevOps.  As a DevOps position, then it gets tricky.  So you get back to the silo.  So we are developers and the DevOps.  So yeah, back to developers and Sysadmins.  I was actually a developer who had to run back to our admins to check up why the fuck is Docker not working again.  Oh, the elastic search containers clustered with each other.  Oh, interesting.  So in that sense, I was a DevOps because I needed to worry about the code  and about infrastructure mess ups.  So it's a tricky word.  Yeah, it has a different depth depending on where you stand.  Which lead me to a question, who runs Docker in production?  And can you share some experience with it?  I'm interested.  And I'm more interested in when it's failing, obviously.  Just one thing, we were actually doing just this new project.  So it was more of a proof of concept, but we've already started to get it out to customers.  And we were working together with this consultancy who told us how to do Docker and Cloud Foundry if someone knows Cloud Foundry.  So our whole infrastructure, we would provide services for Cloud Foundry based on Docker.  So we know like spawn, radices, elastic searches, stuff like that.  But the Docker cluster was actually one machine with all the containers for all services.  So don't do that.  Oh, yeah.  Okay.  Thank you.  So I can share a funny story where the Docker demon crashed on the CI server.  So you can imagine having like 15 super highly paid developers who are just mixing the air.  And it was also fun to debug because, you know, who would have thought of it?  Yeah, but to get to the previous point, how much effort would it take to implement something like Supervisor or whatever process that would monitor the demon?  Probably not much.  Yeah, you're right.  You're right.  Sometimes we're on own barriers.  And we, yeah, I really agree with you.  Just about the DevOps thing.  Yeah, it's kind of a buzzword, except especially for recruiters.  But all we see it's that normally is really like not a single person being in DevOps, but really a team where you have, yeah, people who develops, people who make labs.  That's the thing here.  But just working together and understanding what the other is doing, it's just very important.  And it's also a great time also to a developer to acquire and helping him understand what is not used to do.  Another real world problem.  Go ahead.  Yeah.  So we have our statistic on Grafana.  So it's a very nice board.  One day I was looking at Grafana and I saw the statistics. So it wasn't really important because it was really just the maximum processing time of one of our services.  Basically, the average processing time was still very low.  So we didn't really investigate that thing.  It stayed there for like, I don't know, maybe two or three weeks or even more, maybe, I don't remember.  But we never understood really what was happening with why the maximum processing time was so high.  And then one day, boom, like what?  So my first idea when I saw that the graph was going so low is, oh my God, the service is down.  Actually, no, it was still running.  But what was happening is, so I searched for like one or two understand what was, what happened.  I ended talking with one of the most ops guy in the team and told me, hmm, that's strange because at that moment I deployed an Ansible playbook on one of those servers.  And so we looked at the playbook.  What was the difference?  And the only difference was in the ETC host file.  Basically, the NS server was like all the time queried at each database insertion.  So sometimes it was just overloaded.  So just putting the IPs of the database in the ETC host file of each machine fixed the trick.  So, yeah, that was pretty weird.  Go ahead.  Paul.  Sorry.  Paul.  Here.  Paul.  Sometimes some other stuff I have to sort of like.  And it's not just resolving the database server.  You'll be surprised to see how many code is reversing the DNS.  So you have to not just forward, but also reverse is happening quite a lot.  Yeah.  That looks like this problem.  To be honest, we felt pretty stupid with this one as well.  And this one is pretty interesting because two days ago you made a presentation about using console.  Yeah.  Sounds very contradictory.  Yeah, yeah.  And the thing, the question is have you tried to put a local GNS cache, I don't know, a bind 9, something like that,  and a TTL around 30 seconds or minutes, something like that.  Would it make the trick?  Yes, absolutely.  What's embarrassing with this is that we also lacked consistency in what we do.  You know, on another type of infrastructure, we have local GNS cache, but there we didn't have it.  And when Guillaume says the people were working on the Uncivil Playbook is also to stop normalizing all of these.  So yeah.  So maybe we think that we have something in production and it's running for so long that nothing can happen to it.  And we tend maybe sometimes to forget about its resiliency or performance or just applying the latest of your knowledge,  just for the sake that it's running, I don't care, or I don't need to bother so much unless something weird happens.  In this case, it was good news.  You know, we were satisfied with this shitty processing time.  But on another type of application, it might be not so.  I think that one good trick is to always profile your applications at least once.  And I recently used VMPro from PyPyGuys and it actually just slowed down my service at around 5%.  So it's actually viable to just switch one instance and check what actually your code is doing.  Actually, in that situation, I profiled the code and I didn't have the same results as Synod's Garfan as others.  That's why I was like, what the fuck?  It's not working as it should.  And that's why it wasn't really important.  So like I said, we just let it go.  Well, it was a good surprise when it was fixed.  We came up only with embarrassing examples, so you feel more comfortable sharing this.  I have a comment regarding performance monitoring tools.  You really have to configure it properly.  We had a situation that response time was between 30 and 60 seconds and it was caused by uploading files.  For example, 5GB files was uploaded by a few people and it increased the average response time.  It was the same kind of problem. Sometimes our metrics ever goes down and then we think, oh my god, my application is done.  But it's running.  Yeah.  Who is using, who is not using a metric system?  Who doesn't do metrics on their applications?  Who doesn't have this kind of graph?  Nobody. You all have it.  Yeah, I see you again waving people.  Question.  Yeah, yeah.  You can't just stand in front of the camera and hold your hands.  Question, I was precisely going to ask this tension of this kind of question.  Have you managed, Nien, to put in the graph, in Grafana some kind of percentile graphing so you can know 90%, 95% percentile and 99% percentile of the response time?  So you all know that kind of problem.  We have been trying, we have a hard problem with this.  We are deploying Promethines plus Grafana and using that with elastic surgeon, we are having a lot of problems.  We really calculate the 99% percentile, the 95% percentile.  But have you managed to do this?  Basically, we just show what we use in general is a comparison between the current day and the day seven days ago.  So it gives a good idea to, is it normal or is it weird?  And in using carbon and graphana for the visualization, you also have the annotation feature, which is good,  where you can have a bar on your graph saying, okay, and you can plug it to your deployment or continuous delivery stuff.  So you can have a bar on your graph saying, okay, from this point on, this is version 2.1.  And then you can do also matrix comparison related to code deployment.  It's pretty good.  In disaster recovery, it's also a good thing to have. So when you know them, you broke something and you can do the same with server provisioning and deployment.  At this time, I added a new server.  Maybe it has some weird side effects.  For the percentiles, if you mentioned you've already got elastic search, if you don't have aggregates but have the actual requests logged there,  you can just use Kibana because it has a really nice visualization.  If I recall, it gives you the percentages, the percentiles as well.  You should get that for free from Kibana.  The answer is the problem is to combine it with graphana for the remote audience.  Did anyone come with a question?  I guess we have four minutes left.  So open discussion now.  Question.  Okay.  This is tricky.  I'm looking to know if Edmai has some experience with trying to deploy a new version of your backend and only deploy it to, let's say, 5% of your users,  try it out, see how it handles, and then go for 100%.  Especially in GenX, if you do in GenX, that's really good.  So progressive deployment.  Anyone?  Thank you.  Yes, for the same company I was talking about a bit sooner about the e-commerce,  we were always rolling up the traffic, but it was with H-Epoxy in front of...  I think it was H-Epoxy, then Varnish, then Nginx in that order.  And the new servers were rolling up 10% traffic on the new version.  Then we had the software error monitoring and all the metrics on this server.  We were checking that the response time was not doubling, et cetera.  And after a few hours, a few more servers were joining.  And at the end of the day, all the traffic was rolled up from the...  That's one sort of question.  Depending on your stack, we do a lower level of this.  We run our Python using UwSGI.  And in UwSGI, you have this feature where you have...  It's called TouchChain Reload, where your workers are reloaded one by one after...  And UwSGI will make sure that the one that is reloaded correctly before reloading the others.  So it's a good fail-safe, low-level deployment trick.  And on the side note, if you are really, really committed to trying canary releases,  which is usually called canary releases when you put the bird and try the mine,  if it's pushed through IQVIRNets that resolve this problem in a very reliable way.  But it's obviously very...  It may be too much complicated for your case, but it has exactly this kind of procedure.  When you say, I have a rolling deploy strategy,  when I want to keep an number of pots, which is your application deployed,  and in an hour, would it begin to increase the number?  Yeah.  Kubernetes... This feature in Kubernetes is very nice.  But Kubernetes still doesn't have Elf Check, right?  Kubernetes still doesn't have Elf Check, right?  Yeah.  Elf Check.  Yeah.  Oh.  Okay.  So yes, it has.  Okay.  Thank you.  So that was one of the really bad things that I would add us to the table.  Okay.  Great.  Readiness check.  That's right.  Thank you, Paul.  Yes.  I want, especially thank you, because this is like an interactive format,  and it's a little experiment, just like to have, not to have only like one-sided talks,  just like to have a, on the direction, I want to thank you for like taking a leap of faith  in the first year we tried this.  So I think please give these guys an extra hand, please.  Thank you.  Thank you very much.  Yeah.  Yeah.  Lightning talks up at five.  Thank you.  Thank you.
Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_0_1787.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_1788_2723.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_2724_4655.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_4656_4811.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_4812_7499.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_7500_9923.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_9924_11231.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_11232_12287.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_12288_12371.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_12372_12947.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_12948_13223.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_13224_14975.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_14976_15035.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_15036_15095.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_15096_18011.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_18012_18287.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_18288_18731.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_18732_18947.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_18948_18995.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_18996_19091.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_19092_22859.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_22860_24251.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_24252_26699.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_26700_30491.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_30492_34751.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_34752_35819.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_35820_35939.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_35940_36875.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_36876_36983.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_36984_42359.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_42360_44111.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_44112_49295.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_49296_50651.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_50652_50711.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_50712_53471.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_53472_60875.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_60876_62543.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_62544_62615.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_62616_63671.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_63672_63779.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_63780_68603.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_68604_70835.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_70836_73163.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_73164_78035.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_78036_79475.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_79476_79691.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_79692_82703.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_82704_82763.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_82764_82895.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_82896_88667.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_88668_89147.png Alexys_Jacob_Guillaume_Gelin_-_Planning_for_the_worst_89148_89990.png