Service Discovery for .NET developers
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Cooper, Ian
So you built your shiny new microservice, and now you want to deploy and have other code talk to it. But how will other code find it or even know that it exists? And if you recognize that one of anything is a risk to your availability, how you can enable clients to find the instance of your service that is running. For many people today the answer is a wiki documents machines and instances, and config files bind one service to another with failover involving an operator manually changing the config and rebooting the service.
 OK. Hopefully, you are all here to see me talking about service discovery for.NET developers. Excellent. You are in the right room then. OK. So this is me. I've basically doesn't say much apart from the fact that I'm old. But this is the point I always call out, which is, you know, I'm not standing here claiming I'm some kind of uber smart geek. I just happen to have done some stuff that I want to kind of share with you or understand some stuff. That's where I work. The only real message is it has nothing to do with what I'm talking about, so I'm not pushing anything on you. That's an open source project to work on called Brite. It's a CQRS framework. That's a Twitter handle for it. If you want to find out more about that, please follow at Brite to command. And you should better then find out more about something else we do, which just comes out of Huddle. We support that. OK, what's today's agenda? So we're going to talk a little bit about something we already know to begin with. It'd be nice and easy. We'll talk about point-to-point services, client server. What we want to do is talk about why this is a problem, the problem of availability that relates to point-to-point. And we'll talk about very simply how we solve that. Now, all that stuff is fairly straightforward. And we'll talk about then really why service discovery comes into play once you start to think about the solution to basically the availability problems on point-to-point communication. Then we'll talk about how that all works. Talk about basically there's different patterns we can use, local registry, dynamics of registration. Then we'll go to health checks. Then we'll talk a little bit. There's two forms, client and server side. We'll focus mostly on client because it's the more interesting for you guys. We'll talk a little bit about service discovery. And then if we have some time towards the end, we'll talk a little bit about this notion of zero configuration. Hopefully that is roughly the agenda you're hoping to see. This includes things like DNSSD. What I'll expect you to get from the session, I expect you to find out what service discovery is, why you might need it, and what are the patterns that you'll need to understand in order to use service discovery. I expect you to by the end of this be able to understand when people start talking about things like ZooKeeper, ETCD console, what they're talking about, and why they might have differences, what those differences mean, and how you can use them to implement these given patterns. So one reason to come to this talk is simply to say, I have no idea what ZooKeeper is, and I'd like to understand what ZooKeeper is by the end of this talk. Now that'll happen. The other thing I want to try and point out is that this talk is essentially very straightforward. There's nothing here that's actually that complicated. It's just a bit of a mystery sometimes to folks because you've not had to go and find out about it. So take this as really a way of short circuiting the process if you haven't do all the reading and finding out about stuff, but nothing here is challenging. So this is a nice, gentle, easy talk. None of our brains should be stretched too much. By the end, we can walk away with a warm, positive glow that we know something new, and we can laud it over our colleagues by saying, so you don't understand ZooKeeper then. All right, great. Done the slide, just in case anyone didn't have a chance to take a picture of it earlier. You can get the slides, and you can go and find the demos there. Just give a second for someone to pick up a couple of guys on their phones. Good, okay. Point to point, right? So everyone kind of knows how we do this, right? So this is a diagram, essentially, of a client-server interaction. And we have, essentially, a server. We're offering some kind of service with some kind of API. And a client wants to talk to us and get the service from us. They want to ask some questions. How many orders did we make last week? Can you log this person in for me, basically, to the server? Can you go and get a hold of my document for me that you've stored? And there may be, if we're doing something like HTTP, there may also be something like a proxy involved in interaction, right? And the proxy may say, well, I can cache a certain amount of these requests, and I can have some kind of indicator whether or not they're fresh or stale, and maybe I call you back and say, hey, is the version that I have still actually stale, or is it fresh? Don't need to get a new one. Can I serve my copy to the client? And that's still very much a point-to-point interaction. It's, we may have a level of pass-through, but it's what we call point-to-point. The other thing to bear in mind is that we talk about client and server here. It's easy to think by client, we always mean something running on the user's desktop or something running in the browser. But actually, it's just a role. The client is the person asking for service, and the server is the person kind of giving you the service. So this could equally be a model where we have two services running on our back end, and those two services talk to each other in order to fulfill some request for the customer. Okay? Right. So this will be a quick demo. And what we're gonna do is just demonstrate something to you that you hopefully will understand already, which is how we do a quick bit of client-to-server interaction. Let me find this. So these are all kind of recorded in Camtasia to save me being able to talk and type at the same time. Other people have mastered that particular mystery, but I find it's almost impossible. So, okay, let's see. So here we've got just a self-hosted web service. Okay, so this is basically hosted on a given host.reson.port. We have a configuration file in the self-hosted, web service just running a web API. And essentially we have somewhere we say, there's URI we're actually gonna run. We've got default value in our config file. We're saying, okay, we're gonna run on this address. Okay, so our API runs on a given address. Here we've got a standard loader set up for a self-hosted web API service, looking for a green tracing, formatting, then we're mapping some routes on the controller. It's all boilerplate stuff that you guys write every day. Okay, even if you don't actually use self-hosted or you use IS. And here we're exposing two methods on an orders API, get method and a post method. And we're just from a client. I'll go back a second before you there. It's a bit fast. Let's scrub back to that bit a second. It's so lazy. Sorry guys. Scrub forward. Okay, we're just gonna make a standard call basically across to the server to say, can I add this order model in and when I wanna get the content back, get basically a success code and display out to the console the current set of orders. Okay, so I'm just ordering something very basic. So the details not really that important here. You see this a thousand times. I'm just really showing you this because what we're gonna go through is go and edit this one later on. Okay, it's just on replay. So that's why it's doing that. Okay. Oops. Oh, actually, no, I don't want to. Sorry, let me just towards the end of this thing. There should be, back here. Okay. So we're gonna run the server here. It's just gonna run, sit there waiting for requests. We're gonna run the client, just a batch file. And all it will do is make the request to the server. You'll see the server do some tracing and we'll actually get a response back, which basically is an order. Don't worry about the actual details. There's just essentially saying these are the orders on the server. Okay. All right. And we're doing that quite fast deliberately because you guys have all seen this, hopefully, and done this thousands of times. All right. Let's try and clear that message. The basic problem with that model is this. If that server goes offline for whatever reason, my client is now somewhere out of luck. It's possible that if we've got a proxy here and we've cached the responses it sent us earlier and we don't require revalidation, we haven't got to do a last modified or we haven't got to do a e-tag to say, actually the version I have in my data is still correct. It's still in time, that I may be able to grab hold of that kind of orders list and get a stale version of what's going on. But in most cases, I'm going to find that this is simply a problem for me. I cannot no longer communicate with other servers and so I stop providing my functionality. So I have a dependency in A on the uptime of C. All right. Come on problem. So how do we solve that? Well, we solve this problem every day in a sense when we have a browser and a website and we have a web farm. So we just introduce more servers and we say, well, okay, it doesn't matter if this one's down. I've got two other servers and they can service the request for me. Right. Someone in the back end, they've got a database that we're all talking to so the data is consistent. But A is now going to go to C or D and we're actually going to then provide service to the user. Right. So we create availability by introducing redundancy. This is becoming more important nowadays because many people are taking their monolithic architectures and breaking them up into a microservice architecture. And there are two kinds of ways that there are two kinds of schools that essentially are, you can think of as microservices. One is all the people like me who used to do SOA and maybe there's something like Gorilla SOA where we've essentially dropped the service bus and went for smart endpoints and done pipes. But it's essentially an evolution of SOA. It's all that stuff like people talking about explicit boundaries and bounded context and business capabilities and that kind of thing. The other model is kind of the Unix pipes and filters model of microservices where they say, well, we want lots of tiny little services that we can compose to create new pieces of functionality. Both those schools are equally valid microservices ideas, but both of them, when you break up your monolith, resulting in you having a large number of services that now will talk to each other. So one of the problems is that anyone moving to microservices now has to consider the fact that they have lots of dependencies between services. And if those services, the services I depend on goes down and is not available, then it has a cascade effect and much of the service disappears on me. This is one of the reasons that people talk about microservices being more operationally complex because they have this web of dependencies. So my service A depends upon service B. If I don't make service B redundant, then what's gonna happen is that A is not gonna provide its service when B goes down. And quite often, B may be down for quite transient network reasons, or maybe I need to just actually go away and patch B and install something new on it, or do a release of my service to B. It's one of the things we want to do in microservices, you'll be able to release independently. So maybe B is something I want to actually upgrade with a new version, but I want to take down A just to basically create a new version of B that defeats all objects of microservices. So I'm gonna need a redundant version of B. So this redundancy problem has become something people have to deal with a lot more. Now there are other ways around this problem. You can use something like a lightweight broker like RabbitMQ, and that can solve the problem for you of some of the dependencies between your various services because you just rely on queues instead. That's called decoupled invocation, but you still have a slight problem in that your client has to talk to essentially your RabbitMQ server. So it still needs to know the address of something, and that needs to be redundant because otherwise you've only got one of it, and you're still in the same problem. So the trick is to remember you can't have one of anything. You have to design for failure, and in a modern microservices rather than monolith driven world, you really need to focus on this a lot more. Before, by the way, everybody's always been distributed for a long time because your database is a separate process to your web server, to your code running in your browser, so you're already distributed. There's no way around being distributed nowadays, but here we're talking about a situation where you explicitly rely on other processes to provide part of your functionality, and that provides a problem. How does discovery play into this? Well, the real question becomes, if I can solve the problem essentially of E not being available by having C and D running at the same time in other instances of that service, but the problem is, how does A know to talk to C or D and not E? Now, when we look to the point-to-point scenario earlier, the client simply had the IP address of the server, it was localhost4672 or whatever it was. The problem with that is that referred to E. C and D, if it's on my same box, are probably running on different ports, but if they're running on different machines, they're probably running on different IP addresses. So the client doesn't know where these are and probably doesn't know that this one is down. And service discovery is simply, how do we solve the problem? We're telling the client what servers are available and what's their status. It's nothing more than that. Okay. So I've done this to introduce the question of who do we talk to? Then in a single server, we now probably have a pool of servers, and discovery tells us basically, how do we find the pool of servers so we can load balance between them effectively to avoid problems of failure? Does that all make sense? This stuff is really quite straightforward. Okay. So there are two major classes of patterns we need to be aware of when we start thinking about service discovery. We can either do our service discovery essentially on the client, or we can do it on the server. Look at this client and server models of service discovery. We'll talk a lot more about client-side discovery. Service-side discovery tends to mean, essentially, we talk about a load balancer. Okay. We'll talk about the client-side model instead. Okay. The simplest way for us to understand what we mean by client-side discovery is to do something without using any kind of technology like a zookeeper or console in the mix for the time being. Just imagine to yourself, well, one solution would be for our, let's go back. One solution would be for A, to have in its own config file that we ship with A, a list of all the servers with their addresses. Right, so I simply give A a list saying, well, there's C and there's D and there's E, and these are the addresses they live at, and you can talk to any of them. Okay. So look at demo of that. But again, this is code that you guys should be, you know, hopefully already pretty intimately familiar with as a way of working. Right, so we're just gonna look at something that's a variation of the code we showed you earlier directly in point-to-point. This is our controller, and you can see there in the controller what we're doing before we start the video, is essentially we have a list of servers. We have a retry policy we'll talk about in a second. What we're gonna do is go to our configuration file, read in that list of servers by simply taking the config values, creating a list of server items, which it will have a URI and a timeout value to hit that server. And it's that list we will then use to make our requests. Okay, so this is list basically saying, go away to the config file, basically read from the config file. This should give us a server list. For each one of those items, create a new server item object and add it in. So the config file, we're just using a standard configuration section, and we've just got a whole collection with a load of elements inside it. And basically the key part of the elements is the URI. We have a timeout. You tend to have timeouts when you're doing any kind of cross-demo communication. What it looks like in the config file, it's just a list of servers. So basically you've got two key addresses there. What we're doing with our orders API, in fact we've added the second config file, so we can actually have two configurations. This is a retry policy, and essentially this is using a library called Poly, and it says if we get an exception, all we want to do is take an action, which is essentially to log an error and try the next server in sequence. All that TryNextServer does for us is simply, we just iterate, increment a counter, and move on to the next server. You all want to use something slightly more sophisticated, like a round robin algorithm, but for the purpose of the demo, that works fine. Essentially we're just gonna do the same thing we've done before, which essentially is add an order, and essentially try and post it to the far side. So we're gonna run two servers, run the first instance here, run the second instance here, we're gonna run our client. This is before, and we'll see what happens is the first server you can see reacts, so you can see the detail, but that's just a trace of it reacting when we get our inflation back. Now if I stop the first service, what you'll see is we time out in the first call, and then we'll switch over and call the second service instead. So we run, we time out, and then we call the second service, and we get our response back. Okay, 30 straight forward, everyone kind of with me. So that's, you know, if you can do that, which most of you I'm sure feel pretty comfortable with, you can do client-side service discovery. So we can all go now, and nothing else to talk about, right? But it's straightforward. Okay. Sorry, my phone appears to be telling me about infrastructure alerts, which someone else can respond to, but I apologize for it beeping. Okay. Okay. So the advantage of this particular pattern is it's very straightforward. Most of us understand the right config files. I'll read through the list, how to use a list. I mean, using a library like Poly, this none of this is particularly challenging, and it's fairly easy to do some kind of localized round-robin algorithm to low balance amongst our servers. Okay. We can even use this to actually do proper load balancing, not simply just coping with failure. We can actually balance, especially our clients calling out across all these servers. So it's an easy way and cheap way to get load balancing solved is a problem. Okay. There are obviously a few issues here. So the first is that this is quite difficult to manage in the, you know, I may be actually operationally deploying new servers, particularly if I'm in a cloud environment with new addresses on a quite regular basis. And if I have to ship a new config file out to all my clients when I do that, that can be quite a hard problem of asynchronous. It's bad enough as this, if we've got a microservices environment, and those are all internal servers that I'm running, where I have to update all their configuration files now to basically take account of the new servers. But it's kind of even harder if these things are actually off-site applications running on my client space, where they may choose not to actually update those clients as often as I want to. I mean, I've got to not get the new list of servers. So the problem is a sufficient number of services, service site is going to break down for you, and if you've got genuine clients outside your firewall, that's going to be a problem. The other issue is there's no health checks in here. So I'm still talking potentially to servers that are dead, and the only one I'm going to find out about that problem is essentially my error on my call, and I iterate over to the next server. I put this slide in because you may think, well, this is just a naughty demo solution that Ian's put in. We're going to move on to the real solutions later, and no one would use this kind of model in production. And the answer is that's not true. So how many of you use Redis today? A number of you. And how many of you use, say, the service that Redis client? We're doing that. OK, so the service that Redis client does its ability to cope with essentially failure. So Redis has a leader follower model, and essentially you can talk to the pool of followers on that given leader to provide availability of your Redis server, particularly if you're under load. And what it does essentially, if you're going to get documentation for it, you provide it a list essentially of hosts, three to only one where effectively you can read from and the right one you write to. So it has redundancy across those read-only hosts. And that's done by you putting that information in the config file in your service. So that's a genuine model that lots of people are using in production today to provide availability. It's not just a toy. However, we want to do something better. So what we're going to look at is we're going to look at using registration service. And the simplest idea here is you have something, essentially, is a database that we talk to. And from that database, we are going to get information about where our services live. At a really simple level, you could go away and use any kind of key value store for this service registry. So you could just register to hold this. You could even use your SQL Server database. But what you're concerned about here is that this service registry has two things it needs. The first is you don't want this to be another point of failure for you, so you need to have redundancy. So the data needs to be across multiple nodes. I need to have some way of solving the problem of how do I find my service registry? Because I don't want that to be, it's the chicken and egg problem, effectively. It's OK, I can talk to my service registry. It will give me the answer. Well, where is my service registry? So those kind of problems of what service registry toolkits tend to solve. OK, here's a list of the ones you may have heard of. ZooKeeper. ZooKeeper is kind of the oldest and kind of the most famous. We'll talk about CP versus AP in a second. It's generally a service register on startup, and they de-register themselves when they're done. It's very usable from Java. Otherwise, you have to work with a C library. There are some.NET wrappers, but generally can be kind of a bit of hard work to work with. Airbnb's Smart Stack, and that tends to work basically with Hadoop to manage the cluster for ZooKeeper. Netflix's Eureka. That's pretty good for rest and Java and design for use on AWS. If you nod in that scenario, it's probably not going to be that helpful to you. Etcd is a simple distributed key value store with an HTTP JSON API and Rafa consensus. It's kind of the granddaddy of the more simple ones. And of those, there's really SkyDNS and console, which kind of come further down, which give you DNS support as well as HTTP JSON API. Right. What are these things like CP and Rafa consensus mean? Who knows about CAP there in the audience? Most of you have an idea. OK, so we'll just check for those that you don't. Consistency, availability, and partition tolerance. You essentially can't get all three. So these three distributed systems, you want to have multiple nodes. Essentially, I've got a problem. When I ask multiple nodes a question, I want to get either a consistent answer. They all agree who won yesterday's Everton versus Arsenal game. An available answer, what was the last score that you actually had for that game? And essentially, they can't really deal with partition tolerance, which essentially says the network is divided and the nodes can't talk to each other. But you can, 10 times, I either have to choose CA or CP rather, which is basically, I will be consistent when I give you a response, or AP, which means I'll be available when I give you a response. I'm going to give you the right answer or nothing at all, or I'll give you an answer, maybe style. You can use CA. Some people say you can't. You can. But CA essentially says, in the event of our network partition, my data center is probably on fire. And the last thing I care about is whether I'm either consistent or available. OK. So you can see nearly all of these are CP, apart from Netflix, Eureka, which is AP. So Netflix cares about availability more than does about consistency in its answer. But most of them, you can actually, their APIs, the CP APIs, respond to allowing you to say, actually, I'm prepared to have style results. In other words, be AP. Yes, that's a question. I mentioned that it's de-registered. But if the server is down, the server didn't actually get to de-registered. It's still an entry. There is still an entry. We'll talk about that later. So that's a good question. So the question was, the thing that the service is de-registering from a service registry, isn't that a problem if, essentially, the service doesn't goes down without being able to run de-registration? It would have an entry in there before it be saying it's available. It's not. We cover health checks. We actually cover what really happens. This is a bit of a short slide. The real version comes later. OK. So the examples, we're going to do some examples. We're going to use console to do our examples when we talk about code today. I am not pushing console particularly over any of the others for any reason. I have no skin in any of these games. We like console. It's very good. Nice, straightforward HTTP JSON. There are some reasonable.NET clients available for it. And it seems to work reasonably well. OK. It's free open source. It's created by Hashicorp. People know who Hashicorp are. They create vagrant terraform. So they're generally in the distributed systems, management space. They have quite a number of open source tools. And vagrant is more lifecycle of virtual machines. It's a useful tool if you want to run VMs locally. A lot used to do vagrant up and you get given a VM. OK. Provide service discovery by either DNS or HTTP plus JSON. We'll show you most of the examples using HTTP and JSON. But we'll talk a little bit about DNS, why you might not want to use it as a.NET dev towards the end. It has sports health checks. So you can essentially determine whether or not any service entries actually are currently live. You can download it from Hashicorp. Consulio is the site, effectively. And it's a useful documentation. It's quite well documented if you need to use that. Generally, there are a couple of.NET clients. They wrap the HTTP JSON API. So you can just talk, if you want to, you could talk raw or JSON and HTTP if you wanted to to them. It's actually quite useful, though, sometimes to use someone else's abstraction built on that. The PlayFab one is pretty useful. They have some documentation. It's not great. There are some examples in the project that are probably better where finding out what it does. You'll see some code here as well. So my code's available. So you can go and have a look at my code for our world as well. So what Consul consists of, we need to download from Consul what's called an agent. And we run the Consul agent. It says all the nodes in the cluster. What we mean is that we run the Consul agent on the same machine as the service we want to essentially register. And then we run a number of other Consul agents to essentially provide the service catalog. The idea is, basically, when we run our agent, we run it either client or server mode. Now, server mode essentially says I am the catalog of all the services. And you need to run either three or five of those. It needs to be an odd number so that you can essentially have voting. Because the servers use something called the Raft algorithm. And the Raft algorithm basically uses a stupid system piece of kit, which essentially says I want to make sure that essentially the data across all of my services is consistent. So I'll use voting to make sure that essentially if I try and write to the leader, all the followers are updated at the same time. The client essentially sits with the service on the server. And it forwards nearly all of its requests to the servers. So it's a way of you saying, I don't have to worry about whether servers live. Because all I need to know is essentially where the agent that's on the same machine as I am is. And it knows where the servers are. So how does it do its service discovery? Well, the answer is basically it uses a protocol called gossip. And essentially, gossip tells it where the servers are currently running. So when you talk to the agent, it should know where the servers are, that it has to talk to the service. The local agent also does the health checks. And it calls you and says, I'm going to do the health checks. It's more efficient. And then it gives the results back to the centralized service catalog. So he's making a local call, usually, for that health check. A typical console deployment looks something like this. So you've got, in one data center, you've got three nodes. One of those nodes is the leader. All the requests are forwarded to the leader. And you have two additional servers, which are there for redundancy and can take over as the leader. In the event that the leader disappears, they'll get promoted to become a leader. And you can survive with three, the failure of one node, with five, you can survive the failure of a lot more. And you can also basically have more than one data center. And the data center can use basically a wide-area network version of gossip to communicate information about the service catalog. Germany console has a number of HDTBA JSON endpoints. But the one you care about most is agent. So these are actually direct on the server. What you tend to want to do is call the local agent to you. And the local agent itself will forward the request to a number of these other endpoints. So the first registration pattern we'll talk about for use where the service registry is called sidecar. I don't know that term sidecar. I prefer the term sidekick. The different problem with sidecar is, in my memory, sidecars are attached to the motorcycle, whereas sidekicks can work alongside the superhero. And so actually what tends to happen is you tend to have another process running whose job is to do the registry of your services rather than your service to the registry itself. The reason why that's a significant win, potentially sometimes, over the client registering itself is you may not control the client or be able to modify its code. But you still need to register its existence and the number of nodes of it that you have. So we'll do a quick demo. And we'll do a demo of quite how that actually works in practice. Everyone following along? No burning questions at this point? OK. Yeah, but most people tend to, as I've seen, tend to run their server cluster on Unix and the agents locally on Windows. OK. OK, so what we're doing here is we've added a new Windows service into the mix. I use TopShelf to Windows services, by the way, in most cases. This is actually just a normal program, actually, to be fair. This is just a console application. The console application is just going to say, I'm going to go and register essentially a set of services. Now, all this console application we're going to look at does is, previously, we had in our configuration file a list of servers. And we actually had that in the client. All we're going to do in this case, you just move that list of servers into the configuration file of our sidekick, and it will register them instead. Then we're going to do this on one machine with different ports. But the reality is you would probably want to be doing this on actually separate actual sidecars running on individual machines alongside the servers that they're registering. But we'll just show you this where I'm doing it. So you can see the simple transition from the client has the list, the cycle has the list, and it gives it to the service registrar. So here we've got basically our console application, which is the registration. Registration essentially reads the configuration file. We call it an agent service registration class. We call the console and actually register it. This is just the same configuration file we had earlier. So Playfab's code, which we take in basically by NuGet, does most of the hard work for us of actually doing this, figuring out how to call the actually JSON API. I do register to get clear on the old ones out. We're in the register afterwards. In our consumer, now what we have to do essentially is ask to serve the list of services. We ask console for the list of services. We want anything tagged with orders. So we know that services we're interested in. We get a hold of and essentially add to our list of servers, the same as we had before. But rather than get from a config file, we're getting more for registry, which is basically the config file used. First thing we do is spin up console locally. We're only going to have one agent because it's a demo. But essentially, a real infrastructure would use a number running your catalog and a local agent. We can get away with the public domain just running one. It has its own config file. You can register stuff in the config file. I wouldn't tend to use that that much, it's less dynamic. And then we will run a couple of servers. I'll run the psychic first. The psychic will register our servers. You can see the service registration happening there. We'll run the first service and run the second service. Then what we'll do is run the client. We'll see if it behaved the same way as we did before. We'll basically just talk to the first service from the list we get from the client registration and get the response back. Then we'll do the same we did with the localized config. We'll stop that first server. We'll run this. We'll time out and we'll talk to the second server. So it's just the same solution we had before, but the config file is now essentially being turned into registration and centralized service. Let me get our response back. OK, that makes sense. It's all fairly straightforward. It's essentially saying, I have a list of servers. I'm either going to put them in my config file locally with my actual client. It's going to know about where the services are. And it's going to talk to them in a low balance way. Or you're going to say, that's too difficult for me to manage. I don't want to manage all of those pieces. So what I'm going to do instead is go for a model where I take those servers and actually register them myself in the service registry. And I can essentially pretty much use a similar technique of putting things in a config file and then actually registering those with my server for registration. And then my client, it's going to get hold of that list from my registration service. The agent, it's actually what I talk to in it, talks to the actual service. Get hold of the list of the servers that meet my required tier. They've got a tag on them saying orders. Then I make that call directly to get hold of my list of servers and I just start iterating through them and around robin algorithms the same as before. So my client takes responsibility. It's probably calling client-side-server-discovery for the work of doing the load balancing amongst the servers on the server side. People like this model because we don't take a dependency within our service on the service registration that we're using. So my orders API service there doesn't have to know anything about console or how I'm using it. Only my side-car service needs to understand that. So when I start switching from console to Sky DNS, then I can just simply replace my side-cars. I never have to go and actually impact my service directly itself. It's also very great because if I don't control the orders API service, someone else wrote it, or it's very hard for me to synchronize changing it and get its deployments out, I can simply use service registration still without actually impacting the code which may belong to somebody else. And you've got some things like registrator for using things like Docker. Registrator essentially listens for new Docker machines coming into an environment, looks at exposed ports, and registers them. So registrator is a way of getting console to work with Docker in a straightforward fashion. Obviously, if we're not using some kind of automated process like registrator, one of the issues here is that I have to maintain that list in my side-car saying, what am I actually registering? And I have to remember to go away and do that so I deploy to some new location that may be an issue for me. Change the port. I've got to remember to go away and change the side-car registration of the port at the same time. So do you have to sync up? We have a dependency now, essentially, which we have to be cautious of between our service and the side-car service, which has to exist and run in order for our service to register and the service register. So we have to make sure those dependencies are managed. This we can rely on things like console itself to manage basically the availability of that. We have to be concerned about, is our registration service going to run? Do we remember to run it as part of our deployment script, that kind of thing? So if registration says, well, if I have control of the client, I can solve the problem a little bit of forgetting to actually do my registration, needing to have the dependency on an additional service, but just saying, let the orders API service register itself when it starts up. So I am going to talk to the registrar and do the work. And then from the client point of view, there is no difference. The client just simply still calls console agent, gets the list of the services, and essentially stores them locally, and then does some kind of round robin out of them to load balance between them. See that? Hopefully you guys are kind of picking up that this stuff is not exactly rocket science, right? Someone actually, last time I gave this demo, live tweeted it to his team in the office who had implemented it by the time that we'd finished talking. So it's fairly straightforward. OK, so here basically in our control, we've got the code that essentially reads from the console, goes to the console agents API, says give me the list of services in response. Going to iterate over that list of services, finding that tag to being orders. Obviously it has to be somewhere distinguishing all the services, registered tags, and how you do that. And create an entry in my own internal service list, register that. OK. And all the only different systems are going to be now how we look at basically doing the registration of the API itself. So we'll flick over in a second. We've seen this before. There's a poly policy essentially for doing the retry. There's our client. OK, so on our self-hosted server, we're adding now a new thing called register service along with everything else we do with a self-hosted service. And then here we simply say, well, let's talk basically to a console, effectively, and do an agent registration. And we'll store our value for our URI in our config file. So it's the old config file we saw earlier that essentially says somewhere we have to store our URI in our port so that when we run, we can identify where we are. And here we're just saying, here's our timeout. We've got two defines. We can actually show you switching between the two, from ID and essentially the. So we just store information to register with console. So we'll start up console again. And then we'll basically run a server. And we'll see as you run the server. It's probably very hard for you to read, but your scroll past, you can actually see that essentially they've registered this service. Second one will come up. And again, we'll register the service. And now we're really back to the same model as we had before. We'll run the client, the client talks to the first service, says, OK, I've got my response back. We kill the first client. Run it again. We'll timeout talking to the first. We'll talk to the second, because we have the listener. So essentially you have availability to survive across failure of one of your services. And you provided that information out of a centralized service registry, which essentially has managed to get you away from the problem of localize config file, in that essentially you've got this one location that's much easier to manage. And the service itself is registering when it starts up. So there's really no problem about these additional sidecar services running when you basically deploy your service and start it running till it registers itself and say, hey, I'm here. I'm alive. When you stop it, it can kind of deregister. OK, we'll talk a little bit now about this registration deregistration. If I can't deregister, what happens? Just on a loop, that's why they keep playing again, so you can look at it while we talk over it again. OK, from the current slide. OK, the advantage of registration dynamic, as we saw, basically, service registers themselves. Disadvantages are essentially that the client now depends on console, so if we have to switch, basically, our registration services, we have to go and edit all of our particular services and say, we're no longer using console, we switch to Sky DNS, if everyone rewrite everything. And it doesn't work for apps, so we don't own. And again, we still get the problem of no health checks. We still got health checks. But we're all, everyone with me, no burning questions you need to ask at this point. I'm determined by the end of this, you will all be experts on service discovery. OK, health checks. So the problem we have before was essentially what we're doing is this weird thing, kind of when we look at it, that we're saying, OK, stop the server and then we'll call the first one. I keep saying, should we call the first one? It'll time out because basically you can't talk to the server. And then essentially, then it will call our second server and not be correct. And that call to the first server is pretty wasteful. That server's not up while we're calling it. So what we really want to do is make sure that list is managed such that we know the items in the list that are currently passing a health check that says the service registry could talk to them. And when it talked to them or when they talked to it, it knew that they were alive as of a certain time ago. Now, we still can't get away without retry. The problem is that essentially, let's imagine that I use a model where my service goes and talks to the service registry and says, I'm alive every so often. That's one model. So we call it a lease. I go to the service registry and I say, I'm alive. And a few minutes later, I go to it and I say, I'm alive. And I have to say, I'm alive with a certain frequency to avoid it saying, well, I don't think he's alive because I haven't talked to him in a while. But the problem could be you could get a list of the server and you could say, I'm going to refresh my list on the server to see which servers I need to use every so often. And it's getting basically a lease every so often. And in those time windows, I could ask it for data. I could get data back saying it's alive. I could then use that stale data to make a call. And that service could be down even though because I haven't got fresh information from my service registry saying it's down. Or it could go down during that lease window. And the service says to me, it's alive, but actually it's not. It was alive and it's currently operating on its lease. So you still need to retry in the event of failure because you still may fail. But hopefully, we have a better handle now. If we get the frequencies right, we'll have less instances of that and more instances of us being able to say, OK, I know this service is unhealthy. I'm not talking to it. And there's very useful things about deployments. We want to take servers down where essentially you're probably taking them down for long enough that essentially that would make sense to use a health check. So generally there are two models of health checks. Push versus pull, essentially. So either the client has to say to the registry, I'm alive. And it says, OK, you've got a lease for five minutes. Come back to me and talk to me again in five minutes. I'm alive. The advantage of that model, the lease model, is that essentially the service, if it doesn't talk to you because it's gone down, it's dead. It doesn't have to deregister anyway with you. Disadvantages, you have to have some kind of timer in your service saying, I need to talk to the registration service every five minutes, please. So you require some resources on your service in terms of a thread and a timer, et cetera, working. The other model is pull. And that's essentially you register a health endpoint. So you say, I've got an endpoint. It says, tpapnpoint. If you hit it, it's going to give you 200 OK back and say, I am alive. And what happens is the registration service can call you and say, OK, you're alive? And you say, yeah, I'm alive. And that's a very easy model to implement that just requires a call across to you. And that's probably the model most people go for the first time around. But just create a simple HTTP API endpoint that the other guy can call is a very straightforward model. And that's the model we'll actually kind of show you here. Right. So. Oops. Health check. So what happens is here we've registered a health endpoint. We're just going to say 200 OK back. We're saying healthy is true. So we can change it in the config file and actually have one of them say healthy is false. So I'm just cheating that the server is not well. And essentially what happens is when we get a hold of it, we create an agent service check. And the agent service check essentially says, go into this health status endpoint, do it every so often of your five seconds, and time out after three seconds if you can't talk to us. The agent does the service check. That's because essentially that's a local call. I think they need to gossip to communicate back to the service registry, the fact. When I get my list, I then check the status of the service and the list of services. And I want to know that it's passing. So I still get the full list back, but I want to know that they're actually passing. So what we're going to do is, again, we'll run console, same as before, and then we'll run two agents. Two services, rather. And you'll notice, because I've done it before, that it's basically you've got some health status in there, but you can come back. But if you can see it, it's actually got some information saying, hey, I was looking for health status. I can't find it. This is going to run. It's going to say, here's my health status. And it's actually going to start saying, oh, I'm in trouble because this is the one returning false. And it's saying, I'm critical. Saying, I'm critical. So it's just returning false to healthy is telling the service registry that I'm not working. I'm the second one. This one says, OK, I'm in sync. And it's passing. So you can see it's passing. One is critical. Two is passing. One is critical. And two is passing. So essentially, console now knows that this one is dead, and this one is alive. So we'll run a client. We'll run the client. What we want to see this time is that we don't try talking to this server. We go straight to this one. So there we go. We ran it. And we just talk straight to the server. Normally, we go to one first, but we didn't. Again, we'll try it again. So we're just always going straight to two because we know that one is not healthy. Try it third time. Straight to two. So we lose that initial check that says, hey, you're going to talk to the first service. Oh, it's not healthy. Now I ran Robin to the next one. The health check has told us the first one wasn't healthy, and we avoided calling it in the first place. It makes sense. That's the agent service check again. And that's basically us calling the agent service check by Playfabs console library. We're just making the HVJson call for us. OK. Shut that one down. OK. Right. So we'll talk a little bit about zero configuration next. I forgot to set the subscribers first. OK, set the subscribers. Everyone's happy that they get client-side discovery. And you're happy you get how you work with things like console and my extension, etcd, SkyDNS, zookeeper, those kind of things, right? Feel good? If you can go to your colleagues and say, I know everything now. I know what zookeeper is. I know what it's for. Trust me. OK. The other model, which we use all the time, is essentially what we call self-serve discovery, which is essentially some kind of hardware software load balancer. This is what you use, essentially, for your web farm. You stick a load balancer. The load balancer is doing essentially exactly the same job. The load balancer has a pool of servers registered with it that say, essentially, I know that all these services exist. It can do health checks. So it can say, hey, I'm going to talk to these services and I'll direct traffic to them if they're healthy. If they don't respond because they're unhealthy, I won't direct traffic to them. I'll direct it elsewhere. So when your server goes down behind load balancer and generally the traffic gets directed to other live sites. So load balancer can do all this for you, and you don't need to have a service registration tool like console or zookeeper. So why would you want console or zookeeper? Well, a couple of things. First of all, generally your load balancer is external facing. And essentially, it is essentially your firewall. So in order to use the load balancer, you would have to expose your internal services as ports on your load balancer and therefore make them available to anybody outside. So if these are internal services that you don't want to be client facing across the firewall, you don't really want to use your load balancer to do that. And you could say, well, that's just fine. I'll have an internal load balancer. And that internal load balancer will do all my microservices as opposed to my external load balancer, so everybody lives outside. I hope you have a lot of money. Load balancers are expensive, so you may not want to do that. The other thing about those internal load balancers, essentially, is that people even when they can afford production don't tend to be able to afford to run that in all of their environments. So what tends to happen is that they tend to want to use something else in their staging, whatever environments. And actually, if you want the same thing in all your environments, you may be better off actually using something like a console to work with rather than to do your internal microservices rather than having an internal load balancer in production but everywhere else using console to do it. Kind of becomes a pointless. You've gone through all the effort of using console as well. But load balancers are useful for solving the problem certainly when you're externally facing. Oops, a daisy. OK. Avantages to load balancers, they basically come for free. You don't have to do any development for them. They're pretty well organized and done. Disadvantages, load balancers can be a single point of failure. You have to deploy them in pairs. They have to be active, active, or active passive. Generally, you can't have one load balancer, and that basically means it's quite costly and as we said, external versus internal facing. You can combine the service registration and the load balancing. So particularly if you use a software load balancer, so if you use one like HA Proxy or Nginx, what you can tend to find is that they can actually ask a service registry where the services are and then actually then use load balancer amongst those services for you. So console, I think, or console template. Console template is essentially you spin up a console template and you say, monitor my service registry and take this template and go and update this file. So commonly use an Nginx. What happens is that you say, OK, monitor my service registry. When I get new services in there or services drop out, overwrite the Nginx config file, give it a new config file. It will restart and it will then now load balance amongst my new set of services. So console template lets you marry this two world of software load balances with service registries. OK, zero configuration. We've not got a long, but I'll run through a little bit of zero configuration. Zero configuration is this notion that, hey, configuration is messy and painful. And we don't do that with consumer items, right? I take my MacBook home and I plug in a printer at home and my MacBook discovers my printer and how does it do that? So this idea is called DNS service discovery. So it's a version of DNS that says I don't want to just discover the names of my website to go and look at pictures of cats. But I actually want to do something more useful. And so this notion of what called DNS serve and DNS text records. And the DNS serve records essentially say what is the results of the domain name and text and configuration. And the idea is what you query, basically DNS for some pointer records. And they come back like a text record pair. And then you can get from the serve text pair, you can essentially get the IP address and port of the service. You can get an A record, which also helps you do that. Pretty complicated. But it actually works like a box or something like console because console will essentially expose DNS as well as HTTP JSON. So rather than using the HTTP JSON API to console about your services, you could use DNS. But isn't that useful from your own code? The problem is that.NET's built-in DNS classes take the machines DNS server and say, I'm going to root all requests of that. And you can't point it at a different DNS and arbitrary DNS server. So you have to get to a third-party DNS library and you get it in that says, OK, I can point to basically some arbitrary DNS server and ask it questions about my code. Once you've done that, you might as well have got down Playfab's HTTP JSON card and use that to actually ask on so where your services were. But there can be some value to you in some circumstances in having DNS entries basically exposed. Generally, you probably don't want to. You can do this model where effectively you get something like console to be your DNS entry for your machine. And then essentially, you get it to forward request because it doesn't have to your main DNS. But that's really only if you have a lot of value in having DNS over HTTP JSON. Whoops. Too far. The other interesting thing in terms of zero configuration, though, is that essentially, I think all of the service registration tools are essentially just key value stores. And it isn't just that we can store service cover information in there. We can store practically anything in there that we can express as a key value pair in terms of configuration. When we think about configuration for a service, there tend to be three things we can do. One is there are things we can configure at design time. Generally, we should do those in code. And there are things that we can need to configure at runtime. They're generally things that are affected by our environment or the need operationally to adjust the characteristics of the system. And generally, the things where effectively we need to configure them operationally are either things we could put inside a service registry or inside a key value store to say that on this, when we're running and staging, these are the addresses you need to use. When we're running inside QA, these are the addresses you need to use. When we're running in production, these are the addresses and values you need to use. Those are feature switches that's great. They're different in production to basically staging. And so I'm just going to read them straight out of my key value store instead of having them elsewhere. If they're machine dependent, in other words, you can't get them out of a registration service. You need to know what machine you're running on. You put them in environment variables. And that means you can essentially look to a world where you don't have config files anymore. Don't have to manage config files. Don't have to do all that, temperatize config merging when you deploy and go through all that hell. You can just say, I can offload config to be a problem where I put it in environment variables on the machine, or I put it in a centralized registration service, or I'm configuring it in code. It particularly becomes useful with things like Docker, where essentially you tend to have a service per Docker instance. So essentially the environment variables you are configuring really belong to that service rather than anything else. That's a dream which essentially is death to config files. I hate config files. We seem to spend endless amounts of time dealing with config files and template merging strategies. And where does the globalized version live for each individual server? And who controls the ops or development, et cetera. This stuff just makes that problem go away. Cool. Conclusion, what did you should you have learned? Hopefully you have learned the following things. If not, I failed. You basically know that service discovery is a runtime discovery service location. And that if you have basically a service environment with multiple services, you're going to need to provide availability. And you provide availability to provide redundancy. And if you have redundancy, you need to know where to find these servers. You've got more than one. And the way you do that is essentially service discovery. Particularly important in cloud environments where things basically may change their addresses on a regular basis. So you keep a console, et cetera, DS service registries. Just a pretty key value stores. Most of them have some kind of HTTP JSON API. Some support things like DNS, et cetera. And you query them at runtime for configuration information. And all this stuff is fairly straightforward. And most of you should be able to go back to your colleagues and say, hey, I know all about Zookeeper and Console. Because that's pretty much all there is to it. Cool. Smiling faces rather than puzzle faces? We have, I think, literally probably a couple of minutes. So if anyone has any burning questions, then they'd like to stand at the microphone in the middle, which is quite intimidating, or ask me their question. I will repeat it for the purposes of the mic. Feel free. Or you can always catch me later. This stuff is really quite straightforward. If you have any questions you don't feel comfortable asking in front of an audience, just feel come and find me. And I will try and explain. OK, any questions? Everyone good? Everyone happy? You know what service discovery is? You can tick that box. Cool. Thanks very much, guys. Thank you.
 OK.  Hopefully, you are all here to see me talking about service  discovery for.NET developers.  Excellent.  You are in the right room then.  OK.  So this is me.  I've basically doesn't say much apart from the fact  that I'm old.  But this is the point I always call out, which is,  you know, I'm not standing here claiming  I'm some kind of uber smart geek.  I just happen to have done some stuff  that I want to kind of share with you  or understand some stuff.  That's where I work.  The only real message is it has nothing  to do with what I'm talking about,  so I'm not pushing anything on you.  That's an open source project to work on called Brite.  It's a CQRS framework.  That's a Twitter handle for it.  If you want to find out more about that,  please follow at Brite to command.  And you should better then find out more about something  else we do, which just comes out of Huddle.  We support that.  OK, what's today's agenda?  So we're going to talk a little bit about something  we already know to begin with.  It'd be nice and easy.  We'll talk about point-to-point services, client server.  What we want to do is talk about why this is a problem,  the problem of availability that relates to point-to-point.  And we'll talk about very simply how we solve that.  Now, all that stuff is fairly straightforward.  And we'll talk about then really why service discovery comes  into play once you start to think about the solution  to basically the availability problems on point-to-point  communication.  Then we'll talk about how that all works.  Talk about basically there's different patterns  we can use, local registry, dynamics of registration.  Then we'll go to health checks.  Then we'll talk a little bit.  There's two forms, client and server side.  We'll focus mostly on client because it's  the more interesting for you guys.  We'll talk a little bit about service discovery.  And then if we have some time towards the end,  we'll talk a little bit about this notion of zero configuration.  Hopefully that is roughly the agenda you're hoping to see.  This includes things like DNSSD.  What I'll expect you to get from the session,  I expect you to find out what service discovery is,  why you might need it, and what are the patterns  that you'll need to understand in order to use service discovery.  I expect you to by the end of this  be able to understand when people start talking about things  like ZooKeeper, ETCD console, what they're talking about,  and why they might have differences,  what those differences mean, and how you can use them  to implement these given patterns.  So one reason to come to this talk  is simply to say, I have no idea what ZooKeeper is,  and I'd like to understand what ZooKeeper is  by the end of this talk.  Now that'll happen.  The other thing I want to try and point out  is that this talk is essentially very straightforward.  There's nothing here that's actually that complicated.  It's just a bit of a mystery sometimes to folks  because you've not had to go and find out about it.  So take this as really a way of short circuiting the process  if you haven't do all the reading and finding out about stuff,  but nothing here is challenging.  So this is a nice, gentle, easy talk.  None of our brains should be stretched too much.  By the end, we can walk away with a warm, positive glow  that we know something new,  and we can laud it over our colleagues by saying,  so you don't understand ZooKeeper then.  All right, great.  Done the slide, just in case anyone didn't have a chance  to take a picture of it earlier.  You can get the slides, and you can go and find the demos there.  Just give a second for someone to pick up  a couple of guys on their phones.  Good, okay.  Point to point, right?  So everyone kind of knows how we do this, right?  So this is a diagram, essentially,  of a client-server interaction.  And we have, essentially, a server.  We're offering some kind of service  with some kind of API.  And a client wants to talk to us and get the service from us.  They want to ask some questions.  How many orders did we make last week?  Can you log this person in for me, basically, to the server?  Can you go and get a hold of my document for me  that you've stored?  And there may be, if we're doing something like HTTP,  there may also be something like a proxy  involved in interaction, right?  And the proxy may say, well, I can cache  a certain amount of these requests,  and I can have some kind of indicator  whether or not they're fresh or stale,  and maybe I call you back and say,  hey, is the version that I have still actually stale,  or is it fresh?  Don't need to get a new one.  Can I serve my copy to the client?  And that's still very much a point-to-point interaction.  It's, we may have a level of pass-through,  but it's what we call point-to-point.  The other thing to bear in mind  is that we talk about client and server here.  It's easy to think by client,  we always mean something running on the user's desktop  or something running in the browser.  But actually, it's just a role.  The client is the person asking for service,  and the server is the person kind of giving you the service.  So this could equally be a model  where we have two services running on our back end,  and those two services talk to each other  in order to fulfill some request for the customer.  Okay?  Right.  So this will be a quick demo.  And what we're gonna do is just demonstrate something to you  that you hopefully will understand already,  which is how we do a quick bit of client-to-server interaction.  Let me find this.  So these are all kind of recorded in Camtasia  to save me being able to talk and type at the same time.  Other people have mastered that particular mystery,  but I find it's almost impossible.  So, okay, let's see.  So here we've got just a self-hosted web service.  Okay, so this is basically hosted  on a given host.reson.port.  We have a configuration file in the self-hosted,  web service just running a web API.  And essentially we have somewhere we say,  there's URI we're actually gonna run.  We've got default value in our config file.  We're saying, okay, we're gonna run on this address.  Okay, so our API runs on a given address.  Here we've got a standard loader set up  for a self-hosted web API service,  looking for a green tracing, formatting,  then we're mapping some routes on the controller.  It's all boilerplate stuff that you guys write every day.  Okay, even if you don't actually use self-hosted  or you use IS.  And here we're exposing two methods on an orders API,  get method and a post method.  And we're just from a client.  I'll go back a second before you there.  It's a bit fast.  Let's scrub back to that bit a second.  It's so lazy.  Sorry guys.  Scrub forward.  Okay, we're just gonna make a standard call  basically across to the server to say,  can I add this order model in  and when I wanna get the content back,  get basically a success code  and display out to the console the current set of orders.  Okay, so I'm just ordering something very basic.  So the details not really that important here.  You see this a thousand times.  I'm just really showing you this  because what we're gonna go through is go  and edit this one later on.  Okay, it's just on replay.  So that's why it's doing that.  Okay.  Oops.  Oh, actually, no, I don't want to.  Sorry, let me just towards the end of this thing.  There should be, back here.  Okay.  So we're gonna run the server here.  It's just gonna run, sit there waiting for requests.  We're gonna run the client,  just a batch file.  And all it will do is make the request to the server.  You'll see the server do some tracing  and we'll actually get a response back,  which basically is an order.  Don't worry about the actual details.  There's just essentially saying  these are the orders on the server.  Okay.  All right.  And we're doing that quite fast deliberately  because you guys have all seen this,  hopefully, and done this thousands of times.  All right.  Let's try and clear that message.  The basic problem with that model is this.  If that server goes offline for whatever reason,  my client is now somewhere out of luck.  It's possible that if we've got a proxy here  and we've cached the responses it sent us earlier  and we don't require revalidation,  we haven't got to do a last modified  or we haven't got to do a e-tag to say,  actually the version I have in my data is still correct.  It's still in time,  that I may be able to grab hold of that kind of orders list  and get a stale version of what's going on.  But in most cases,  I'm going to find that this is simply a problem for me.  I cannot no longer communicate with other servers  and so I stop providing my functionality.  So I have a dependency in A on the uptime of C.  All right. Come on problem.  So how do we solve that?  Well, we solve this problem every day  in a sense when we have a browser and a website  and we have a web farm.  So we just introduce more servers  and we say, well, okay, it doesn't matter if this one's down.  I've got two other servers  and they can service the request for me.  Right. Someone in the back end,  they've got a database that we're all talking to  so the data is consistent.  But A is now going to go to C or D  and we're actually going to then provide service to the user.  Right. So we create availability  by introducing redundancy.  This is becoming more important nowadays  because many people are taking their monolithic architectures  and breaking them up into a microservice architecture.  And there are two kinds of ways that there are two kinds  of schools that essentially are,  you can think of as microservices.  One is all the people like me who used to do SOA  and maybe there's something like Gorilla SOA  where we've essentially dropped the service bus  and went for smart endpoints and done pipes.  But it's essentially an evolution of SOA.  It's all that stuff like people talking about  explicit boundaries and bounded context  and business capabilities and that kind of thing.  The other model is kind of the Unix pipes  and filters model of microservices where they say,  well, we want lots of tiny little services  that we can compose to create new pieces of functionality.  Both those schools are equally valid microservices ideas,  but both of them, when you break up your monolith,  resulting in you having a large number of services  that now will talk to each other.  So one of the problems is that anyone moving  to microservices now has to consider the fact  that they have lots of dependencies between services.  And if those services, the services I depend on  goes down and is not available,  then it has a cascade effect  and much of the service disappears on me.  This is one of the reasons that people talk  about microservices being more operationally complex  because they have this web of dependencies.  So my service A depends upon service B.  If I don't make service B redundant,  then what's gonna happen is that A is not gonna  provide its service when B goes down.  And quite often, B may be down for quite transient  network reasons, or maybe I need to just actually go away  and patch B and install something new on it,  or do a release of my service to B.  It's one of the things we want to do in microservices,  you'll be able to release independently.  So maybe B is something I want to actually upgrade  with a new version, but I want to take down A  just to basically create a new version of B  that defeats all objects of microservices.  So I'm gonna need a redundant version of B.  So this redundancy problem has become something  people have to deal with a lot more.  Now there are other ways around this problem.  You can use something like a lightweight broker  like RabbitMQ, and that can solve the problem for you  of some of the dependencies between your various services  because you just rely on queues instead.  That's called decoupled invocation,  but you still have a slight problem in that your client  has to talk to essentially your RabbitMQ server.  So it still needs to know the address of something,  and that needs to be redundant  because otherwise you've only got one of it,  and you're still in the same problem.  So the trick is to remember you can't have one of anything.  You have to design for failure,  and in a modern microservices rather than  monolith driven world, you really need to focus  on this a lot more.  Before, by the way, everybody's always been distributed  for a long time because your database  is a separate process to your web server,  to your code running in your browser,  so you're already distributed.  There's no way around being distributed nowadays,  but here we're talking about a situation  where you explicitly rely on other processes  to provide part of your functionality,  and that provides a problem.  How does discovery play into this?  Well, the real question becomes,  if I can solve the problem essentially  of E not being available by having C and D  running at the same time in other instances of that service,  but the problem is, how does A know to talk to C or D and not E?  Now, when we look to the point-to-point scenario earlier,  the client simply had the IP address of the server,  it was localhost4672 or whatever it was.  The problem with that is that referred to E.  C and D, if it's on my same box,  are probably running on different ports,  but if they're running on different machines,  they're probably running on different IP addresses.  So the client doesn't know where these are  and probably doesn't know that this one is down.  And service discovery is simply,  how do we solve the problem?  We're telling the client what servers are available  and what's their status.  It's nothing more than that.  Okay.  So I've done this to introduce the question  of who do we talk to?  Then in a single server,  we now probably have a pool of servers,  and discovery tells us basically,  how do we find the pool of servers  so we can load balance between them effectively  to avoid problems of failure?  Does that all make sense?  This stuff is really quite straightforward.  Okay.  So there are two major classes of patterns  we need to be aware of  when we start thinking about service discovery.  We can either do our service discovery  essentially on the client,  or we can do it on the server.  Look at this client and server models of service discovery.  We'll talk a lot more about client-side discovery.  Service-side discovery tends to mean,  essentially, we talk about a load balancer.  Okay.  We'll talk about the client-side model instead.  Okay.  The simplest way for us to understand  what we mean by client-side discovery  is to do something without using any kind of technology  like a zookeeper or console in the mix for the time being.  Just imagine to yourself,  well, one solution would be for our,  let's go back.  One solution would be for A,  to have in its own config file that we ship with A,  a list of all the servers with their addresses.  Right, so I simply give A a list saying,  well, there's C and there's D and there's E,  and these are the addresses they live at,  and you can talk to any of them.  Okay.  So look at demo of that.  But again, this is code that you guys should be,  you know, hopefully already pretty intimately familiar  with as a way of working.  Right, so we're just gonna look at something  that's a variation of the code we showed you earlier  directly in point-to-point.  This is our controller,  and you can see there in the controller  what we're doing before we start the video,  is essentially we have a list of servers.  We have a retry policy we'll talk about in a second.  What we're gonna do is go to our configuration file,  read in that list of servers  by simply taking the config values,  creating a list of server items,  which it will have a URI and a timeout value  to hit that server.  And it's that list we will then use to make our requests.  Okay, so this is list basically saying,  go away to the config file,  basically read from the config file.  This should give us a server list.  For each one of those items,  create a new server item object and add it in.  So the config file,  we're just using a standard configuration section,  and we've just got a whole collection  with a load of elements inside it.  And basically the key part of the elements is the URI.  We have a timeout.  You tend to have timeouts when you're doing  any kind of cross-demo communication.  What it looks like in the config file,  it's just a list of servers.  So basically you've got two key addresses there.  What we're doing with our orders API,  in fact we've added the second config file,  so we can actually have two configurations.  This is a retry policy,  and essentially this is using a library called Poly,  and it says if we get an exception,  all we want to do is take an action,  which is essentially to log an error  and try the next server in sequence.  All that TryNextServer does for us is simply,  we just iterate, increment a counter,  and move on to the next server.  You all want to use something  slightly more sophisticated,  like a round robin algorithm,  but for the purpose of the demo, that works fine.  Essentially we're just gonna do the same thing  we've done before,  which essentially is add an order,  and essentially try and post it to the far side.  So we're gonna run two servers,  run the first instance here,  run the second instance here,  we're gonna run our client.  This is before,  and we'll see what happens is the first server  you can see reacts,  so you can see the detail,  but that's just a trace of it reacting  when we get our inflation back.  Now if I stop the first service,  what you'll see is we time out in the first call,  and then we'll switch over  and call the second service instead.  So we run, we time out,  and then we call the second service,  and we get our response back.  Okay, 30 straight forward,  everyone kind of with me.  So that's, you know,  if you can do that,  which most of you I'm sure feel pretty comfortable with,  you can do client-side service discovery.  So we can all go now,  and nothing else to talk about, right?  But it's straightforward.  Okay.  Sorry, my phone appears to be telling me  about infrastructure alerts,  which someone else can respond to,  but I apologize for it beeping.  Okay.  Okay.  So the advantage of this particular pattern is  it's very straightforward.  Most of us understand the right config files.  I'll read through the list,  how to use a list.  I mean, using a library like Poly,  this none of this is particularly challenging,  and it's fairly easy to do some kind of localized  round-robin algorithm to low balance amongst our servers.  Okay.  We can even use this to actually do proper load balancing,  not simply just coping with failure.  We can actually balance,  especially our clients calling out across all these servers.  So it's an easy way and cheap way  to get load balancing solved is a problem.  Okay.  There are obviously a few issues here.  So the first is that this is quite difficult to manage  in the, you know,  I may be actually operationally deploying new servers,  particularly if I'm in a cloud environment  with new addresses on a quite regular basis.  And if I have to ship a new config file  out to all my clients when I do that,  that can be quite a hard problem of asynchronous.  It's bad enough as this,  if we've got a microservices environment,  and those are all internal servers that I'm running,  where I have to update all their configuration files now  to basically take account of the new servers.  But it's kind of even harder if these things  are actually off-site applications  running on my client space,  where they may choose not to actually update those clients  as often as I want to.  I mean, I've got to not get the new list of servers.  So the problem is a sufficient number of services,  service site is going to break down for you,  and if you've got genuine clients outside your firewall,  that's going to be a problem.  The other issue is there's no health checks in here.  So I'm still talking potentially to servers that are dead,  and the only one I'm going to find out about that problem  is essentially my error on my call,  and I iterate over to the next server.  I put this slide in because you may think,  well, this is just a naughty demo solution that Ian's put in.  We're going to move on to the real solutions later,  and no one would use this kind of model in production.  And the answer is that's not true.  So how many of you use Redis today?  A number of you.  And how many of you use, say, the service that Redis client?  We're doing that.  OK, so the service that Redis client  does its ability to cope with essentially failure.  So Redis has a leader follower model,  and essentially you can talk to the pool of followers  on that given leader to provide availability of your Redis  server, particularly if you're under load.  And what it does essentially, if you're going to get documentation  for it, you provide it a list essentially of hosts,  three to only one where effectively you can read from  and the right one you write to.  So it has redundancy across those read-only hosts.  And that's done by you putting that information in the config  file in your service.  So that's a genuine model that lots of people are using  in production today to provide availability.  It's not just a toy.  However, we want to do something better.  So what we're going to look at is we're  going to look at using registration service.  And the simplest idea here is you have something, essentially,  is a database that we talk to.  And from that database, we are going  to get information about where our services live.  At a really simple level, you could go away  and use any kind of key value store  for this service registry.  So you could just register to hold this.  You could even use your SQL Server database.  But what you're concerned about here  is that this service registry has two things it needs.  The first is you don't want this to be another point of failure  for you, so you need to have redundancy.  So the data needs to be across multiple nodes.  I need to have some way of solving the problem of how  do I find my service registry?  Because I don't want that to be, it's  the chicken and egg problem, effectively.  It's OK, I can talk to my service registry.  It will give me the answer.  Well, where is my service registry?  So those kind of problems of what service registry toolkits  tend to solve.  OK, here's a list of the ones you may have heard of.  ZooKeeper.  ZooKeeper is kind of the oldest and kind of the most famous.  We'll talk about CP versus AP in a second.  It's generally a service register on startup,  and they de-register themselves when they're done.  It's very usable from Java.  Otherwise, you have to work with a C library.  There are some.NET wrappers, but generally can be kind of  a bit of hard work to work with.  Airbnb's Smart Stack, and that tends to work basically  with Hadoop to manage the cluster for ZooKeeper.  Netflix's Eureka.  That's pretty good for rest and Java and design  for use on AWS.  If you nod in that scenario, it's probably  not going to be that helpful to you.  Etcd is a simple distributed key value store  with an HTTP JSON API and Rafa consensus.  It's kind of the granddaddy of the more simple ones.  And of those, there's really SkyDNS and console,  which kind of come further down, which  give you DNS support as well as HTTP JSON API.  Right.  What are these things like CP and Rafa consensus mean?  Who knows about CAP there in the audience?  Most of you have an idea.  OK, so we'll just check for those that you don't.  Consistency, availability, and partition tolerance.  You essentially can't get all three.  So these three distributed systems,  you want to have multiple nodes.  Essentially, I've got a problem.  When I ask multiple nodes a question,  I want to get either a consistent answer.  They all agree who won yesterday's Everton versus Arsenal  game.  An available answer, what was the last score  that you actually had for that game?  And essentially, they can't really  deal with partition tolerance, which essentially says  the network is divided and the nodes can't talk to each other.  But you can, 10 times, I either have to choose CA or CP  rather, which is basically, I will be consistent when I give  you a response, or AP, which means I'll be available when  I give you a response.  I'm going to give you the right answer or nothing at all,  or I'll give you an answer, maybe style.  You can use CA.  Some people say you can't.  You can.  But CA essentially says, in the event of our network partition,  my data center is probably on fire.  And the last thing I care about is  whether I'm either consistent or available.  OK.  So you can see nearly all of these are CP,  apart from Netflix, Eureka, which is AP.  So Netflix cares about availability more than does  about consistency in its answer.  But most of them, you can actually,  their APIs, the CP APIs, respond to allowing you to say,  actually, I'm prepared to have style results.  In other words, be AP.  Yes, that's a question.  I mentioned that it's de-registered.  But if the server is down, the server  didn't actually get to de-registered.  It's still an entry.  There is still an entry.  We'll talk about that later.  So that's a good question.  So the question was, the thing that the service is  de-registering from a service registry,  isn't that a problem if, essentially, the service doesn't  goes down without being able to run de-registration?  It would have an entry in there before it  be saying it's available.  It's not.  We cover health checks.  We actually cover what really happens.  This is a bit of a short slide.  The real version comes later.  OK.  So the examples, we're going to do some examples.  We're going to use console to do our examples  when we talk about code today.  I am not pushing console particularly over any of the  others for any reason.  I have no skin in any of these games.  We like console.  It's very good.  Nice, straightforward HTTP JSON.  There are some reasonable.NET clients available for it.  And it seems to work reasonably well.  OK.  It's free open source.  It's created by Hashicorp.  People know who Hashicorp are.  They create vagrant terraform.  So they're generally in the distributed systems,  management space.  They have quite a number of open source tools.  And vagrant is more lifecycle of virtual machines.  It's a useful tool if you want to run VMs locally.  A lot used to do vagrant up and you get given a VM.  OK.  Provide service discovery by either DNS or HTTP plus  JSON.  We'll show you most of the examples using HTTP and JSON.  But we'll talk a little bit about DNS,  why you might not want to use it as a.NET dev  towards the end.  It has sports health checks.  So you can essentially determine whether or not any  service entries actually are currently live.  You can download it from Hashicorp.  Consulio is the site, effectively.  And it's a useful documentation.  It's quite well documented if you need to use that.  Generally, there are a couple of.NET clients.  They wrap the HTTP JSON API.  So you can just talk, if you want to, you could talk raw or  JSON and HTTP if you wanted to to them.  It's actually quite useful, though, sometimes to use  someone else's abstraction built on that.  The PlayFab one is pretty useful.  They have some documentation.  It's not great.  There are some examples in the project that are probably  better where finding out what it does.  You'll see some code here as well.  So my code's available.  So you can go and have a look at my code  for our world as well.  So what Consul consists of, we need to download from Consul  what's called an agent.  And we run the Consul agent.  It says all the nodes in the cluster.  What we mean is that we run the Consul agent on the same  machine as the service we want to essentially register.  And then we run a number of other Consul agents to  essentially provide the service catalog.  The idea is, basically, when we run our agent, we run it  either client or server mode.  Now, server mode essentially says I am the catalog of all  the services.  And you need to run either three or five of those.  It needs to be an odd number so that you can essentially  have voting.  Because the servers use something called the Raft  algorithm.  And the Raft algorithm basically uses a stupid system  piece of kit, which essentially says I want to make  sure that essentially the data across all of my services  is consistent.  So I'll use voting to make sure that essentially if I try  and write to the leader, all the followers are updated at  the same time.  The client essentially sits with the service on the server.  And it forwards nearly all of its requests to the servers.  So it's a way of you saying, I don't have to worry about  whether servers live.  Because all I need to know is essentially where the agent  that's on the same machine as I am is.  And it knows where the servers are.  So how does it do its service discovery?  Well, the answer is basically it uses a protocol called  gossip.  And essentially, gossip tells it where the servers are  currently running.  So when you talk to the agent, it should know where the  servers are, that it has to talk to the service.  The local agent also does the health checks.  And it calls you and says, I'm going to do the health checks.  It's more efficient.  And then it gives the results back to the centralized  service catalog.  So he's making a local call, usually, for that health check.  A typical console deployment looks something like this.  So you've got, in one data center, you've got three nodes.  One of those nodes is the leader.  All the requests are forwarded to the leader.  And you have two additional servers, which are there for  redundancy and can take over as the leader.  In the event that the leader disappears, they'll get  promoted to become a leader.  And you can survive with three, the failure of one node,  with five, you can survive the failure of a lot more.  And you can also basically have more than one data center.  And the data center can use basically a wide-area network  version of gossip to communicate information about the  service catalog.  Germany console has a number of HDTBA JSON endpoints.  But the one you care about most is agent.  So these are actually direct on the server.  What you tend to want to do is call the local agent to you.  And the local agent itself will forward the request to a  number of these other endpoints.  So the first registration pattern we'll talk about for use  where the service registry is called sidecar.  I don't know that term sidecar.  I prefer the term sidekick.  The different problem with sidecar is, in my memory,  sidecars are attached to the motorcycle, whereas sidekicks  can work alongside the superhero.  And so actually what tends to happen is you tend to have  another process running whose job is to do the registry of  your services rather than your service to the  registry itself.  The reason why that's a significant win, potentially  sometimes, over the client registering itself is you may  not control the client or be able to modify its code.  But you still need to register its existence and the number  of nodes of it that you have.  So we'll do a quick demo.  And we'll do a demo of quite how that actually  works in practice.  Everyone following along?  No burning questions at this point?  OK.  Yeah, but most people tend to, as I've seen,  tend to run their server cluster on Unix and the agents  locally on Windows.  OK.  OK, so what we're doing here is we've added a new Windows  service into the mix.  I use TopShelf to Windows services, by the way, in most  cases.  This is actually just a normal program, actually, to be  fair.  This is just a console application.  The console application is just going to say, I'm going to  go and register essentially a set of services.  Now, all this console application we're going to look  at does is, previously, we had in our configuration file a  list of servers.  And we actually had that in the client.  All we're going to do in this case, you just move that list  of servers into the configuration file of our  sidekick, and it will register them instead.  Then we're going to do this on one machine with different  ports.  But the reality is you would probably want to be doing this  on actually separate actual sidecars running on  individual machines alongside the servers that they're  registering.  But we'll just show you this where I'm doing it.  So you can see the simple transition from the client  has the list, the cycle has the list, and it gives it to the  service registrar.  So here we've got basically our console application, which is  the registration.  Registration essentially reads the configuration file.  We call it an agent service registration class.  We call the console and actually register it.  This is just the same configuration file we had  earlier.  So Playfab's code, which we take in basically by NuGet,  does most of the hard work for us of actually doing this,  figuring out how to call the actually JSON API.  I do register to get clear on the old ones out.  We're in the register afterwards.  In our consumer, now what we have to do essentially is ask  to serve the list of services.  We ask console for the list of services.  We want anything tagged with orders.  So we know that services we're interested in.  We get a hold of and essentially add to our list of  servers, the same as we had before.  But rather than get from a config file, we're getting  more for registry, which is basically the  config file used.  First thing we do is spin up console locally.  We're only going to have one agent because it's a demo.  But essentially, a real infrastructure would use a  number running your catalog and a local agent.  We can get away with the public domain just running one.  It has its own config file.  You can register stuff in the config file.  I wouldn't tend to use that that much, it's less dynamic.  And then we will run a couple of servers.  I'll run the psychic first.  The psychic will register our servers.  You can see the service registration happening there.  We'll run the first service and run the second service.  Then what we'll do is run the client.  We'll see if it behaved the same way as we did before.  We'll basically just talk to the first service from the  list we get from the client registration and get the  response back.  Then we'll do the same we did with the localized config.  We'll stop that first server.  We'll run this.  We'll time out and we'll talk to the second server.  So it's just the same solution we had before, but the  config file is now essentially being turned into  registration and centralized service.  Let me get our response back.  OK, that makes sense.  It's all fairly straightforward.  It's essentially saying, I have a list of servers.  I'm either going to put them in my config file locally with  my actual client.  It's going to know about where the services are.  And it's going to talk to them in a low balance way.  Or you're going to say, that's too difficult for me to  manage.  I don't want to manage all of those pieces.  So what I'm going to do instead is go for a model where I  take those servers and actually register them myself in the  service registry.  And I can essentially pretty much use a similar technique  of putting things in a config file and then actually  registering those with my server for registration.  And then my client, it's going to get hold of that list from  my registration service.  The agent, it's actually what I talk to in it, talks to the  actual service.  Get hold of the list of the servers that meet my  required tier.  They've got a tag on them saying orders.  Then I make that call directly to get hold of my list of  servers and I just start iterating through them and  around robin algorithms the same as before.  So my client takes responsibility.  It's probably calling client-side-server-discovery  for the work of doing the load balancing amongst the  servers on the server side.  People like this model because we don't take a  dependency within our service on the service  registration that we're using.  So my orders API service there doesn't have to know  anything about console or how I'm using it.  Only my side-car service needs to understand that.  So when I start switching from console to Sky DNS, then I  can just simply replace my side-cars.  I never have to go and actually impact my service  directly itself.  It's also very great because if I don't control the  orders API service, someone else wrote it, or it's very  hard for me to synchronize changing it and get its  deployments out, I can simply use service  registration still without actually impacting the code  which may belong to somebody else.  And you've got some things like registrator for using  things like Docker.  Registrator essentially listens for new Docker machines  coming into an environment, looks at exposed ports, and  registers them.  So registrator is a way of getting console to work with  Docker in a straightforward fashion.  Obviously, if we're not using some kind of  automated process like registrator, one of the issues  here is that I have to maintain that list in my side-car  saying, what am I actually registering?  And I have to remember to go away and do that so I deploy  to some new location that may be an issue for me.  Change the port.  I've got to remember to go away and change the side-car  registration of the port at the same time.  So do you have to sync up?  We have a dependency now, essentially, which we have to  be cautious of between our service and the side-car  service, which has to exist and run in order for our service  to register and the service register.  So we have to make sure those dependencies are managed.  This we can rely on things like console itself to manage  basically the availability of that.  We have to be concerned about, is our registration service  going to run?  Do we remember to run it as part of our deployment script,  that kind of thing?  So if registration says, well, if I have control of the  client, I can solve the problem a little bit of forgetting  to actually do my registration, needing to have  the dependency on an additional service, but just saying,  let the orders API service register itself when it starts  up.  So I am going to talk to the registrar and do the work.  And then from the client point of view,  there is no difference.  The client just simply still calls console agent,  gets the list of the services, and essentially stores them  locally, and then does some kind of round robin  out of them to load balance between them.  See that?  Hopefully you guys are kind of picking up that this stuff is  not exactly rocket science, right?  Someone actually, last time I gave this demo, live tweeted  it to his team in the office who had implemented it by the  time that we'd finished talking.  So it's fairly straightforward.  OK, so here basically in our control, we've got the code  that essentially reads from the console, goes to the console  agents API, says give me the list of services in response.  Going to iterate over that list of services, finding that  tag to being orders.  Obviously it has to be somewhere distinguishing all the  services, registered tags, and how you do that.  And create an entry in my own internal service list,  register that.  OK.  And all the only different systems are going to be now how  we look at basically doing the registration of the API  itself.  So we'll flick over in a second.  We've seen this before.  There's a poly policy essentially for doing the retry.  There's our client.  OK, so on our self-hosted server, we're adding now a new  thing called register service along with everything else we  do with a self-hosted service.  And then here we simply say, well, let's talk basically to  a console, effectively, and do an agent registration.  And we'll store our value for our URI in our config file.  So it's the old config file we saw earlier that essentially  says somewhere we have to store our URI in our port so that  when we run, we can identify where we are.  And here we're just saying, here's our timeout.  We've got two defines.  We can actually show you switching between the two,  from ID and essentially the.  So we just store information to register with console.  So we'll start up console again.  And then we'll basically run a server.  And we'll see as you run the server.  It's probably very hard for you to read, but your scroll  past, you can actually see that essentially they've  registered this service.  Second one will come up.  And again, we'll register the service.  And now we're really back to the same model as we had before.  We'll run the client, the client talks to the first service,  says, OK, I've got my response back.  We kill the first client.  Run it again.  We'll timeout talking to the first.  We'll talk to the second, because we have the listener.  So essentially you have availability to survive  across failure of one of your services.  And you provided that information out of a centralized  service registry, which essentially has managed to get  you away from the problem of localize config file, in that  essentially you've got this one location that's much easier  to manage.  And the service itself is registering when it starts up.  So there's really no problem about these additional sidecar  services running when you basically deploy your service  and start it running till it registers itself and say, hey,  I'm here.  I'm alive.  When you stop it, it can kind of deregister.  OK, we'll talk a little bit now about this registration  deregistration.  If I can't deregister, what happens?  Just on a loop, that's why they keep playing again,  so you can look at it while we talk over it again.  OK, from the current slide.  OK, the advantage of registration dynamic, as we saw,  basically, service registers themselves.  Disadvantages are essentially that the client now depends on  console, so if we have to switch, basically, our  registration services, we have to go and edit all of our  particular services and say, we're no longer using console,  we switch to Sky DNS, if everyone rewrite everything.  And it doesn't work for apps, so we don't own.  And again, we still get the problem of no health checks.  We still got health checks.  But we're all, everyone with me, no burning questions you  need to ask at this point.  I'm determined by the end of this, you will all be experts  on service discovery.  OK, health checks.  So the problem we have before was essentially what we're  doing is this weird thing, kind of when we look at it, that  we're saying, OK, stop the server and then we'll call the  first one.  I keep saying, should we call the first one?  It'll time out because basically you can't talk to  the server.  And then essentially, then it will call our second server  and not be correct.  And that call to the first server is pretty wasteful.  That server's not up while we're calling it.  So what we really want to do is make sure that list is  managed such that we know the items in the list that are  currently passing a health check that says the service  registry could talk to them.  And when it talked to them or when they talked to it, it  knew that they were alive as of a certain time ago.  Now, we still can't get away without retry.  The problem is that essentially, let's imagine that I  use a model where my service goes and talks to the service  registry and says, I'm alive every so often.  That's one model.  So we call it a lease.  I go to the service registry and I say, I'm alive.  And a few minutes later, I go to it and I say, I'm alive.  And I have to say, I'm alive with a certain frequency to  avoid it saying, well, I don't think he's alive because I  haven't talked to him in a while.  But the problem could be you could get a list of the  server and you could say, I'm going to refresh my list on  the server to see which servers I need to use every so  often.  And it's getting basically a lease every so often.  And in those time windows, I could ask it for data.  I could get data back saying it's alive.  I could then use that stale data to make a call.  And that service could be down even though because I haven't  got fresh information from my service registry saying it's  down.  Or it could go down during that lease window.  And the service says to me, it's alive, but actually it's  not.  It was alive and it's currently operating on its lease.  So you still need to retry in the event of failure because  you still may fail.  But hopefully, we have a better handle now.  If we get the frequencies right, we'll have less  instances of that and more instances of us being able to  say, OK, I know this service is unhealthy.  I'm not talking to it.  And there's very useful things about deployments.  We want to take servers down where essentially you're  probably taking them down for long enough that essentially  that would make sense to use a health check.  So generally there are two models of health checks.  Push versus pull, essentially.  So either the client has to say to the registry, I'm alive.  And it says, OK, you've got a lease for five minutes.  Come back to me and talk to me again in five minutes.  I'm alive.  The advantage of that model, the lease model, is that  essentially the service, if it doesn't talk to you because  it's gone down, it's dead.  It doesn't have to deregister anyway with you.  Disadvantages, you have to have some kind of timer in your  service saying, I need to talk to the registration service  every five minutes, please.  So you require some resources on your service in terms of a  thread and a timer, et cetera, working.  The other model is pull.  And that's essentially you register a health endpoint.  So you say, I've got an endpoint.  It says, tpapnpoint.  If you hit it, it's going to give you 200 OK back and say,  I am alive.  And what happens is the registration service can call  you and say, OK, you're alive?  And you say, yeah, I'm alive.  And that's a very easy model to implement that just requires a  call across to you.  And that's probably the model most people go for the first  time around.  But just create a simple HTTP API endpoint that the other guy  can call is a very straightforward model.  And that's the model we'll actually kind of show you here.  Right.  So.  Oops.  Health check.  So what happens is here we've registered a health endpoint.  We're just going to say 200 OK back.  We're saying healthy is true.  So we can change it in the config file and actually have  one of them say healthy is false.  So I'm just cheating that the server is not well.  And essentially what happens is when we get a hold of it, we  create an agent service check.  And the agent service check essentially says, go into this  health status endpoint, do it every so often of your five  seconds, and time out after three seconds if you can't talk  to us.  The agent does the service check.  That's because essentially that's a local call.  I think they need to gossip to communicate back to the service  registry, the fact.  When I get my list, I then check the status of the  service and the list of services.  And I want to know that it's passing.  So I still get the full list back, but I want to know that  they're actually passing.  So what we're going to do is, again, we'll run console, same  as before, and then we'll run two agents.  Two services, rather.  And you'll notice, because I've done it before, that it's  basically you've got some health status in there, but you  can come back.  But if you can see it, it's actually got some information  saying, hey, I was looking for health status.  I can't find it.  This is going to run.  It's going to say, here's my health status.  And it's actually going to start saying, oh, I'm in  trouble because this is the one returning false.  And it's saying, I'm critical.  Saying, I'm critical.  So it's just returning false to healthy is telling the  service registry that I'm not working.  I'm the second one.  This one says, OK, I'm in sync.  And it's passing.  So you can see it's passing.  One is critical.  Two is passing.  One is critical.  And two is passing.  So essentially, console now knows that this one is dead,  and this one is alive.  So we'll run a client.  We'll run the client.  What we want to see this time is that we don't try talking  to this server.  We go straight to this one.  So there we go.  We ran it.  And we just talk straight to the server.  Normally, we go to one first, but we didn't.  Again, we'll try it again.  So we're just always going straight to two because we  know that one is not healthy.  Try it third time.  Straight to two.  So we lose that initial check that says, hey, you're going  to talk to the first service.  Oh, it's not healthy.  Now I ran Robin to the next one.  The health check has told us the first one wasn't healthy,  and we avoided calling it in the first place.  It makes sense. That's the agent service check again.  And that's basically us calling the agent service check by  Playfabs console library.  We're just making the HVJson call for us.  OK. Shut that one down.  OK.  Right.  So we'll talk a little bit about zero configuration next.  I forgot to set the subscribers first.  OK, set the subscribers.  Everyone's happy that they get client-side discovery.  And you're happy you get how you work with things like  console and my extension, etcd, SkyDNS, zookeeper, those  kind of things, right?  Feel good?  If you can go to your colleagues and say, I know  everything now.  I know what zookeeper is.  I know what it's for.  Trust me.  OK.  The other model, which we use all the time, is essentially  what we call self-serve discovery, which is essentially  some kind of hardware software load balancer.  This is what you use, essentially, for your web  farm.  You stick a load balancer.  The load balancer is doing essentially exactly the same  job.  The load balancer has a pool of servers registered with it  that say, essentially, I know that all these services  exist.  It can do health checks.  So it can say, hey, I'm going to talk to these services  and I'll direct traffic to them if they're healthy.  If they don't respond because they're unhealthy,  I won't direct traffic to them.  I'll direct it elsewhere.  So when your server goes down behind load balancer and  generally the traffic gets directed to other live sites.  So load balancer can do all this for you, and you don't need  to have a service registration tool like console or zookeeper.  So why would you want console or zookeeper?  Well, a couple of things.  First of all, generally your load balancer is external  facing.  And essentially, it is essentially your firewall.  So in order to use the load balancer, you would have to  expose your internal services as ports on your load balancer  and therefore make them available to anybody outside.  So if these are internal services that you don't want to  be client facing across the firewall, you don't really want  to use your load balancer to do that.  And you could say, well, that's just fine.  I'll have an internal load balancer.  And that internal load balancer will do all my microservices  as opposed to my external load balancer, so everybody  lives outside.  I hope you have a lot of money.  Load balancers are expensive, so you  may not want to do that.  The other thing about those internal load balancers,  essentially, is that people even when they can afford  production don't tend to be able to afford to run that in  all of their environments.  So what tends to happen is that they tend to want to use  something else in their staging, whatever environments.  And actually, if you want the same thing in all your  environments, you may be better off actually using  something like a console to work with rather than to do  your internal microservices rather than having an  internal load balancer in production but everywhere else  using console to do it.  Kind of becomes a pointless.  You've gone through all the effort of using  console as well.  But load balancers are useful for solving the problem  certainly when you're externally facing.  Oops, a daisy.  OK.  Avantages to load balancers, they basically come for free.  You don't have to do any development for them.  They're pretty well organized and done.  Disadvantages, load balancers can be a single point of  failure.  You have to deploy them in pairs.  They have to be active, active, or active passive.  Generally, you can't have one load balancer, and that  basically means it's quite costly and as we said,  external versus internal facing.  You can combine the service registration and the load  balancing.  So particularly if you use a software load balancer, so if  you use one like HA Proxy or Nginx, what you can tend to  find is that they can actually ask a service registry  where the services are and then actually then use load  balancer amongst those services for you.  So console, I think, or console template.  Console template is essentially you spin up a console  template and you say, monitor my service registry and take  this template and go and update this file.  So commonly use an Nginx.  What happens is that you say, OK, monitor my service  registry.  When I get new services in there or services drop out,  overwrite the Nginx config file, give it a new config file.  It will restart and it will then now load balance amongst my  new set of services.  So console template lets you marry this two world of  software load balances with service registries.  OK, zero configuration.  We've not got a long, but I'll run through a little bit of  zero configuration.  Zero configuration is this notion that, hey, configuration  is messy and painful.  And we don't do that with consumer items, right?  I take my MacBook home and I plug in a printer at home and  my MacBook discovers my printer and how does it do that?  So this idea is called DNS service discovery.  So it's a version of DNS that says I don't want to just  discover the names of my website to go and look at  pictures of cats.  But I actually want to do something more useful.  And so this notion of what called DNS serve and  DNS text records.  And the DNS serve records essentially say what is the  results of the domain name and text and configuration.  And the idea is what you query, basically DNS for some  pointer records.  And they come back like a text record pair.  And then you can get from the serve text pair, you can  essentially get the IP address and port of the service.  You can get an A record, which also helps you do that.  Pretty complicated.  But it actually works like a box or something like console  because console will essentially expose DNS as well as  HTTP JSON.  So rather than using the HTTP JSON API to console about your  services, you could use DNS.  But isn't that useful from your own code?  The problem is that.NET's built-in DNS classes take the  machines DNS server and say, I'm going to root all requests  of that.  And you can't point it at a different DNS and arbitrary DNS  server.  So you have to get to a third-party DNS library and  you get it in that says, OK, I can point to basically some  arbitrary DNS server and ask it questions about my code.  Once you've done that, you might as well have got down  Playfab's HTTP JSON card and use that to actually ask on  so where your services were.  But there can be some value to you in some circumstances in  having DNS entries basically exposed.  Generally, you probably don't want to.  You can do this model where effectively you get something  like console to be your DNS entry for your machine.  And then essentially, you get it to forward request because  it doesn't have to your main DNS.  But that's really only if you have a lot of value in having  DNS over HTTP JSON.  Whoops.  Too far.  The other interesting thing in terms of zero  configuration, though, is that essentially, I think all of  the service registration tools are essentially just key  value stores.  And it isn't just that we can store service cover  information in there.  We can store practically anything in there that we can  express as a key value pair in terms of configuration.  When we think about configuration for a service,  there tend to be three things we can do.  One is there are things we can configure at design time.  Generally, we should do those in code.  And there are things that we can need to  configure at runtime.  They're generally things that are affected by our  environment or the need operationally to adjust the  characteristics of the system.  And generally, the things where effectively we need to  configure them operationally are either things we could put  inside a service registry or inside a key value store to  say that on this, when we're running and staging, these  are the addresses you need to use.  When we're running inside QA, these are the addresses you  need to use.  When we're running in production, these are the  addresses and values you need to use.  Those are feature switches that's great.  They're different in production to basically staging.  And so I'm just going to read them straight out of my key  value store instead of having them elsewhere.  If they're machine dependent, in other words, you can't get  them out of a registration service.  You need to know what machine you're running on.  You put them in environment variables.  And that means you can essentially look to a world  where you don't have config files anymore.  Don't have to manage config files.  Don't have to do all that, temperatize config merging  when you deploy and go through all that hell.  You can just say, I can offload config to be a  problem where I put it in environment variables on the  machine, or I put it in a centralized registration  service, or I'm configuring it in code.  It particularly becomes useful with things like Docker, where  essentially you tend to have a service per Docker instance.  So essentially the environment variables you are  configuring really belong to that service rather than  anything else.  That's a dream which essentially is death to  config files.  I hate config files.  We seem to spend endless amounts of time dealing with  config files and template merging strategies.  And where does the globalized version live for  each individual server?  And who controls the ops or development, et cetera.  This stuff just makes that problem go away.  Cool.  Conclusion, what did you should you have learned?  Hopefully you have learned the following things.  If not, I failed.  You basically know that service discovery is a runtime  discovery service location.  And that if you have basically a service environment with  multiple services, you're going to need to provide  availability.  And you provide availability to provide redundancy.  And if you have redundancy, you need to know where to find  these servers.  You've got more than one.  And the way you do that is essentially  service discovery.  Particularly important in cloud environments where things  basically may change their addresses on a regular basis.  So you keep a console, et cetera,  DS service registries.  Just a pretty key value stores.  Most of them have some kind of HTTP JSON API.  Some support things like DNS, et cetera.  And you query them at runtime for  configuration information.  And all this stuff is fairly straightforward.  And most of you should be able to go back to your  colleagues and say, hey, I know all about  Zookeeper and Console.  Because that's pretty much all there is to it.  Cool.  Smiling faces rather than puzzle faces?  We have, I think, literally probably a couple of minutes.  So if anyone has any burning questions, then they'd like to  stand at the microphone in the middle, which is quite  intimidating, or ask me their question.  I will repeat it for the purposes of the mic.  Feel free.  Or you can always catch me later.  This stuff is really quite straightforward.  If you have any questions you don't feel comfortable asking  in front of an audience, just feel come and find me.  And I will try and explain.  OK, any questions?  Everyone good?  Everyone happy?  You know what service discovery is?  You can tick that box.  Cool.  Thanks very much, guys.  Thank you.
Service_Discovery_for__0_1007.png Service_Discovery_for__1008_1427.png Service_Discovery_for__1428_4979.png Service_Discovery_for__4980_8063.png Service_Discovery_for__8064_8135.png Service_Discovery_for__8136_8207.png Service_Discovery_for__8208_8783.png Service_Discovery_for__8784_8831.png Service_Discovery_for__8832_9047.png Service_Discovery_for__9048_9107.png Service_Discovery_for__9108_9299.png Service_Discovery_for__9300_9779.png Service_Discovery_for__9780_10787.png Service_Discovery_for__10788_10835.png Service_Discovery_for__10836_11087.png Service_Discovery_for__11088_11231.png Service_Discovery_for__11232_12167.png Service_Discovery_for__12168_12611.png Service_Discovery_for__12612_14123.png Service_Discovery_for__14124_14903.png Service_Discovery_for__14904_22271.png Service_Discovery_for__22272_22679.png Service_Discovery_for__22680_23267.png Service_Discovery_for__23268_23759.png Service_Discovery_for__23760_24995.png Service_Discovery_for__24996_25583.png Service_Discovery_for__25584_25727.png Service_Discovery_for__25728_26027.png Service_Discovery_for__26028_26819.png Service_Discovery_for__26820_27803.png Service_Discovery_for__27804_28439.png Service_Discovery_for__28440_32675.png Service_Discovery_for__32676_34223.png Service_Discovery_for__34224_38615.png Service_Discovery_for__38616_44267.png Service_Discovery_for__44268_45875.png Service_Discovery_for__45876_47135.png Service_Discovery_for__47136_47339.png Service_Discovery_for__47340_47387.png Service_Discovery_for__47388_49931.png Service_Discovery_for__49932_50159.png Service_Discovery_for__50160_50207.png Service_Discovery_for__50208_52895.png Service_Discovery_for__52896_53063.png Service_Discovery_for__53064_53291.png Service_Discovery_for__53292_54827.png Service_Discovery_for__54828_58823.png Service_Discovery_for__58824_58883.png Service_Discovery_for__58884_59063.png Service_Discovery_for__59064_59171.png Service_Discovery_for__59172_60479.png Service_Discovery_for__60480_60527.png Service_Discovery_for__60528_61535.png Service_Discovery_for__61536_62879.png Service_Discovery_for__62880_63047.png Service_Discovery_for__63048_64475.png Service_Discovery_for__64476_65003.png Service_Discovery_for__65004_70907.png Service_Discovery_for__70908_71075.png Service_Discovery_for__71076_71123.png Service_Discovery_for__71124_75731.png Service_Discovery_for__75732_75959.png Service_Discovery_for__75960_79967.png Service_Discovery_for__79968_80015.png Service_Discovery_for__80016_80159.png Service_Discovery_for__80160_88835.png Service_Discovery_for__88836_90311.png Service_Discovery_for__90312_91471.png