Scalable Realtime Architectures in Python
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Baker, Jim
Jim Baker - Scalable Realtime Architectures in Python This talk will focus on you can readily implement highly scalable and fault tolerant realtime architectures, such as dashboards, using Python and tools like Storm, Kafka, and ZooKeeper. We will focus on two related aspects: composing reliable systems using at-least-once and idempotence semantics and how to partition for locality. - Increasingly we are interested in implementing highly scalable and fault tolerant realtime architectures such as the following: * Realtime aggregation. This is the realtime analogue of working with batched map-reduce in systems like Hadoop. * Realtime dashboards. Continuously updated views on all your customers, systems, and the like, without breaking a sweat. * Realtime decision making. Given a set of input streams, policy on what you like to do, and models learned by machine learning, optimize a business process. One example includes autoscaling a set of servers. Obvious tooling for such implementations include Storm (for event processing), Kafka (for queueing), and ZooKeeper (for tracking and configuration). Such components, written respectively in Clojure (Storm), Scala (Kafka), and Java (ZooKeeper), provide the desired scalability and reliability. But what may not be so obvious at first glance is that we can work with other languages, including Python, for the application level of such architectures. (If so inclined, you can also try reimplementing such components in Python, but why not use something that's been proven to be robust?) In fact Python is likely a better language for the app level, given that it is concise, high level, dynamically typed, and has great libraries. Not to mention fun to write code in! This is especially true when we consider the types of tasks we need to write: they are very much like the data transformations and analyses we would have written of say a standard Unix pipeline. And no one is going to argue that writing such a filter in say Java is fun, concise, or even considerably faster in running time. So let's look at how you might solve such larger problems. Given that it was straightforward to solve a small problem, we might approach as follows. Simply divide up larger problems in small one. For example, perhaps work with one customer at a time. And if failure is an ever present reality, then simply ensure your code retries, just like you might have re-run your pipeline against some input files. Unfortunately both require distributed coordination at scale. And distributed coordination is challenging, especially for real systems, that will break at scale. Just putting a box in your architecture labeled **"ZooKeeper"** doesn't magically solve things, even if ZooKeeper can be a very helpful part of an actual solution. Enter the Storm framework. While Storm certainly doesn't solve all problems in this space, it can support many different types of realtime architectures and works well with Python. In particular, Storm solves two key problems for you. **Partitioning**. Storm lets you partition streams, so you can break down the size of your problem. But if the a node running your code fails, Storm will restart it. Storm also ensures such topology invariants as the number of nodes (spouts and bolts in Storm's lingo) that are running, making it very easy to recover from such failures. This is where the cleverness really begins. What can you do if you can ensure that **all the data** you need for a given continuously updated computation - what is the state of this customer's account? - can be put in **exactly one place**, then flow the supporting data through it over time? We will look at how you can readily use such locality in your own Python code. **Retries**. Storm tracks success and failure of events being processed efficiently through a batching scheme and other cleverness. Your code can then choose to retry as necessary. Although Storm also supports exactly-once event processing semantics, we will focus on the simpler model of at-least-once semantics. This means your code must tolerate retry, or in a word, is idempotent. But this is straightforward. We have often written code like the following: seen = set() for record in stream: k = uniquifier(record) if k not in seen: seen.add(k) process
 Okay, so I'm going to talk about, as we, I guess we're almost have everyone in and this should still just be introductory material about scalable real-time architectures in Python, what do I mean by that? I want you to walk away from this session with a couple key ideas, okay, specifically around partitioning and fault tolerance and how we can achieve that in building such scalable real-time architectures. My focus will be on storm, but the ideas are applicable to other tools as well, such as Spark, streaming, or tools that you might roll yourself. And I think the reason that we want to be doing this is of course we're dealing with more data, but we also want to be more responsive to that data. I will show Python, but more at the end. So I'm core developer Jython, you may have seen my state of Jython lightning talk yesterday. Let me plug this book if you're ever interested in Jython. This is a great book. I work at Rackspace on these types of issues. I have had the chance to work in distributed computing for a while and especially failures at scale. I teach the principles of programming language course occasionally at the University of Colorado, which is fun, but actually is done in Scala, not Python. And I work with this user meetup on storm. We're probably going to be changing this to real-time streaming in the near future. What sort of real-time architectures would you like to build? Well, I can think of a few and I'll just mention a few since we are time constrained. Maybe real-time aggregation. This is your classic approach that you'd be doing in Hadoop, but again streaming. So you're not looking at this on a batch by day basis instead as it comes in you're updating your counts or other aggregate aggregations. Perhaps you're building the dashboard. So some extension of this of the real-time aggregation. I'm particularly interested in the idea of decision-making where you will be responding to information in your environment and taking some action. So what are some of the common real-time characteristics of such systems? Well, you're consuming streams of events. You are being event-oriented. As an event occurs, you may take some action. You may want to compute. You may be doing something downstream. You want to go and minimize the latency from the arrival of that event to that potential computation. Certainly not hours. Ideally, going down to seconds or below in terms of that threshold of latency. Such systems are often called complex event processing or you could call it stream processing. It doesn't matter. These are the concepts. The one thing that we are not doing here is doing hard real-time systems. Oh, one last thing. You might have written one of your own type systems. In fact, I'm going to show you what you might have written in the past. So you might have written something like this. Does this look familiar? Well, it should because it is about as generic as you get with a Unix pipeline. But what I wanted to show here is that you have compositions. You're being able to go and build this pipeline out of reusable filters. Another nice property. You can rerun this pipeline if any intermediate step failed. But some problems. I obviously didn't show some of the details around tailing, whatnot. But there's also these aspects of how do you go and implement these other things such as joins, windows. But most importantly, how would you scale that thing up? I mean, I think we can all think about how that could be done. You might go and in some way describe one set of files as being processed by this machine. One set of files by some other machine. But you have to go and actually manage that partitioning yourself. So at Rackspace, we're using this framework for complex event processing called ESPR, which has very much the same properties or again, this homegrown code. And here's the problem. You need to ensure all relevant events about that given customer. It's in one place. If I'm going to be able to know something about this given customer, I need to bring in all the relevant information. I have to go and put that in one place for that to happen. We have to have some locality. Again, in terms of this Rackspace example where we're implementing global alarms, we want to allow customers some degree of customization. So we don't want to make it too hard coded. Then run some computation. So again, a very classic complex event processing system. And simple sharding, the one that we might just know, be able to do quite readily by say, well, maybe we'll do it all by all by customers and whatnot. It doesn't necessarily get us as far as we'd like because you might have again sharded on some of the key. You might need to shard on multiple keys. So what observations can we make? Obviously small problems are easy. How do you make a small, a large problem easier? You divide and conquer it. To divide and conquer, you need to have some horizontal scaling. We're no longer building systems such that they always will require just that larger machine and said we build so that they can scale to a cluster of such machines. But what do we know? The more machines we add to the mix, the more likely we're going to have failure as well, especially since we like to use commodity systems. And once we have failure, then we have to go and coordinate. So I have seen, and maybe you've seen in your own environments, that sometimes people will propose add Zookeeper or add some coordination system. And that just doesn't go and make the problem go away. Even if Zookeeper is awesome, you still need to go and consider how do you manage failure. So yes, Zookeeper can go and give you all this in its toolbox. And it's fantastic as a consequence. But I'll tell you that that doesn't, you know, just assuming that you have distributed locking in your environment doesn't tell you how to recover from a failure and then going and releasing those distributed locks. The solution is not reboot the cluster. How many have done that? So that's not a solution. We want to be, it's okay for a given node in the cluster to fail, but rebooting the cluster defeats the purpose of running the cluster. So Storm has some terminology which I will introduce by, as we go through, but there is this idea of an events source which we call spouts. We have event processing nodes called bolts and some topology to link it together in a directed acyclic graph. There is strong support for partitioning fault tolerance. These key elements that we started with in terms of thinking about how would I go and be able to build up a problem that I could divide and conquer on. Storm is written in closure, but it exposes a Java API. Hence my use of Jython here, although of course you could use some IPC mechanism to talk with, you know, a CPython for example, and that was actually done in the talk on Tuesday where they were looking at the parsley system for that support. It uses ZooKeeper to manage things, but you don't necessarily see it, although you can use it as a resource for your own coordination and as part of the Apache incubator. Actually there's some other Apache incubator projects that are competing in this space. Probably the most notable would be Spark Stream. It actually looks great in terms of the support you can do with that, especially with its Python integration. I think it's the top contender and has nice properties that one can look at that in terms of being functional that you do not see in Storm. These others, SAMHSA and S4, I don't know, I think that these are not nearly as competitive and there's some interesting stuff around there like 0VM as well that you could potentially be doing. So Storm lets you partition streams so you can break down the size of your problem, that returning to that idea of being able to divide and conquer your problem. If the node fails, Storm will restart it. But here's an even more important aspect of what that means. Oftentimes when we are thinking about building systems, especially distributed systems, we want to think about what are the invariants provided by the underlying framework. What does it give us? And this actually helps explain also why you have this distinction between an event source and Storm and event bolts, these processing nodes. Because sources are the things that you are producing events that have to eventually be acknowledged in Storm. So that you can ensure that all events are in fact eventually processed. And when I said eventually, by the way, it actually means in an eventually consistent scheme. And likewise, this idea of bolts, distinguishing that, because you can always replay for them. Hence this topology and the topology in terms of how many nodes you might have. So this is describing how your cluster is being split up in terms of resources. Here's the perhaps one more important invariance that's being provided. The number of nodes for a given spout, maybe a Kafka event source. The number of nodes for a given bolt, maybe that something that is actually doing some interesting processing of that event is held constant. So during the lifetime of this topology, you know that you always have seven nodes for your Kafka spout. What does that mean in terms of your ability to know exactly what's going on in your environment in order to maintain appropriate counters in other state? It's very strong. So Storm is taking into account one extremely important aspect of how you divided that problem up. Because during the lifetime of this topology, if you always know that someone is handling this problem, and you can always go think about it if we had people here, you can always go to that person. Now you don't have to think about, oh, what happens if we add some additional people? It's nice to be able to scale up problems, but that is a separate issue in terms of scaling up the size of a topology. You would do that outside of a given run at your real-time computation. And if that isn't sufficient motivation, I'll give you an example in just a moment. Okay, I think I'm running a little bit slow, so I'm gonna try to speed up now. But these are most important concepts, your takeaways. So you have computational locality. You know that the events for a given node are the ones that are supposed to be there, because that's how you define your routing. You can route on some sort of what they call field groupings. Again, an example is your customer, a tenant in a cloud, a region of some kind, some way of breaking up the problem space in terms of your events. And again, what possibilities do you have if you knew all of the information you needed to know about a given customer, and it was in one place? It changes how you think about things. So normally when you go and write your queries and say, you know, using a relational database, the data is over there and you bring it to you for a given computation. Instead, in systems like Storm, you move the computation to the data. That is the fundamental idea in a map-reduced system, like Storm, like Hadoop. But unlike something like Hadoop, this is in real time. So you're able to keep all of that data in memory, compute on it in real time, and do something interesting. So I should mention, of course, since it's mapping, you might have multiple customers on the given node, so you have to consider that. But that's an easy distinction to make. So again, you will know that it will be on this node and the only on this node. There are other ways of routing, such as random shuffling, global grouping, which means that there's just one node. Everything goes there. Obviously not scalable, but useful for getting totals. Storm will track that success. All you have to do is consider what your retries are. And there are other ways of doing this in terms of doing exactly once event processing, but knowing that at least once all events were successfully observed and computed on is pretty fantastic. Again, think back to that pipeline, that UNIX pipeline I was showing you. That's that same idea. If the pipeline goes down, I can retry it. Of course, you have to handle retry. But what does that look like? Here's the first Python code. It's easiest. If you've already seen that record, then you can ignore it. Otherwise, process it. If you re-try this computation, then you, if you haven't seen it in the context of actually being successful with it, then you can do that retry. That's a merge function. It can be that simple. But it will depend on the nature of your problem. For instance, if I am doing something that's transactional in its nature, you are wiring me $1,000. You can retry that many times, and I'll be very happy about that. So don't think that retries are always going to be successful. That's the nature of eventually consistent systems. Sometimes you actually have to have strong consistency. There are other ways. You could go and have some sort of balancing compensation or something like that and say, well, give me back that $1,000. Actually, in real wiring of funds, that in fact does happen. Another thing, and you see this in any type of streaming systems, your streaming should not capture everything that comes through. This is that old thing. In order to do your computations, you have to download the Internet. If you're doing web crawling, sure you do that. But again, that's done in an appropriate system. So you have to window. Another caveat would be around there are no query languages. You have to build your own. That's actually kind of interesting. Again, with Parsley has some approaches. I expect that we will see these things emerge over time. Think of these as building blocks. That give you these capabilities. But again, these capabilities are a lot like a UNIX pipeline. Imagine, and we know what we could build with those. So you may have to make your own, maybe use some of these other libraries. Okay, I'm going to go and just skip through this a little bit. Except that I will say that Zookeeper is an important aspect of most systems that you would build. And so you probably will be interacting with in some way. So your spouse are ultimately responsible. They're event sources. They're responsible for ensuring that all events were eventually played through successfully, eventually acknowledged. And you need to go and ensure that appropriate handshaking. So as you are consuming from Kafka, you are updating your offsets in it accordingly. I should tell you that again, as some experienced Zookeeper, if you actually try to do one thing I didn't mention at the very beginning, this is that people run a million events a second through a storm node. And of course, you can have many of those nodes running. If you try to do a million events or more per second against Zookeeper, that's not going to work. You have to do some sort of clever batching just like a storm does. Okay. So you get into things like this in terms of the Kafka handshaking, what that would look like. And I will go back through this in just a moment, but I need to move on. So you can have spouse like this. And some of these are already written mostly in Java or in Scala, but the advantage of storm is it's multilingual in the sense that it all runs on a JVM. You could run feeds like Twitter through. You can pull in data from Cassandra. The important thing to know is it's not that difficult to implement. I'll show you in just a moment. Another thing is that you can push or pull events into the system. There isn't something where storm is just saying it will only accept events at a certain time. You can always push events in. But it will ask you when it wants events as well. So you can use that to help balance things. Again, you're going to be using that topology sizing invariant to know how your work is being split up. So you might go and have something like a real time dashboard. You might be using Rackspace Cloud files or some other cloud provider to send things out. You might have some sort of real-time decision-making. I'm interested in auto scaling. In an environment you might often have contradictory information about how things are working. Converge on some action. That's the point of why you're bringing all that data in one place. You may have something along this line where you are doing some sort of real-time aggregation. A couple minutes I will describe Python on Storm using Clam, which I briefly described in the lightning talk yesterday as well. Let's face it, Python is a great fit for writing your Storm code. We have this system called Clam that allows you to readily wrap your Python classes so they can be readily used from Java. In some cases in just one line of code. I discussed this yesterday where you could go and have some bar-clamp class. This is sort of a hello world example. You can readily in a couple lines that are unique to Clam. Just go and add it. You can construct an Uber jar. I'm in Germany. I can say this. An Uber jar, your one jar or a single jar, build it, has everything and distribute it to Storm. You can use it in this fashion. But here's what it really looks like. Here's some code that you could readily use. Say you have a monitoring spout. You're going to be opening up your connection to Kafka. Again, I actually gave this talk at Rackspace. It looks like there's an internal system that, of course, we talk about publicly called Adam Hopper. But again, some others have a feed like Twitter. You can go and read parse events by, as it's asking for it, from next to next to but again if you have something where you're pushing events you can emit at any time and then you're responsible for managing these callbacks fail and act. What do you do under those circumstances? Perhaps you want to go and do some sort of computation on this. What would it look like? This is the pseudocode associated with it. This is all that you need to, this is the basic thing that you need to implement. You need to implement these three methods. You're pretty much done. How complicated is that? It could be like your UNIX pipeline. It could be something that's really simple or could be more complicated. Again, if I'm trying to weigh events that are telling me, oh, it's going this way and it's going that way and I have to figure out what really is happening. But that's the advantage of having it all in one place. Conclusions about this. Storm, less you horizontally skate out your real-time architecture. You have to consider partitioning and fault tolerance. In fact, to me, these are the key questions you answer. This is how you actually think about what it means to divide and conquer your large problem so that it works on this cluster of machines. Answer these questions and you get to go from what you were previously doing in terms of a simple UNIX pipeline, which was analyzing log files, something very similar, to something that can be scalable and, again, real-time. You can choose your favorite language, but I know what your favorite language here is. It's Python. You could again use some of the mechanisms that are out there in terms of communicating via C Python or whatnot, but you could also use, again, Jython and type directly into this big data system. So I definitely would advocate for it. I should. Your strategies will work with Storm, such as Test Urban Development. Have fun. So this talk is available on my GitHub. Repo Jim Baker talks and any questions? Yes. Yes. Yeah, so that's a great question. Yeah, sure. So the question is the long lines of they currently have multiple data centers. They're running a storm cluster in each of these and they need to go and consolidate that information in one place. Now, the first thing to know is that you do not want to, there are very few systems that will span multiple data centers. And Storm is not one of those systems. Okay. You are going to be running a storm cluster in one data center, another storm cluster in some other data center. The way that you would do the spanning problem is use some queue to ensure that data is pushed to some central data center. You may need to go, of course, consider what failure you might have. It gets more complicated. So you basically move that problem to your queues. Use something like Kafka, for example. Yes. That's the best. Yeah, I mean, there's certainly not something that you can do in Storm because, again, it really is depending on the fact that, you know, it's running on ZooKeeper. And ZooKeeper doesn't span multiple data centers. I mean, yes, you can in a theoretical sense. That's not going to be great. So don't do that. Use systems that actually are proven to work. And you're not just doing some interesting innovation along those lines, which you will find is not fantastic innovation. So we are, we are, so yeah, I don't know if there are any other questions. This is something that I can take out afterwards because I obviously would love to spend lots of time on it. Anything else? So I guess we're done. So please ask me those questions that you might have.
 Okay, so I'm going to talk about, as we, I guess we're almost have everyone in and this  should still just be introductory material about scalable real-time architectures in  Python, what do I mean by that? I want you to walk away from this session with a  couple key ideas, okay, specifically around partitioning and fault tolerance  and how we can achieve that in building such scalable real-time architectures. My  focus will be on storm, but the ideas are applicable to other tools as well, such  as Spark, streaming, or tools that you might roll yourself. And I think the  reason that we want to be doing this is of course we're dealing with more data,  but we also want to be more responsive to that data. I will show Python, but more  at the end. So I'm core developer Jython, you may have seen my state of Jython  lightning talk yesterday. Let me plug this book if you're ever interested in  Jython. This is a great book. I work at Rackspace on these types of issues. I have  had the chance to work in distributed computing for a while and especially  failures at scale. I teach the principles of programming language course  occasionally at the University of Colorado, which is fun, but actually is done  in Scala, not Python. And I work with this user meetup on storm. We're probably  going to be changing this to real-time streaming in the near future. What sort of  real-time architectures would you like to build? Well, I can think of a few and  I'll just mention a few since we are time constrained. Maybe real-time  aggregation. This is your classic approach that you'd be doing in Hadoop, but  again streaming. So you're not looking at this on a batch by day basis instead as  it comes in you're updating your counts or other aggregate aggregations. Perhaps  you're building the dashboard. So some extension of this of the real-time  aggregation. I'm particularly interested in the idea of decision-making where you  will be responding to information in your environment and taking some action. So  what are some of the common real-time characteristics of such systems? Well,  you're consuming streams of events. You are being event-oriented. As an event  occurs, you may take some action. You may want to compute. You may be doing  something downstream. You want to go and minimize the latency from the arrival  of that event to that potential computation. Certainly not hours. Ideally,  going down to seconds or below in terms of that threshold of latency. Such  systems are often called complex event processing or you could call it stream  processing. It doesn't matter. These are the concepts. The one thing that we are  not doing here is doing hard real-time systems. Oh, one last thing. You might have  written one of your own type systems. In fact, I'm going to show you what you  might have written in the past. So you might have written something like this.  Does this look familiar? Well, it should because it is about as generic as you get  with a Unix pipeline. But what I wanted to show here is that you have  compositions. You're being able to go and build this pipeline out of reusable  filters. Another nice property. You can rerun this pipeline if any intermediate  step failed. But some problems. I obviously didn't show some of the details  around tailing, whatnot. But there's also these aspects of how do you go and  implement these other things such as joins, windows. But most importantly, how  would you scale that thing up? I mean, I think we can all think about how that  could be done. You might go and in some way describe one set of files as being  processed by this machine. One set of files by some other machine. But you have  to go and actually manage that partitioning yourself. So at Rackspace,  we're using this framework for complex event processing called ESPR, which has  very much the same properties or again, this homegrown code. And here's the  problem. You need to ensure all relevant events about that given customer. It's in  one place. If I'm going to be able to know something about this given  customer, I need to bring in all the relevant information. I have to go and  put that in one place for that to happen. We have to have some locality. Again, in  terms of this Rackspace example where we're implementing global alarms, we  want to allow customers some degree of customization. So we don't want to make  it too hard coded. Then run some computation. So again, a very classic  complex event processing system. And simple sharding, the one that we might just  know, be able to do quite readily by say, well, maybe we'll do it all by all by  customers and whatnot. It doesn't necessarily get us as far as we'd like  because you might have again sharded on some of the key. You might need to shard  on multiple keys. So what observations can we make? Obviously small problems  are easy. How do you make a small, a large problem easier? You divide and  conquer it. To divide and conquer, you need to have some horizontal scaling. We're  no longer building systems such that they always will require just that larger  machine and said we build so that they can scale to a cluster of such machines.  But what do we know? The more machines we add to the mix, the more likely we're  going to have failure as well, especially since we like to use commodity systems.  And once we have failure, then we have to go and coordinate. So I have seen, and  maybe you've seen in your own environments, that sometimes people  will propose add Zookeeper or add some coordination system. And that just  doesn't go and make the problem go away. Even if Zookeeper is awesome, you still  need to go and consider how do you manage failure. So yes, Zookeeper can go and  give you all this in its toolbox. And it's fantastic as a consequence. But I'll  tell you that that doesn't, you know, just assuming that you have distributed  locking in your environment doesn't tell you how to recover from a failure and  then going and releasing those distributed locks. The solution is not  reboot the cluster. How many have done that? So that's not a solution. We want to  be, it's okay for a given node in the cluster to fail, but rebooting the  cluster defeats the purpose of running the cluster. So Storm has some  terminology which I will introduce by, as we go through, but there is this idea of  an events source which we call spouts. We have event processing nodes called  bolts and some topology to link it together in a directed acyclic graph.  There is strong support for partitioning fault tolerance. These key elements that  we started with in terms of thinking about how would I go and be able to  build up a problem that I could divide and conquer on. Storm is written in  closure, but it exposes a Java API. Hence my use of Jython here, although of  course you could use some IPC mechanism to talk with, you know, a CPython for  example, and that was actually done in the talk on Tuesday where they were  looking at the parsley system for that support. It uses ZooKeeper to manage  things, but you don't necessarily see it, although you can use it as a resource  for your own coordination and as part of the Apache incubator. Actually there's  some other Apache incubator projects that are competing in this space. Probably  the most notable would be Spark Stream. It actually looks great in terms of the  support you can do with that, especially with its Python integration. I think  it's the top contender and has nice properties that one can look at that in  terms of being functional that you do not see in Storm. These others, SAMHSA and  S4, I don't know, I think that these are not nearly as competitive and there's  some interesting stuff around there like 0VM as well that you could potentially  be doing. So Storm lets you partition streams so you can break down the size  of your problem, that returning to that idea of being able to divide and conquer  your problem. If the node fails, Storm will restart it. But here's an even more  important aspect of what that means. Oftentimes when we are thinking about  building systems, especially distributed systems, we want to think about what are  the invariants provided by the underlying framework. What does it give us? And  this actually helps explain also why you have this distinction between an event  source and Storm and event bolts, these processing nodes. Because sources are  the things that you are producing events that have to eventually be  acknowledged in Storm. So that you can ensure that all events are in fact  eventually processed. And when I said eventually, by the way, it actually means  in an eventually consistent scheme. And likewise, this idea of bolts,  distinguishing that, because you can always replay for them. Hence this  topology and the topology in terms of how many nodes you might have. So this is  describing how your cluster is being split up in terms of resources. Here's  the perhaps one more important invariance that's being provided. The number  of nodes for a given spout, maybe a Kafka event source. The number of nodes for a  given bolt, maybe that something that is actually doing some interesting  processing of that event is held constant. So during the lifetime of this  topology, you know that you always have seven nodes for your Kafka spout.  What does that mean in terms of your ability to know exactly what's going on  in your environment in order to maintain appropriate counters in other state?  It's very strong. So Storm is taking into account one extremely important  aspect of how you divided that problem up. Because during the lifetime of this  topology, if you always know that someone is handling this problem, and you can  always go think about it if we had people here, you can always go to that  person. Now you don't have to think about, oh, what happens if we add some  additional people? It's nice to be able to scale up problems, but that is a  separate issue in terms of scaling up the size of a topology. You would do that  outside of a given run at your real-time computation. And if that  isn't sufficient motivation, I'll give you an example in just a moment. Okay, I  think I'm running a little bit slow, so I'm gonna try to speed up now. But these  are most important concepts, your takeaways. So you have computational  locality. You know that the events for a given node are the ones that are supposed  to be there, because that's how you define your routing. You can route on  some sort of what they call field groupings. Again, an example is your  customer, a tenant in a cloud, a region of some kind, some way of breaking up the  problem space in terms of your events. And again, what possibilities do you have  if you knew all of the information you needed to know about a given customer,  and it was in one place? It changes how you think about things. So normally when  you go and write your queries and say, you know, using a relational database, the  data is over there and you bring it to you for a given computation. Instead, in  systems like Storm, you move the computation to the data. That is the  fundamental idea in a map-reduced system, like Storm, like Hadoop. But unlike  something like Hadoop, this is in real time. So you're able to keep all of that  data in memory, compute on it in real time, and do something interesting. So I  should mention, of course, since it's mapping, you might have multiple  customers on the given node, so you have to consider that. But that's an easy  distinction to make. So again, you will know that it will be on this node and the  only on this node. There are other ways of routing, such as random shuffling,  global grouping, which means that there's just one node. Everything goes there.  Obviously not scalable, but useful for getting totals. Storm will track that  success. All you have to do is consider what your retries are. And there are other  ways of doing this in terms of doing exactly once event processing, but knowing  that at least once all events were successfully observed and computed on  is pretty fantastic. Again, think back to that pipeline, that UNIX pipeline I was  showing you. That's that same idea. If the pipeline goes down, I can retry it.  Of course, you have to handle retry. But what does that look like? Here's the  first Python code. It's easiest. If you've already seen that record, then you  can ignore it. Otherwise, process it. If you re-try this computation, then you, if  you haven't seen it in the context of actually being successful with it, then  you can do that retry. That's a merge function. It can be that  simple. But it will depend on the nature of your problem. For instance, if I am  doing something that's transactional in its nature, you are wiring me $1,000.  You can retry that many times, and I'll be very happy about that. So don't  think that retries are always going to be successful. That's the nature of  eventually consistent systems. Sometimes you actually have to have strong  consistency. There are other ways. You could go and have some sort of  balancing compensation or something like that and say, well, give me back that  $1,000. Actually, in real wiring of funds, that in fact does happen.  Another thing, and you see this in any type of streaming systems, your  streaming should not capture everything that comes through. This is that old  thing. In order to do your computations, you have to download the  Internet. If you're doing web crawling, sure you do that. But again, that's done  in an appropriate system. So you have to window. Another caveat would be around  there are no query languages. You have to build your own. That's actually  kind of interesting. Again, with Parsley has some approaches. I expect that we  will see these things emerge over time. Think of these as building blocks.  That give you these capabilities. But again, these capabilities are a lot like a  UNIX pipeline. Imagine, and we know what we could build with those. So you may have  to make your own, maybe use some of these other libraries. Okay, I'm going to go and  just skip through this a little bit.  Except that I will say that Zookeeper is an important aspect of most systems that  you would build. And so you probably will be interacting with in some way. So your  spouse are ultimately responsible. They're event sources. They're responsible  for ensuring that all events were eventually played through successfully,  eventually acknowledged. And you need to go and ensure that appropriate  handshaking. So as you are consuming from Kafka, you are updating your offsets in  it accordingly. I should tell you that again, as some experienced Zookeeper, if  you actually try to do one thing I didn't mention at the very beginning, this is  that people run a million events a second through a storm node. And of course,  you can have many of those nodes running. If you try to do a million events or more  per second against Zookeeper, that's not going to work. You have to do some sort of  clever batching just like a storm does. Okay. So you get into things like  this in terms of the Kafka handshaking, what that would look like. And I will go  back through this in just a moment, but I need to move on. So you can have spouse  like this. And some of these are already written mostly in Java or in Scala, but  the advantage of storm is it's multilingual in the sense that it all  runs on a JVM. You could run feeds like Twitter through. You can pull in data  from Cassandra. The important thing to know is it's not that difficult to  implement. I'll show you in just a moment. Another thing is that you can push or  pull events into the system. There isn't something where storm is just saying it  will only accept events at a certain time. You can always push events in. But  it will ask you when it wants events as well. So you can use that to help balance  things. Again, you're going to be using that topology sizing invariant to know  how your work is being split up. So you might go and have something like a real  time dashboard. You might be using Rackspace Cloud files or some other  cloud provider to send things out. You might have some sort of real-time  decision-making. I'm interested in auto scaling. In an environment you might  often have contradictory information about how things are working. Converge on  some action. That's the point of why you're bringing all that data in one  place. You may have something along this line where you are doing some sort of  real-time aggregation. A couple minutes I will describe Python on Storm using  Clam, which I briefly described in the lightning talk yesterday as well. Let's  face it, Python is a great fit for writing your Storm code. We have this  system called Clam that allows you to readily wrap your Python classes so  they can be readily used from Java. In some cases in just one line of code. I  discussed this yesterday where you could go and have some bar-clamp class. This is  sort of a hello world example. You can readily in a couple lines that are  unique to Clam. Just go and add it. You can construct an Uber jar. I'm in Germany.  I can say this. An Uber jar, your one jar or a single jar, build it, has everything  and distribute it to Storm. You can use it in this fashion. But here's what it  really looks like. Here's some code that you could readily use. Say you have a  monitoring spout. You're going to be opening up your connection to Kafka.  Again, I actually gave this talk at Rackspace. It looks like there's an  internal system that, of course, we talk about publicly called Adam Hopper. But  again, some others have a feed like Twitter. You can go and read parse events  by, as it's asking for it, from next to next to but again if you have something  where you're pushing events you can emit at any time and then you're  responsible for managing these callbacks fail and act. What do you do under  those circumstances? Perhaps you want to go and do some sort of computation on  this. What would it look like? This is the pseudocode associated with it. This is  all that you need to, this is the basic thing that you need to implement. You need  to implement these three methods. You're pretty much done. How complicated is that?  It could be like your UNIX pipeline. It could be something that's really simple  or could be more complicated. Again, if I'm trying to weigh events that are  telling me, oh, it's going this way and it's going that way and I have to figure  out what really is happening. But that's the advantage of having it all in one  place. Conclusions about this. Storm, less you horizontally skate out your real-time  architecture. You have to consider partitioning and fault tolerance. In fact,  to me, these are the key questions you answer. This is how you  actually think about what it means to divide and conquer your large problem so  that it works on this cluster of machines. Answer these questions and you get to go  from what you were previously doing in terms of a simple UNIX pipeline, which  was analyzing log files, something very similar, to something that can be  scalable and, again, real-time. You can choose your favorite language, but I  know what your favorite language here is. It's Python. You could again use  some of the mechanisms that are out there in terms of communicating via C  Python or whatnot, but you could also use, again, Jython and type directly into  this big data system. So I definitely would advocate for it. I should. Your  strategies will work with Storm, such as Test Urban Development. Have fun. So this  talk is available on my GitHub. Repo Jim Baker talks and any questions? Yes.  Yes.  Yeah, so that's a great question. Yeah, sure. So the question is the long lines of  they currently have multiple data centers. They're running a storm cluster in  each of these and they need to go and consolidate that information in one place.  Now, the first thing to know is that you do not want to, there are very few systems  that will span multiple data centers. And Storm is not one of those systems. Okay.  You are going to be running a storm cluster in one data center, another storm  cluster in some other data center. The way that you would do the spanning  problem is use some queue to ensure that data is pushed to some central data  center. You may need to go, of course, consider what failure you might have. It  gets more complicated. So you basically move that problem to your queues. Use  something like Kafka, for example. Yes. That's the best. Yeah, I mean, there's  certainly not something that you can do in Storm because, again, it really is  depending on the fact that, you know, it's running on ZooKeeper. And ZooKeeper  doesn't span multiple data centers. I mean, yes, you can in a theoretical sense.  That's not going to be great. So don't do that. Use systems that actually are  proven to work. And you're not just doing some interesting innovation along  those lines, which you will find is not fantastic innovation. So we are, we are,  so yeah, I don't know if there are any other questions. This is something that I  can take out afterwards because I obviously would love to spend lots of  time on it. Anything else? So I guess we're done. So please ask me those  questions that you might have.
Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_0_383.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_384_815.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_816_875.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_876_1007.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_1008_1127.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_1128_1655.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_1656_1715.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_1716_1871.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_1872_2135.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_2136_2459.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_2460_2615.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_2616_2927.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_2928_2987.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_2988_3059.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_3060_3119.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_3120_4307.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_4308_5843.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_5844_6575.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_6576_7523.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_7524_10619.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_10620_14231.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_14232_15515.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_15516_15959.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_15960_20567.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_20568_23291.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_23292_23591.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_23592_23687.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_23688_23903.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_23904_23999.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_24000_24179.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_24180_24635.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_24636_26159.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_26160_27947.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_27948_30011.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_30012_30287.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_30288_32399.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_32400_33371.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_33372_33671.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_33672_34319.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_34320_38627.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_38628_38687.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_38688_38867.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_38868_39191.png Jim_Baker_-_Scalable_Realtime_Architectures_in_Python_39192_42758.png