The Beating Heart of CQRS (or, Actor-based Concurrency and the CLR)
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Blasucci, Paulmichael
It's not enough to simply model a domain as commands and events. A robust solution needs to deliver said commands and events (collectively, hereafter: messages) quickly and reliably. MessageRouter, a recently open-sourced tool developed at Quicken Loans, solves exactly this problem.Using CQRS as a backdrop, this talk will explore the use of actor-based concurrency (via F#'s built-in asynchronous programming features) to build the message-dispatching core of a real-world system. In addition to the purely technical aspects of MessageRouter, this session will also discuss solutions to some of the challenges faced in building polyglot enterprise software on the common-language runtime (i.e. Mono or .NET). While no prior exposure to CQRS or F# is assumed, attendees will benefit from having, at least, a basic working knowledge of the common-language runtime.
 Alright, so welcome. Welcome. Are you enjoying the conference so far? Cool. Before we dig in, just a quick show of hands. How many people here are comfortable developing in C-sharp? Excellent. How many people here are comfortable developing in VB.net? Excellent. I love that there's always a couple in every room. How many people here are comfortable developing in F-sharp? Wow. Awesome. So this is good. This is a little something for everyone here, hopefully. My name is Paul Blizzucci. I'm on Twitter at P Blizzucci or go to GitHub for P Blizzucci. A little bit about me. I've been building software professionally for 17 years. I am the co-founder of the Nashville F-sharp Meetup, a co-organizer of the New York City F-sharp Meetup, and in 2014 and 2015, Microsoft awarded me an MVP for my work in the.NET community. So what we're going to talk about today is the beating heart of CQRS or actor-based concurrency in the CLR. What we're really going to talk about is a library, cleverly enough called message router that routes messages. What we mean by that is it's going to basically in memory dispatch strongly typed messages to things that can process those messages. This came about because we had a CQRS and event sourcing system in my company that we were building. I should mention we've open sourced the library since building it, so we'll get to that in a bit. But how many people are familiar with CQRS? Awesome. How many people are familiar with event sourcing? Great. So we won't spend much time on this because the talk is really not about CQRS or event sourcing, but briefly for people who don't know, CQRS is command query responsibility separation. It says that you use one path through your code and one set of models to handle the domain logic, the write, the things that will actually change your system. And then you have a separate path through your code with a separate model where you serve optimized queries in a read-only fashion. And this helps you scale in a number of scenarios, particularly if you can tolerate what we call eventual consistency, meaning your queries might be a second or two behind the actual latest state of the data. Event sourcing, meanwhile, is this notion of storing your data not as the current state like we normally would in a relational data model, but as a series of changes over time. And you can always get back to the current state by replaying those changes in memory. The canonical example of this is your checking account or an accountant's ledger. They don't actually change values. A mistaken debit is corrected by then adding a credit. It's a series of events and you can total them up to find out what your current balance is. So that's all we're going to talk about for that. At a high level, architecturally, this is sort of what we wind up with. We wind up with a client, typically a web browser, that's interacting with some services through a well-known interface. In this case, you know, REST, so HTTP, create, you know, post, put, delete the whole bit. The query sides will read from a cache or optionally from a data store if there's no data in the cache. The command side will put the messages into a queue and then some other service will actually process those commands. What we're actually interested in today is this little orange box up here, message router. This is a library which is fed command and event messages and routes them to the appropriate message handler. And actually, since we're in the UK, this is not the view from 35,000 feet, it's the view from 10.7 kilometers. Everything you're going to see in this presentation is open source. You can go to this URL here to see exactly the code we're looking at. This is a little snapshot of the solution. One of the other things we're going to talk about in this talk is we very specifically selected F-sharp for certain reasons to write this library, but the intention was to have it consumed not only from F-sharp but from other languages on the CLR, other teams within our company that were using C-sharp, that were using VB.net. So we'll talk a bit about what that means for the design of a library. Hopefully, time-willing, we'll also get into some nitty-gritty details of some of the F-sharp stuff, but we're going to look at a bunch of different things. The main thing, though, that I want to point out, and this is sort of the best tip I can give anyone, if you're writing a library and you want to say, my library supports the CLR, supports multiple languages on the CLR, is not just an F-sharp library, is not just a C-sharp library, you cannot simply rely on the CLR to do the dirty work for you. You have to code for it, you have to be consciously pursuing it, you have to test for it. So if you're going to say you support C-sharp and VB, in addition to your F-sharp projects, you also need some C-sharp projects and some VB projects to, at the very least, exercise your APIs and make sure that things are smooth, because there's a world of difference between the CLR will manage my types for me versus C-sharp developers have an excellent experience using this API. Documentation goes a long way, but you've really got to think through the API and the testing. So before I dig in, any questions, I know I blew kind of quickly through the CQRS stuff. Awesome. If you have questions at any point or if I'm going too fast, just stick up your hand and we'll go. So what do we mean, what does it mean to really design a library that is going to be consumed from multiple languages on the CLR? Well, it means we're limited in certain ways. We have to balance every language has discrete features and then there are common idioms that cut across languages and we have to balance those two against each other. And in particular for us, this means in message router, our public API is primarily focused around interfaces. Even though in some cases we might have an interface of one method which in a language like F-sharp, we would just use a function and life goes on, that's not going to be the best experience across the board. So we work with interfaces. We keep our generic constraints extremely simple because there is a small subset of generic constraints that are meaningful in all.NET languages and then each language adds additional generic constraints on top that so we have to be very careful of that. We use tasks or the task parallel library as our unit of concurrency at least in our public APIs because we'll see when we look at F-sharp, it offers much more powerful mechanisms for doing concurrency but those are very much not friendly in C-sharp or VB.NET. Fortunately, they play very well with tasks so we use tasks as a unit of work and it turns out we'll see in at least one case the kind of parallelism and concurrency we want. Tasks are actually a really good fit for because it is CPU bound rather than IO bound concurrency which is something that TPL is very, very good at. Cooperative cancellation sort of ties in with tasks. Again, some languages in.NET allow for automatic cancellation and some languages don't so we have to make the cancellation explicit. And then the last thing is this notion of inversion of control sort of as a concept. We don't formally endorse dependency injection or any sort of particular library for doing this but we do make it possible to work with the interfaces and the resolution of the types and whatnot in a way that promotes inversion of control whether you're doing that through higher order functions or through something like AutoFac or what have you all supported. So let's take a look now at some of what this looks like. So, oh, that's the other thing, backing up just a minute. In order to get the greatest reach, we define a common library in C-sharp and this will be the library that consumers of the message router use. This will be the library where you want to define message handlers. All you have to do is reference this one library. We don't make people take in the whole kit and caboodle if all they're doing is defining a simple message handler to handle a simple message. And backing up just so I'm clear when I say message, I do mean either a command or an event routed through the system. For the most part, you can use them interchangeably. We'll see later there is one point in the code where the distinction between command and event becomes important but for the most part, if I say messages, it could be a command, it could be an event, not a big deal. The message router itself we'll see is written in F-sharp and then we have tests in a variety of languages to cover ourselves. So what does some of that C-sharp code look like? Well, first you could see we're working in terms of interfaces and this is almost everything we define in the sort of common space. There's one additional interface which serves as a generic wrapper over type resolution so that you can do inversion of control but basically we've got these five interfaces. Our messages are basically either marker interfaces for either a command or an event and we have handler interfaces which will handle a command or an event and we use a very simple generic constraint to make sure that we have strong typing. So if I implement I handle command add two numbers, I will only be able to receive messages that are I command for add two numbers. So a little bit of simple generic constraints to make sure things route up in a strongly typed fashion. And then the message router interface itself is how external code interacts with the message router. Basically you call route, passing it a message and a couple of callbacks that it will invoke on successful completion or on failure. Does that make sense? Pretty straightforward. Here's where the task bit comes in. In order to handle an event you have to return me a task, I will give you a cancellation token and same idea for command. You'll get a cancellation token and you'll return a task and then the message router will manage the execution of that. So that's where TPL and concurrency comes into play. So not the most complicated model. We've had good luck with a couple of different teams in my company being able to create many, many handlers and messages to support a number of scenarios and they all use the message router implementation that we've written in Fsharp which is fairly robust. Question so far. Terrific. So let's look at an example of what implementing this might look like just for clarity sake in case you're looking at interfaces and having trouble seeing what that looks like in real life. The command itself is very simple. We define a struct with two properties. It implements the marker interface. Our handler. Again, I've omitted some of the like constructory bits but basically we're implementing I handle command for a specific command where that command implements the marker interface. So that's pretty straightforward. And then in the implementation of handle command, it's fairly trivial and instant specific but the important points are we're going to create a new factory and we're going to have, or sorry, we're going to create a new task from the factory and we're going to have that task make use of the cancellation token that we pass in. So what actually goes on in here is sort of irrelevant. In this case, it's the add command. It takes an add command with two integers and it adds them and creates an added event. Interesting side note. For CQRS, for commands versus events, your commands are in the imperative. You want to do something. You want to add. Your events always represent something that happened in the past. They are always given in past tense. So added, summed, you know, corrected, so on and so forth. Questions? Terrific. So what does the message router itself actually look like? Well, it turns out that it's got to do a bunch of things. It's got to load handlers because remember our handlers are written by other teams. They're loaded dynamically. We don't necessarily know anything about them other than that they will meet this public contract. It has to dispatch messages and it has to manage errors because no system is perfect. Errors will occur. This component, though, needs to be very robust because if you'll think back to our original diagram, sort of the way the system works, if the message router falls over, the rest of the system can't really do much. I mean, we can serve read-only queries that will get progressively stellar as time goes on, but it needs to be fairly robust and keep on trucking, as they say. So that's where we were. So F sharp is actually a natural fit for this for a number of reasons. One, things get fairly complicated when you start doing reflective metaprogramming and you start having concurrent code and you start having to manage a number of things. Default immutability in F sharp makes that a lot simpler. A lot of things you just don't have to think about. You don't have to worry about component A accidentally pushing buttons on component B and changing your internal state and chasing weird errors. Async workflows, which I mentioned a little before, are a very powerful mechanism for coding concurrency. In a lot of ways they are more expressive than task parallel library and you can do lots of very powerful things with them. More importantly, built on top of async workflows, you have this notion of mailbox processors or agents, which are how F sharp supports actor-based concurrency. For those of you familiar with Erlang, actor-based concurrency is the idea that two independent objects that share no state communicate only by passing messages to one another. Now in F sharp, actor-based concurrency is limited to a single process. It is not a multi-process mechanism, but it winds up being a great way to add structure, control, and error handling on top of fairly complicated concurrent code inside of a single process. We additionally have this thing called code quotations, which is a way of taking F sharp code and rather than just executing it, giving you back an expression tree that you can programmatically manipulate of the code. It turns out this makes reflective metaprogramming much, much easier to do. I think anyone who's ever worked with any of the reflection API in.NET can agree that anything that makes that easier is a lot better. Active patterns are, backing up a minute, F sharp generally supports what's known as pattern matching, which is a declarative way of decomposing your code by saying what you expect it to look like rather than imperatively peaking into individual values. But it's relatively fixed. There are a finite number of patterns that the compiler knows and understands. C sharp even has this with the limit of it only understands one pattern, the compile time literal. F sharp actually understands about 17 patterns, but it's a finite number. What active patterns does is it lets you extend that. It basically lets you teach the compiler how to pattern match against more things. And again, we use that in message router to make reflective metaprogramming simpler, to sort of tame the unruliness that is the BCL system.reflection API. So we'll see a bunch of examples of that going forward. Questions? Wow. Okay, cool. So what is this code? So what is it actually? So there's two things we'll look at. We'll look at handling a message and loading up a handler for a message. Loading up a handler is actually a lot more complicated. So we'll start with the simple act of handling a message. What I do is I get a call on my route function or my route method on my message router. And when I say have actions here, in these slides when we talk about actions, I don't mean the system.action delegate. I mean you have to load up a handler and create a function that can then execute a message against the handler. We're referring to those as actions. So those are the things that we are going to build with reflective metaprogramming. So we'll look and say, do we have any actions for a particular message? If we do, we spawn a new worker which in this case is just an agent or an actor. We spawn a worker. The worker understands the message that was sent through to the route. And then we post the actions to it. And what it does is it actually asynchronously using fsharp's concurrency mechanisms tries to execute the message against each individual handler, against each message handler. And it accumulates up the results, sorry, accumulates up the errors. The messages themselves are expected to be, the handling of the messages themselves are expected to be, to not return any values. So it collects up any errors and it calls the appropriate callback depending on whether it completed fine or if there were no errors. So the important thing here that's not obvious from this diagram is that there's potentially several actions going on simultaneously for a given worker. And there are potentially many workers going on for at any one time in the message handler. But for each call to a route, we will generate a worker. It will do its thing and then go away. So this actually, this model actually scales up pretty well. What does a lot of this code actually look like? Well, so the first off, this is the function, so working backwards, working backwards from here, this is the function that actually runs the actions. You'll see it's called batch actions. It takes the two callbacks for success and failure. It takes the actual message that we're going to route. And it takes this agent, which is a mailbox processor. This is your actor. This is basically just a queue where you are guaranteed to get messages in a thread safe fashion. And the whole thing is handled, it's wrapped in this little async offset, which is how we know it's an async workflow rather than strictly linear code. But what's cool about asyncs is you can see it looks like we're just writing linear code. Now, the fact is we're writing nonlinear code. There's a bunch of continuations happening and a bunch of thread yielding going on and whatnot. But we don't have to worry about all that. Our code just reads like straight procedural linear straightforward code. We have two main things that we do. One is to, if we've given a command, we actually run the handlers for a command. And if we are given an event, we run the handlers for an event. The main difference between commands and events, if it's not obvious from this code, is that for a given message, you will only ever have one command handler for that command. You can't have seven different command handlers for the same command. You can, however, have as many event handlers as you want for a given event. So that's why we split them out. You'll notice we sort of cheat and our command is actually a list of one command. And then our event is an actual list of events where we actually execute them. We get our results back. So once we decide what our handlers are, command or event, we basically execute them, gathering up the results. We then, based on the results, we will, oh, wrong button, we will either call the completion callback or the error callback. And then this whole thing just returns unit because this operation is not strictly fire and forget, but there's no real meaningful return value from this function because continuation will happen in the callbacks. Questions? Wow. Okay. Cool. So digging in a bit deeper, this is what our actual message router type looks like. When we construct it, we give it a sort of a global error callback that will happen if something goes horribly wrong. So you'll see when we build up in a minute, we have ways of monitoring things. We also give it the list of handlers it should know about. And this resolver is a abstraction over inversion of control so that you can have a way to dynamically look up a different handler based on a message type. We create a cancellation token source, which if you're familiar with TPL, we'll use that as part of our cooperative cancellation of our tasks. And then we define a couple of different things. The first thing we do is here's us actually spawning a worker which takes a message and two callbacks. And you'll see this is that batch actions function that we just looked at on the previous slide. It takes in these inputs and returns us an asynchronous workflow. And what may not be obvious, but I want to stress this, is much like with tasks where you return a task and it hasn't completed yet, it's a thing that you hold onto that will be completed later. Async is the same way. So going back a slide, this function doesn't actually, when you invoke it, it doesn't actually execute. It returns this asynchronous workflow which is a block of code that will be executed at some point in the future. So what we do is we first create that block of code. We then pair it with a cancellation token using this cancel with function which creates an agent for us. This basically says, now I've got my actor, I can send it messages, it will execute code, I can cancel it using my cancellation token. We then also call this, this with monitor function. What this does is what's interesting about agents and actors is we can arrange them into cooperative chains, supervision chains in this case. So we will say, connect this agent to this supervisor and this route x function, this basically just, this provides an additional context to the supervisor about what's actually happening. And what this will do is the supervisor will keep an eye on this agent and if something unforeseen goes wrong, the supervisor will catch this and respond accordingly. Then the last thing we do is just we start it which just starts the agent listening for messages. Now up here at the supervisor we define a trap error function which is very, very simple. It just, it makes sure that errors, unhandled errors get caught. We again pair it with a cancellation token to create an agent and then we start it. So this is actually a very flexible model. In our particular use case, what we mostly do with this is these are unhandled exceptions, these are things we don't expect to happen. We mostly handle it by logging the exception, notifying downstream via like an error email or a specific log message or a scom alert or whatever. And we make sure the system overall is still stable and then we move on. In theory, you could do more complex things with a supervision tree. You could retry certain actions. You could, that's very implementation dependent. For our purposes, we just want to know that an error occurred and move on. It turns out for our particular use cases, knowing that we failed but that the system is still stable and knowing that as quickly as possible is way more important than trying to get something to complete successfully. But that will vary based on use cases. So questions? All right. So finally, this is the actual route function inside our message router. And you can see it corresponds rather nicely, well, it actually doesn't. If you look at this compared to our interface, you'll see that the order of events is, or the order of, sorry, arguments is reversed. Our interface took the message first and then the successful callback and then the error callback. Here they are reversed. The reason why we've done this is this is something you'll see when you try to support multiple languages from a library. In order to satisfy type inference and type resolution across the CLR, you will occasionally have to invert the order of inputs so that you can have two different overloads that are actually different and take different inputs but that the compiler can't tell which one is which. So you have to rearrange the order of arguments between them so that the compiler is smart enough to say, oh, you put the message last. You must mean this overload versus you put the message first. You mean that overload. That's something that you'll just, you'll run into as you're trying to support C sharp and F sharp and VB and all these things. Now, another alternative is to have two different methods with the same inputs. But then it's weird, like why would you call route versus, no, really route this. They're both meant to be called route. That's the clear way to do it. So as a bit of a hacky workaround, you do have to rearrange the order of arguments. And you'll see this all over the BCL. For instance, if you're working with the copy function, the static copy method on the array type, you'll see that there's four or five different overloads where they're basically taking the same parameters, but they have to change the order of them so that the compiler knows, oh, you mean to copy from a native pointer into the array versus copying from the array into the native pointer. It's just a, it's a.NET thing. If you haven't seen it before, trust me, someday you will. So we get the type of our incoming message. We go and look it up in our catalog of handlers, say, hey, do we have a handler? And then we fall into one of these different cases for whether or not we have a handler. If we have either a command handler or an event handler, we spawn a worker and pass it the appropriate message. This arrow here is a shortcut for posting a message to an agent. So if it's command, we post it the run command message. If it's an event, we post it the run event message, which if we go back a second, two slides, you'll see these are the messages here that we're matching against to determine what to do. Where we passed run command, okay, run a command. Where we passed run event, okay, run for the events. So that's what we do in the case of actually having a command or a command handler or some event handlers. In the case that we have no handlers or something went tragically wrong, all we do is post a message to our supervisor. Because remember that supervisor we created on the previous slide, that's an agent also. So in addition to monitoring the worker agents, we can directly send it messages saying, hey, this happened, that happened. In the case of getting a message that doesn't have a handler, we just want to log that, hey, somebody did something funny, we'll look into it later. In the case that something went horribly wrong, we don't really have much else to do except say, hey, something went horribly wrong, make a note of it and make sure everything's okay. Questions? Yeah? So that's our usual, that is a custom form. Yes, this is actually a custom operator that I added, which I mean, it's relatively trivial. It's just aliasing the post method. Just a way of cutting down the sort of parentheses, helping things read a little bit cleaner. You'll find, I'm not a huge fan of custom operators, but this one tends to crop up a lot in F sharp when working with mailbox processors or agents. So it's reasonably simple, I think. Other questions? All right. So that was the easy part of what the message router has to do. It just has to look up, say, do I have a handler? If so, give it a message and go and life goes on. This is the tricky part. This is where we do that look up to actually figure out if we have a handler. So we start at this orange box here. We got a message, known type. Basically, have I seen this type of message before? Not the actual instance of the message, the data could be different, but have I seen the type of the message before? Yes. Okay. Do I have actions? And here we mean basically handlers. Have I already looked up, loaded, compiled? Do I have a handler call ready to go? If I do, great. I spawn a worker. If I don't, nope, I log in error. And that's sort of what we saw on the other slide. And then this is the interesting bit here on the left. I haven't seen this type before. Well, I checked to see is it an event. If it is, do I have at least one handler for it? If I do, I build an action, and again here an action is a function that makes it easy to cache and efficiently call these handlers without having to recompile them, reload them over and over and over again. And once I've built it, I add it to my catalog and then I run the worker. If I didn't have at least one handler for this event, well, I can't do much else, so I log in error. Oh, also if we fail to build the action for one reason or another, we log in error. The command logic is very similar, except that instead of checking for at least one handler, we check that our catalog contains exactly one handler. It would be an error if we initialized the catalog with two different command handlers. So while this looks fairly complicated, the good news is it is lazy and happens exactly once. It's lazy and memorized. So this complex, well, I mean this bit on the left happens, you know, every time you send it a message. This complex bit on the right happens the first time we see a message of a given type and only if we see a message of a given type. So even if we have a handler that understands divide by 27 as a command, if we never send it a divide by 27 command message, we'll never bother to load up that handler. And like I said, we only do it once and then we cache it and it's actually pretty efficient. This lookup over here is just a lookup into a concurrent dictionary, so it's a very relatively speaking cheap lookup. It would be cheaper if we didn't have to be thread safe, but we do. So what are some ways that we make reflection better? I talked about using active patterns and code quotations to make reflective metaprogramming look a bit better. And so here we'll start with we've defined a bunch of active patterns. The first is a active pattern that will recursively walk through an expression tree looking for particular shapes in the expression tree. And when I say expression tree, do people know what I mean by that? Questions about that? Okay, so we've got code as a tree and we're going to programmatically traverse it and we're going to say, is this a method call? Great, then return some method. Is this a lambda function in which case keep walking over the body of the lambda? Is this any other combination of expressions? Well, yeah, we're going to walk through each sub-expression trying to keep track of the ones that return some rather than none. And if we don't match any of these conditions, we've got something we can't really process, so we return none. This sum and none is an option type. Is everyone here familiar with the option type in F sharp? Questions about this? Basically, this is like saying, you know, we didn't find anything that matched, but here we found something that matched. And then we just call the function and it recursively walks over itself. We wrap the whole thing in an active pattern because the way active patterns work, at least partial active patterns. So there are different kinds of active patterns. We're only going to look at partial active patterns here. There are total active patterns as well, which we won't look at. Partial active patterns will return some or none. And that's how you tell the compiler that this match succeeded or no, this match, this is not a match and you should move on. So when we actually try to match against this, in the cases where we return some, we'll get a method handle and where it's none, we'll actually just move on. Similarly, we define a interface active pattern, which now this is what's known as a parameterized pattern because in addition to the value we're matching against, we take an additional input called target that lets us fine tune the behavior. And what we're doing here is we are making sure that whatever suspect type we're matching against is a strict interface implementation of the target type. And that has strict interface is not the best worded function. It's a helper function because the actual logic for finding out if a type implements an interface gets a little crazy in the CLR because you have to account for things like is this an interface? Is this a generic interface? Is this an interface inheriting an interface? Is this a class implementing interface? Is this a class inheriting a class that implements an interface? So things get a little weird there. So we wrap up that into a little helper function. And basically if we do have a match, then we return some and the suspect type, otherwise return none. The important thing here to note is that, and it's sort of not obvious here, but we'll see it later, the target type and the suspect type aren't going to be the same type. In fact, if they were the same type, this call would return false because an interface is not a strict implementation of itself. It just, it can't be. So if we return some target rather than some suspect, it would change the return type of this match statement. And then what we do is because active patterns, while they are cool and they extend pattern matching, they're also functions. So we can treat them as first class functions. So one of the cool things we do here is we partially apply the arguments to our interface function. And now we have a much simpler active pattern that will check for I command instances or I event instances without us having to repeat that boilerplate code over and over and over again. So questions. Okay. Moving on. So now we actually will look at quotations. And quotations look like regular F sharp code, but are set off by these little sort of, you know, angle bracket at sign, at sign angle bracket. And what these tell the compiler is the expression inside of it, don't execute that. Turn that into an expression tree. Turn that into a data structure that I can programmatically manipulate. And we use them in a couple of ways. Starting at the top, this get method for event, it takes in a message type. It uses the message type to get a I handle event type that has the generic argument provided for the generic parameter. Yes, it's very confusing. In the CLR, a generic parameter is when you say like T, but a generic argument is when you give an actual type and move T. Sort of not great wording, but that's what it's called. So we take our open generic and we use the actual message type to say give me a type that is this generic with the generic parameter filled in by this type argument. Once we have that, we create an expression that attempts to invoke the handle function on a particular, on a particular type. And then we pass that, we match that against our method active pattern that we defined on the previous slide. And basically what this one's giving us is if in this expression tree, the expression tree that results from this quotation, we can pull out the method name, which 99.987 percent of the time we can. We then have the actual target type and then we can actually get the actual method from its method name. Because if you know anything about reflective metaprogramming in the CLR, the way you invoke methods on a type is very stringy. You get the string representation of the name and you use that to invoke the method on the type. So we put a bit of strong typing around this by virtue of the code quotation and the active pattern. And then obviously if we have nothing, we raise a quantum flux error because really truly we should never, ever have this. This shouldn't be possible. This means something has fundamentally changed in the internals of reflection in the CLR and we need to start over. So that's what our quantum flux error gives us. This is just a type safe way of getting things so that you don't have to deal with lots of, oh, did I get the string right? Did I, did I fat finger the string? Did I remember to pass the argument to tell it whether it's a case sensitive search or a case insensitive search for the method name? We would just get rid of all that stuff. We were strongly typed in our strongly typed language. We do something, I didn't show it here, but we do something very, very similar for commands. There's a function called get method for command where the only difference is instead of I handle event, we use the I handle command interface. So I didn't show that here, but you can imagine same thing. Now this method to function is interesting. This is where we build that action. Remember I said we create these action functions that we then cache so that we can efficiently invoke the same handler over and over again with different message instances without having to pay the price of recompiling. This is the function where we do that. So we take in a, a resolver which is a generic interface that, or a simple interface that we provide in our core library which lets us resolve a type, basically. It's, it's your basic, you know, IOC dependency injection thing. And it's just an interface so you can use it to implement on top of whatever mechanism you want. We also take in a method info which we will have gotten from a call to get method for event. This actually returns a tuple of the target interface type and, or sorry, the target type and the method info. So the method info part that we return from this function will actually get passed into here. Actually both of these arguments, the type and the method will be these two arguments here. What we do is we first check to make sure that our resolver instance knows that that type exists. So that's what this can resolve does. We then have an expression, or sorry, a quotation for accessing that handler instance inside the resolver. And the reason why we get an expression here is because we can take this expression tree and splice it into this other expression. So what we've got here is two separate quotations that are, so basically two separate expression trees that we are combining together into a larger tree that is the actual function that we want to build. And so basically what we do is once we, sorry, once we've got the tree for getting our handler, we go ahead and create a function that will take in a message and a cancellation token and will actually invoke our method on our target, passing it the inputs, and then we have to, you know, by default it's getting us an object. We want to strongly type the result up to a task because remember ultimately this handler is going to be I handle event dot handle or I handle command dot handle, both of which will return you a task. So now we've got our two expressions or our larger expression with our smaller expression sliced into it. We then compile it into an actual, it's actually a delegate. It's not even a first class function. It's not an F sharp function. I'm sorry, it is an F sharp function. We compile it into an F sharp function and then we wrap it in sum because this could fail. So our method returns sum with our compiled action if we need it and if we couldn't resolve, we return none saying, hey, guess what, we're not going to build an action here. So this is probably the most complicated bit of the code. There's not a lot going on in message router except for this. This is pretty gnarly stuff. But again, what's great about this is the use of quotations has let me write strongly typed F sharp in a way that is identical to if I wasn't doing reflective metaprogramming and yet I'm still doing reflective metaprogramming. For folks who have worked with Lisp or other languages that have strong macro support, it's somewhat similar conceptually. I won't claim that there's any actual similarities because people who are into Lisp will get all over me. But it's conceptually similar to this model of I'm writing what looks like normal code but I'm not treating it like normal code. I'm treating it as expressions. And the main reason we do this is to get strong type safety and to not deal with such clunky reflection APIs. For instance, if you tried to do this using, so C sharp also supports expression trees. It doesn't support quotations but it does support expression trees. If you try to do this in C sharp or VB, what you would have is you would have this messy, messy chain of method calls where you've got a lambda and inside the lambda you've got a call to a let and inside the let you've got a call to a var and inside the var you've got a call to an access. And like it just becomes very clunky and unreadable and not actually strongly typed in the way we would like or in a way that's easy to reason about but F sharp quotations let us do that. So questions about this? Because the most performant and efficient way to do that is to build a delegate and hang on to the delegate. Because ultimately what we have in memory is we have a concurrent dictionary where the key is going to be our message type and the value is going to be a delegate. We don't want to keep looking up the handler every time because that can be expensive. We want to just say for a message type give me a delegate and so it turns out just in the nature of.NET with reflection. Reflectively building a delegate and then caching it is going to be a lot faster than reflectively looking up the handler every time. And if we just held the reference to the handler directly, I suppose technically we could do that but it would be a little bit more awkward to invoke because again it's a reflective call so we'd have to invoke it in sort of a less strongly typed way. The delegate gives us a nice strongly typed wrapper over that if that makes sense. Yes, invoking it by reflection versus invoking the delegate there is over time especially at volume there is an appreciable performance difference. At least on CLR version 4.5 on Windows. Obviously benchmarks are going to vary a little bit. And then finally this is where we use some of those things that we just saw. So you'll see here we use those two methods that contain the quotations to get our interface, our strongly, our closed generic interface. The actual handler itself or the actual, you know, the pointer to the handler, the method info. And the actual, we passed that into here to build up the actual action. Those are the two functions we just saw in the previous slide. We then, remember we start with a list of handler types. These are the types we know about. The resolver is just used to complete building the call. We basically choose any of the ones that implement the interface. So now what we've gone from is matching against an open generic, sorry, a marker interface that is tied to an open generic interface. We've gone and created a concrete interface, a filled in instance of the interface if you will. And now we're going through all our handlers and just finding the handlers that implement that interface. And here we're doing this to build the event handlers. So we want all of the ones that implement that interface. This build action here, this is a poor naming on my part. Method to function actually takes in more arguments. We've gone ahead and partially applied it to get this build action function, which will ultimately take in the, so if we back up one slide, you'll see method to function takes in a resolver, an action and a handler type. Here we've pre-applied the resolver because we're going to use the same resolver all the time. We've pre-applied the handle, the action, you could say, the method info because we got that from our previous call. And what that gives us is a function that we can then pass an interface type to and it will build up our delegate for us. So we choose only the types that match our strongly generically defined interface. We build up delegates for them and then we wrap them into a container. This event handlers is just part of the union that says we have event handlers. Here's the list of them. We have a command handler. We have one of them. It's just a little wrapper to help identify what we've got in our list here. For people not familiar with F-sharp, sequence is basically an Ionumerable. In fact, not basically, it is an Ionumerable. So anywhere you see seek, just mentally insert Ionumerable if that's more comfortable for you. We haven't talked about this much. This vertical bar carrot sign, that's what's known as a pipe operator. This lets you take the trailing argument of one function and put it at the front of the function. And what this lets us do is this lets us build a nice linear pipeline. So this is conceptually very similar to a fluent interface in like C-sharp or JavaScript. But in F-sharp, we do it with this nifty little pipe operator because of a number of not important for this discussion details. So this is our function that builds event handlers. The function to build command handlers looks very, very similar except it calls get method for command instead of get method for event. And instead of just choosing all of the ones that match and then mapping over all of them, it does a little extra work to make sure we have exactly one that comes back. And then in extract handlers, this is sort of our root level thing. We check the message type to say, remember these are the other two active patterns we defined. Is this a command? Okay, build command handlers. Is this an event? Okay, go build event handlers. And then we fall down the chain. If we get a message that does not implement i command or i event, neither of our first two cases will match. So we'll fall into our third case, which is we return an error saying, hey, this message type is not a valid message type. Please try again. Questions? So there you can see we're using the functions that have the quotations in them. Then we're using some active patterns. So we got here a little faster than I thought we would. That's cool, though. In conclusion, I want to just stress a few things. Designing from any languages does require a bit more thought. It does require a bit more care, but it's totally worth it. We had a tremendously positive reception, uptake, and usage from the teams that use this library. And they know nothing about F-sharp. They don't know an action from an actor from a function. They know their C-sharp. They know their VB. That's it. And they've been very, very happy. So you can get that good experience as a library author, as a library maintainer. You do have to do a bit more work, but it is absolutely worth it. That's point one. Point two, and this is a good one to stress. It will be a recurring theme if you hang out in room seven today. F-sharp builds simple solution, helps you build simple solutions to complex problems. Not that routing messages is terribly a complex problem, but managing concurrency, managing reflection, these things can be quite complex. F-sharp gives us a lot of nice tools to help simplify all of this. And this last bullet point is if you take nothing else away from this talk, this is the bullet point to take away. This is real and happening in production now. This code is actually used at one of the largest retail mortgage lenders in the United States on multiple projects and multiple teams. So this is not a toy example. This is not a very niche example. This is not something that's limited to a highly academic field. This is something that a large company is put into production in the real world. So that, if you take nothing else away from this talk, if everything I've said for the past 47 minutes is completely in one ear or not the other, please hang on to this because someday someone will ask you, well, nobody really uses F-sharp in production, do they? No, it turns out they do. And they don't just do it at startups. They don't just do it for scientific research. They don't just do it in green field projects. They do it as extensions to brown field projects. They do it in places where they are not predominantly an F-sharp shop. They do it in polyglot scenarios. This is real. If you want more information about F-sharp, F-sharp.org is the home site for the F-sharp software foundation, which is a tremendous nonprofit that helps steer the life and development of F-sharp as a language and as a community. I encourage you to join if you're at all interested in F-sharp or at least visit the page. They have great information for helping you get started. F-sharp for fun and profit is a terrific reference resource, especially if you're an enterprise developer and you want to learn more about functional programming and more about F-sharp. I highly recommend you go check that out. And the author of that content is actually speaking later today. Say hi, Scott. TryF-sharp.net is a great resource where if you're willing to deal with silver light being enabled, you can actually play around with F-sharp 100% in your browser. So you don't even need to know. There's a billion different ways to get F-sharp on your computer. F-sharp.org details all of them. But if you're really, really like, I just don't want to install anything. I don't want to have to think about anything. I don't want to download anything. Let me just point my browser and play with some F-sharp. TryF-sharp lets you do that. And FS-SNP, which is actually the brainchild of another speaker who's speaking later, I think, tomorrow. Or no, actually later today. This is a great community-sourced collection of various F-sharp code snippets. Categorized and tagged and with comments and you can see lots of different things, get lots of ideas, find some different techniques. I highly recommend digging into all of these. Again, I've been Paul Michael Belazzucci. I'm at PeopleLazzucci on Twitter. I'm at PeopleLazzucci on github.com. If you want to find me on LinkedIn, I'm PeopleLazzucci on LinkedIn. Pretty much anywhere you go on the web, I would be PeopleLazzucci, hopefully. At this point, you guys have been awesome. We've got a few minutes. I'd like to open up. Oh, before I open up to questions, I do have stickers. Come get F-sharp stickers because as any F-sharp developer will tell you, you are not a real F-sharp developer unless you take your corporate laptop, cover it with stickers in the hopes that you will piss off the IT department when you hand it back in in a year. So plenty of free stickers. At this point, I'd just like to open up to more questions. Yeah? Yeah. Absolutely. We have a... Sure. Sure. The question was, can I talk through... We've talked through the library. We haven't talked through an instance of using it. We haven't talked through an application that actually uses it. In the source code itself, there's a very trivial application that uses it in two domains. One domain would be doing basic arithmetic. The other domain would be debits and credits to a checking account. But in real life, where we've actually used it, and I will try to do this without violating my nondisclosure agreements and the intellectual property agreements I had to sign, we have a system that takes in various pieces of supplemental data about the market in real time. And it basically massages them and puts them into a data store that are... So the group I work for supports the folks who buy and sell mortgages in the financial markets to create the funding for them. So we have a system that takes in all of this sort of supernumerary market data and massages it into a shape that our traders can look at and puts it into a data store where they can run reports and get at the information. And the way that works is we will have lots of external services that monitor various external... Sorry. We have lots of services outside of the system that monitor external data points when they get new data. They send a command into the system. The command contains the new data. We run the command to store the data and then separately our traders can go to a web page and issue queries against the system to see the latest data. Modulo a second or two of latency because it is an eventually consistent system. We keep it running pretty snappy but it's not real time. Additional questions? Awesome. You guys have been terrific. Please come get stickers. Thank you.
 Alright, so welcome. Welcome. Are you enjoying the conference so far? Cool. Before we dig  in, just a quick show of hands. How many people here are comfortable developing in C-sharp?  Excellent. How many people here are comfortable developing in VB.net? Excellent. I love that  there's always a couple in every room. How many people here are comfortable developing  in F-sharp? Wow. Awesome. So this is good. This is a little something for everyone here,  hopefully. My name is Paul Blizzucci. I'm on Twitter at P Blizzucci or go to GitHub for  P Blizzucci. A little bit about me. I've been building software professionally for 17 years.  I am the co-founder of the Nashville F-sharp Meetup, a co-organizer of the New York City  F-sharp Meetup, and in 2014 and 2015, Microsoft awarded me an MVP for my work in the.NET community.  So what we're going to talk about today is the beating heart of CQRS or actor-based concurrency  in the CLR. What we're really going to talk about is a library, cleverly enough called  message router that routes messages. What we mean by that is it's going to basically in  memory dispatch strongly typed messages to things that can process those messages. This  came about because we had a CQRS and event sourcing system in my company that we were  building. I should mention we've open sourced the library since building it, so we'll get  to that in a bit. But how many people are familiar with CQRS? Awesome. How many people  are familiar with event sourcing? Great. So we won't spend much time on this because  the talk is really not about CQRS or event sourcing, but briefly for people who don't  know, CQRS is command query responsibility separation. It says that you use one path  through your code and one set of models to handle the domain logic, the write, the things  that will actually change your system. And then you have a separate path through your  code with a separate model where you serve optimized queries in a read-only fashion.  And this helps you scale in a number of scenarios, particularly if you can tolerate what we call  eventual consistency, meaning your queries might be a second or two behind the actual  latest state of the data. Event sourcing, meanwhile, is this notion of storing your  data not as the current state like we normally would in a relational data model, but as a  series of changes over time. And you can always get back to the current state by replaying  those changes in memory. The canonical example of this is your checking account or an accountant's  ledger. They don't actually change values. A mistaken debit is corrected by then adding  a credit. It's a series of events and you can total them up to find out what your current  balance is. So that's all we're going to talk about for that. At a high level, architecturally,  this is sort of what we wind up with. We wind up with a client, typically a web browser,  that's interacting with some services through a well-known interface. In this case, you  know, REST, so HTTP, create, you know, post, put, delete the whole bit. The query sides  will read from a cache or optionally from a data store if there's no data in the cache.  The command side will put the messages into a queue and then some other service will actually  process those commands. What we're actually interested in today is this little orange  box up here, message router. This is a library which is fed command and event messages and  routes them to the appropriate message handler. And actually, since we're in the UK, this  is not the view from 35,000 feet, it's the view from 10.7 kilometers. Everything you're  going to see in this presentation is open source. You can go to this URL here to see  exactly the code we're looking at. This is a little snapshot of the solution. One of  the other things we're going to talk about in this talk is we very specifically selected  F-sharp for certain reasons to write this library, but the intention was to have it  consumed not only from F-sharp but from other languages on the CLR, other teams within our  company that were using C-sharp, that were using VB.net. So we'll talk a bit about what  that means for the design of a library. Hopefully, time-willing, we'll also get into some nitty-gritty  details of some of the F-sharp stuff, but we're going to look at a bunch of different  things. The main thing, though, that I want to point out, and this is sort of the best  tip I can give anyone, if you're writing a library and you want to say, my library supports  the CLR, supports multiple languages on the CLR, is not just an F-sharp library, is not  just a C-sharp library, you cannot simply rely on the CLR to do the dirty work for you.  You have to code for it, you have to be consciously pursuing it, you have to test for it. So if  you're going to say you support C-sharp and VB, in addition to your F-sharp projects,  you also need some C-sharp projects and some VB projects to, at the very least, exercise  your APIs and make sure that things are smooth, because there's a world of difference between  the CLR will manage my types for me versus C-sharp developers have an excellent experience  using this API. Documentation goes a long way, but you've really got to think through  the API and the testing. So before I dig in, any questions, I know I blew kind of quickly  through the CQRS stuff. Awesome. If you have questions at any point or if I'm going too  fast, just stick up your hand and we'll go. So what do we mean, what does it mean to really  design a library that is going to be consumed from multiple languages on the CLR? Well,  it means we're limited in certain ways. We have to balance every language has discrete  features and then there are common idioms that cut across languages and we have to balance  those two against each other. And in particular for us, this means in message router, our  public API is primarily focused around interfaces. Even though in some cases we might have an  interface of one method which in a language like F-sharp, we would just use a function  and life goes on, that's not going to be the best experience across the board. So we work  with interfaces. We keep our generic constraints extremely simple because there is a small subset  of generic constraints that are meaningful in all.NET languages and then each language  adds additional generic constraints on top that so we have to be very careful of that.  We use tasks or the task parallel library as our unit of concurrency at least in our  public APIs because we'll see when we look at F-sharp, it offers much more powerful mechanisms  for doing concurrency but those are very much not friendly in C-sharp or VB.NET. Fortunately,  they play very well with tasks so we use tasks as a unit of work and it turns out we'll see  in at least one case the kind of parallelism and concurrency we want. Tasks are actually  a really good fit for because it is CPU bound rather than IO bound concurrency which is  something that TPL is very, very good at. Cooperative cancellation sort of ties in  with tasks. Again, some languages in.NET allow for automatic cancellation and some languages  don't so we have to make the cancellation explicit. And then the last thing is this  notion of inversion of control sort of as a concept. We don't formally endorse dependency  injection or any sort of particular library for doing this but we do make it possible  to work with the interfaces and the resolution of the types and whatnot in a way that promotes  inversion of control whether you're doing that through higher order functions or through  something like AutoFac or what have you all supported. So let's take a look now at some  of what this looks like. So, oh, that's the other thing, backing up just a minute. In  order to get the greatest reach, we define a common library in C-sharp and this will  be the library that consumers of the message router use. This will be the library where  you want to define message handlers. All you have to do is reference this one library.  We don't make people take in the whole kit and caboodle if all they're doing is defining  a simple message handler to handle a simple message. And backing up just so I'm clear  when I say message, I do mean either a command or an event routed through the system. For  the most part, you can use them interchangeably. We'll see later there is one point in the code  where the distinction between command and event becomes important but for the most part,  if I say messages, it could be a command, it could be an event, not a big deal. The message  router itself we'll see is written in F-sharp and then we have tests in a variety of languages  to cover ourselves. So what does some of that C-sharp code look like? Well, first you could  see we're working in terms of interfaces and this is almost everything we define in the  sort of common space. There's one additional interface which serves as a generic wrapper  over type resolution so that you can do inversion of control but basically we've got these five  interfaces. Our messages are basically either marker interfaces for either a command or  an event and we have handler interfaces which will handle a command or an event and we use  a very simple generic constraint to make sure that we have strong typing. So if I implement  I handle command add two numbers, I will only be able to receive messages that are I command  for add two numbers. So a little bit of simple generic constraints to make sure things route  up in a strongly typed fashion. And then the message router interface itself is how external  code interacts with the message router. Basically you call route, passing it a message and a  couple of callbacks that it will invoke on successful completion or on failure. Does that  make sense? Pretty straightforward. Here's where the task bit comes in. In order to handle  an event you have to return me a task, I will give you a cancellation token and same idea  for command. You'll get a cancellation token and you'll return a task and then the message  router will manage the execution of that. So that's where TPL and concurrency comes  into play. So not the most complicated model. We've had good luck with a couple of different  teams in my company being able to create many, many handlers and messages to support a number  of scenarios and they all use the message router implementation that we've written in  Fsharp which is fairly robust. Question so far. Terrific. So let's look at an example  of what implementing this might look like just for clarity sake in case you're looking  at interfaces and having trouble seeing what that looks like in real life. The command  itself is very simple. We define a struct with two properties. It implements the marker  interface. Our handler. Again, I've omitted some of the like constructory bits but basically  we're implementing I handle command for a specific command where that command implements  the marker interface. So that's pretty straightforward. And then in the implementation of handle command,  it's fairly trivial and instant specific but the important points are we're going to create  a new factory and we're going to have, or sorry, we're going to create a new task from  the factory and we're going to have that task make use of the cancellation token that we  pass in. So what actually goes on in here is sort of irrelevant. In this case, it's  the add command. It takes an add command with two integers and it adds them and creates  an added event. Interesting side note. For CQRS, for commands versus events, your commands  are in the imperative. You want to do something. You want to add. Your events always represent  something that happened in the past. They are always given in past tense. So added, summed,  you know, corrected, so on and so forth. Questions? Terrific. So what does the message  router itself actually look like? Well, it turns out that it's got to do a bunch of things.  It's got to load handlers because remember our handlers are written by other teams. They're  loaded dynamically. We don't necessarily know anything about them other than that they will  meet this public contract. It has to dispatch messages and it has to manage errors because  no system is perfect. Errors will occur. This component, though, needs to be very robust  because if you'll think back to our original diagram, sort of the way the system works,  if the message router falls over, the rest of the system can't really do much. I mean,  we can serve read-only queries that will get progressively stellar as time goes on, but  it needs to be fairly robust and keep on trucking, as they say. So that's where we were. So F  sharp is actually a natural fit for this for a number of reasons. One, things get fairly  complicated when you start doing reflective metaprogramming and you start having concurrent  code and you start having to manage a number of things. Default immutability in F sharp  makes that a lot simpler. A lot of things you just don't have to think about. You don't  have to worry about component A accidentally pushing buttons on component B and changing  your internal state and chasing weird errors. Async workflows, which I mentioned a little  before, are a very powerful mechanism for coding concurrency. In a lot of ways they are  more expressive than task parallel library and you can do lots of very powerful things  with them. More importantly, built on top of async workflows, you have this notion  of mailbox processors or agents, which are how F sharp supports actor-based concurrency.  For those of you familiar with Erlang, actor-based concurrency is the idea that two independent  objects that share no state communicate only by passing messages to one another. Now in  F sharp, actor-based concurrency is limited to a single process. It is not a multi-process  mechanism, but it winds up being a great way to add structure, control, and error handling  on top of fairly complicated concurrent code inside of a single process. We additionally  have this thing called code quotations, which is a way of taking F sharp code and rather  than just executing it, giving you back an expression tree that you can programmatically  manipulate of the code. It turns out this makes reflective metaprogramming much, much  easier to do. I think anyone who's ever worked with any of the reflection API in.NET can  agree that anything that makes that easier is a lot better. Active patterns are, backing  up a minute, F sharp generally supports what's known as pattern matching, which is a declarative  way of decomposing your code by saying what you expect it to look like rather than imperatively  peaking into individual values. But it's relatively fixed. There are a finite number of patterns  that the compiler knows and understands. C sharp even has this with the limit of it only  understands one pattern, the compile time literal. F sharp actually understands about  17 patterns, but it's a finite number. What active patterns does is it lets you extend  that. It basically lets you teach the compiler how to pattern match against more things.  And again, we use that in message router to make reflective metaprogramming simpler, to  sort of tame the unruliness that is the BCL system.reflection API. So we'll see a bunch  of examples of that going forward. Questions? Wow. Okay, cool. So what is this code? So  what is it actually? So there's two things we'll look at. We'll look at handling a message  and loading up a handler for a message. Loading up a handler is actually a lot more complicated.  So we'll start with the simple act of handling a message. What I do is I get a call on my  route function or my route method on my message router. And when I say have actions here, in  these slides when we talk about actions, I don't mean the system.action delegate. I mean  you have to load up a handler and create a function that can then execute a message against  the handler. We're referring to those as actions. So those are the things that we are going  to build with reflective metaprogramming. So we'll look and say, do we have any actions  for a particular message? If we do, we spawn a new worker which in this case is just an  agent or an actor. We spawn a worker. The worker understands the message that was sent  through to the route. And then we post the actions to it. And what it does is it actually  asynchronously using fsharp's concurrency mechanisms tries to execute the message against  each individual handler, against each message handler. And it accumulates up the results,  sorry, accumulates up the errors. The messages themselves are expected to be, the handling  of the messages themselves are expected to be, to not return any values. So it collects  up any errors and it calls the appropriate callback depending on whether it completed  fine or if there were no errors. So the important thing here that's not obvious from this diagram  is that there's potentially several actions going on simultaneously for a given worker.  And there are potentially many workers going on for at any one time in the message handler.  But for each call to a route, we will generate a worker. It will do its thing and then go  away. So this actually, this model actually scales up pretty well. What does a lot of  this code actually look like? Well, so the first off, this is the function, so working  backwards, working backwards from here, this is the function that actually runs the actions.  You'll see it's called batch actions. It takes the two callbacks for success and failure.  It takes the actual message that we're going to route. And it takes this agent, which is  a mailbox processor. This is your actor. This is basically just a queue where you are guaranteed  to get messages in a thread safe fashion. And the whole thing is handled, it's wrapped  in this little async offset, which is how we know it's an async workflow rather than  strictly linear code. But what's cool about asyncs is you can see it looks like we're  just writing linear code. Now, the fact is we're writing nonlinear code. There's a bunch  of continuations happening and a bunch of thread yielding going on and whatnot. But  we don't have to worry about all that. Our code just reads like straight procedural linear  straightforward code. We have two main things that we do. One is to, if we've given a command,  we actually run the handlers for a command. And if we are given an event, we run the handlers  for an event. The main difference between commands and events, if it's not obvious from  this code, is that for a given message, you will only ever have one command handler for  that command. You can't have seven different command handlers for the same command. You  can, however, have as many event handlers as you want for a given event. So that's why  we split them out. You'll notice we sort of cheat and our command is actually a list of  one command. And then our event is an actual list of events where we actually execute them.  We get our results back. So once we decide what our handlers are, command or event, we  basically execute them, gathering up the results. We then, based on the results, we will, oh,  wrong button, we will either call the completion callback or the error callback. And then this  whole thing just returns unit because this operation is not strictly fire and forget,  but there's no real meaningful return value from this function because continuation will  happen in the callbacks. Questions? Wow. Okay. Cool. So digging in a bit deeper, this is  what our actual message router type looks like. When we construct it, we give it a sort  of a global error callback that will happen if something goes horribly wrong. So you'll  see when we build up in a minute, we have ways of monitoring things. We also give it  the list of handlers it should know about. And this resolver is a abstraction over inversion  of control so that you can have a way to dynamically look up a different handler based on a message  type. We create a cancellation token source, which if you're familiar with TPL, we'll use  that as part of our cooperative cancellation of our tasks. And then we define a couple  of different things. The first thing we do is here's us actually spawning a worker which  takes a message and two callbacks. And you'll see this is that batch actions function that  we just looked at on the previous slide. It takes in these inputs and returns us an asynchronous  workflow. And what may not be obvious, but I want to stress this, is much like with tasks  where you return a task and it hasn't completed yet, it's a thing that you hold onto that  will be completed later. Async is the same way. So going back a slide, this function  doesn't actually, when you invoke it, it doesn't actually execute. It returns this asynchronous  workflow which is a block of code that will be executed at some point in the future. So  what we do is we first create that block of code. We then pair it with a cancellation  token using this cancel with function which creates an agent for us. This basically says,  now I've got my actor, I can send it messages, it will execute code, I can cancel it using  my cancellation token. We then also call this, this with monitor function. What this  does is what's interesting about agents and actors is we can arrange them into cooperative  chains, supervision chains in this case. So we will say, connect this agent to this supervisor  and this route x function, this basically just, this provides an additional context  to the supervisor about what's actually happening. And what this will do is the supervisor will  keep an eye on this agent and if something unforeseen goes wrong, the supervisor will  catch this and respond accordingly. Then the last thing we do is just we start it which  just starts the agent listening for messages. Now up here at the supervisor we define a trap  error function which is very, very simple. It just, it makes sure that errors, unhandled  errors get caught. We again pair it with a cancellation token to create an agent and  then we start it. So this is actually a very flexible model. In our particular use case,  what we mostly do with this is these are unhandled exceptions, these are things we don't expect  to happen. We mostly handle it by logging the exception, notifying downstream via like  an error email or a specific log message or a scom alert or whatever. And we make sure  the system overall is still stable and then we move on. In theory, you could do more complex  things with a supervision tree. You could retry certain actions. You could, that's very  implementation dependent. For our purposes, we just want to know that an error occurred  and move on. It turns out for our particular use cases, knowing that we failed but that  the system is still stable and knowing that as quickly as possible is way more important  than trying to get something to complete successfully. But that will vary based on use cases. So  questions? All right. So finally, this is the actual route function inside our message  router. And you can see it corresponds rather nicely, well, it actually doesn't. If you  look at this compared to our interface, you'll see that the order of events is, or the order  of, sorry, arguments is reversed. Our interface took the message first and then the successful  callback and then the error callback. Here they are reversed. The reason why we've done  this is this is something you'll see when you try to support multiple languages from  a library. In order to satisfy type inference and type resolution across the CLR, you will  occasionally have to invert the order of inputs so that you can have two different overloads  that are actually different and take different inputs but that the compiler can't tell which  one is which. So you have to rearrange the order of arguments between them so that the  compiler is smart enough to say, oh, you put the message last. You must mean this overload  versus you put the message first. You mean that overload. That's something that you'll  just, you'll run into as you're trying to support C sharp and F sharp and VB and all  these things. Now, another alternative is to have two different methods with the same  inputs. But then it's weird, like why would you call route versus, no, really route this.  They're both meant to be called route. That's the clear way to do it. So as a bit of a hacky  workaround, you do have to rearrange the order of arguments. And you'll see this all over  the BCL. For instance, if you're working with the copy function, the static copy method  on the array type, you'll see that there's four or five different overloads where they're  basically taking the same parameters, but they have to change the order of them so that  the compiler knows, oh, you mean to copy from a native pointer into the array versus copying  from the array into the native pointer. It's just a, it's a.NET thing. If you haven't  seen it before, trust me, someday you will. So we get the type of our incoming message.  We go and look it up in our catalog of handlers, say, hey, do we have a handler? And then we  fall into one of these different cases for whether or not we have a handler. If we have  either a command handler or an event handler, we spawn a worker and pass it the appropriate  message. This arrow here is a shortcut for posting a message to an agent. So if it's  command, we post it the run command message. If it's an event, we post it the run event  message, which if we go back a second, two slides, you'll see these are the messages  here that we're matching against to determine what to do. Where we passed run command, okay,  run a command. Where we passed run event, okay, run for the events. So that's what we  do in the case of actually having a command or a command handler or some event handlers.  In the case that we have no handlers or something went tragically wrong, all we do is post a  message to our supervisor. Because remember that supervisor we created on the previous  slide, that's an agent also. So in addition to monitoring the worker agents, we can directly  send it messages saying, hey, this happened, that happened. In the case of getting a message  that doesn't have a handler, we just want to log that, hey, somebody did something funny,  we'll look into it later. In the case that something went horribly wrong, we don't really  have much else to do except say, hey, something went horribly wrong, make a note of it and  make sure everything's okay. Questions? Yeah?  So that's our usual, that is a custom form.  Yes, this is actually a custom operator that I added, which I mean, it's relatively trivial.  It's just aliasing the post method. Just a way of cutting down the sort of parentheses,  helping things read a little bit cleaner. You'll find, I'm not a huge fan of custom operators,  but this one tends to crop up a lot in F sharp when working with mailbox processors or agents.  So it's reasonably simple, I think. Other questions?  All right. So that was the easy part of what the message router has to do. It just has  to look up, say, do I have a handler? If so, give it a message and go and life goes on.  This is the tricky part. This is where we do that look up to actually figure out if we  have a handler. So we start at this orange box here. We got a message, known type. Basically,  have I seen this type of message before? Not the actual instance of the message, the data  could be different, but have I seen the type of the message before? Yes. Okay. Do I have  actions? And here we mean basically handlers. Have I already looked up, loaded, compiled?  Do I have a handler call ready to go? If I do, great. I spawn a worker. If I don't, nope,  I log in error. And that's sort of what we saw on the other slide. And then this is the  interesting bit here on the left. I haven't seen this type before. Well, I checked to  see is it an event. If it is, do I have at least one handler for it? If I do, I build  an action, and again here an action is a function that makes it easy to cache and efficiently  call these handlers without having to recompile them, reload them over and over and over again.  And once I've built it, I add it to my catalog and then I run the worker. If I didn't have  at least one handler for this event, well, I can't do much else, so I log in error. Oh,  also if we fail to build the action for one reason or another, we log in error. The command  logic is very similar, except that instead of checking for at least one handler, we check  that our catalog contains exactly one handler. It would be an error if we initialized the  catalog with two different command handlers. So while this looks fairly complicated, the  good news is it is lazy and happens exactly once. It's lazy and memorized. So this complex,  well, I mean this bit on the left happens, you know, every time you send it a message.  This complex bit on the right happens the first time we see a message of a given type  and only if we see a message of a given type. So even if we have a handler that understands  divide by 27 as a command, if we never send it a divide by 27 command message, we'll never  bother to load up that handler. And like I said, we only do it once and then we cache  it and it's actually pretty efficient. This lookup over here is just a lookup into a concurrent  dictionary, so it's a very relatively speaking cheap lookup. It would be cheaper if we didn't  have to be thread safe, but we do. So what are some ways that we make reflection better?  I talked about using active patterns and code quotations to make reflective metaprogramming  look a bit better. And so here we'll start with we've defined a bunch of active patterns.  The first is a active pattern that will recursively walk through an expression tree looking for  particular shapes in the expression tree. And when I say expression tree, do people  know what I mean by that? Questions about that? Okay, so we've got code as a tree and  we're going to programmatically traverse it and we're going to say, is this a method  call? Great, then return some method. Is this a lambda function in which case keep walking  over the body of the lambda? Is this any other combination of expressions? Well, yeah, we're  going to walk through each sub-expression trying to keep track of the ones that return  some rather than none. And if we don't match any of these conditions, we've got something  we can't really process, so we return none. This sum and none is an option type. Is everyone  here familiar with the option type in F sharp? Questions about this? Basically, this is like  saying, you know, we didn't find anything that matched, but here we found something  that matched. And then we just call the function and it recursively walks over itself. We wrap  the whole thing in an active pattern because the way active patterns work, at least partial  active patterns. So there are different kinds of active patterns. We're only going to look  at partial active patterns here. There are total active patterns as well, which we won't  look at. Partial active patterns will return some or none. And that's how you tell the  compiler that this match succeeded or no, this match, this is not a match and you should  move on. So when we actually try to match against this, in the cases where we return  some, we'll get a method handle and where it's none, we'll actually just move on. Similarly,  we define a interface active pattern, which now this is what's known as a parameterized  pattern because in addition to the value we're matching against, we take an additional input  called target that lets us fine tune the behavior. And what we're doing here is we are making  sure that whatever suspect type we're matching against is a strict interface implementation  of the target type. And that has strict interface is not the best worded function. It's a helper  function because the actual logic for finding out if a type implements an interface gets  a little crazy in the CLR because you have to account for things like is this an interface?  Is this a generic interface? Is this an interface inheriting an interface? Is this a class implementing  interface? Is this a class inheriting a class that implements an interface? So things get  a little weird there. So we wrap up that into a little helper function. And basically if  we do have a match, then we return some and the suspect type, otherwise return none. The  important thing here to note is that, and it's sort of not obvious here, but we'll see  it later, the target type and the suspect type aren't going to be the same type. In  fact, if they were the same type, this call would return false because an interface is  not a strict implementation of itself. It just, it can't be. So if we return some target  rather than some suspect, it would change the return type of this match statement. And  then what we do is because active patterns, while they are cool and they extend pattern  matching, they're also functions. So we can treat them as first class functions. So one  of the cool things we do here is we partially apply the arguments to our interface function.  And now we have a much simpler active pattern that will check for I command instances or  I event instances without us having to repeat that boilerplate code over and over and over  again. So questions. Okay. Moving on. So now we actually will look at quotations. And quotations  look like regular F sharp code, but are set off by these little sort of, you know, angle  bracket at sign, at sign angle bracket. And what these tell the compiler is the expression  inside of it, don't execute that. Turn that into an expression tree. Turn that into a  data structure that I can programmatically manipulate. And we use them in a couple of  ways. Starting at the top, this get method for event, it takes in a message type. It  uses the message type to get a I handle event type that has the generic argument provided  for the generic parameter. Yes, it's very confusing. In the CLR, a generic parameter  is when you say like T, but a generic argument is when you give an actual type and move T.  Sort of not great wording, but that's what it's called. So we take our open generic and  we use the actual message type to say give me a type that is this generic with the generic  parameter filled in by this type argument. Once we have that, we create an expression  that attempts to invoke the handle function on a particular, on a particular type. And  then we pass that, we match that against our method active pattern that we defined on the  previous slide. And basically what this one's giving us is if in this expression tree, the  expression tree that results from this quotation, we can pull out the method name, which 99.987  percent of the time we can. We then have the actual target type and then we can actually  get the actual method from its method name. Because if you know anything about reflective  metaprogramming in the CLR, the way you invoke methods on a type is very stringy. You get  the string representation of the name and you use that to invoke the method on the type.  So we put a bit of strong typing around this by virtue of the code quotation and the active  pattern. And then obviously if we have nothing, we raise a quantum flux error because really  truly we should never, ever have this. This shouldn't be possible. This means something  has fundamentally changed in the internals of reflection in the CLR and we need to start  over. So that's what our quantum flux error gives us. This is just a type safe way of  getting things so that you don't have to deal with lots of, oh, did I get the string right?  Did I, did I fat finger the string? Did I remember to pass the argument to tell it whether  it's a case sensitive search or a case insensitive search for the method name? We would just get  rid of all that stuff. We were strongly typed in our strongly typed language. We do something,  I didn't show it here, but we do something very, very similar for commands. There's  a function called get method for command where the only difference is instead of I handle  event, we use the I handle command interface. So I didn't show that here, but you can imagine  same thing. Now this method to function is interesting. This is where we build that action.  Remember I said we create these action functions that we then cache so that we can efficiently  invoke the same handler over and over again with different message instances without having  to pay the price of recompiling. This is the function where we do that. So we take in a,  a resolver which is a generic interface that, or a simple interface that we provide in our  core library which lets us resolve a type, basically. It's, it's your basic, you know,  IOC dependency injection thing. And it's just an interface so you can use it to implement  on top of whatever mechanism you want. We also take in a method info which we will have  gotten from a call to get method for event. This actually returns a tuple of the target  interface type and, or sorry, the target type and the method info. So the method info part  that we return from this function will actually get passed into here. Actually both of these  arguments, the type and the method will be these two arguments here. What we do is we  first check to make sure that our resolver instance knows that that type exists. So that's  what this can resolve does. We then have an expression, or sorry, a quotation for accessing  that handler instance inside the resolver. And the reason why we get an expression here  is because we can take this expression tree and splice it into this other expression.  So what we've got here is two separate quotations that are, so basically two separate expression  trees that we are combining together into a larger tree that is the actual function  that we want to build. And so basically what we do is once we, sorry, once we've got the  tree for getting our handler, we go ahead and create a function that will take in a message  and a cancellation token and will actually invoke our method on our target, passing it  the inputs, and then we have to, you know, by default it's getting us an object. We want  to strongly type the result up to a task because remember ultimately this handler is going to  be I handle event dot handle or I handle command dot handle, both of which will return you  a task. So now we've got our two expressions or our larger expression with our smaller  expression sliced into it. We then compile it into an actual, it's actually a delegate.  It's not even a first class function. It's not an F sharp function. I'm sorry, it is  an F sharp function. We compile it into an F sharp function and then we wrap it in sum  because this could fail. So our method returns sum with our compiled action if we need it  and if we couldn't resolve, we return none saying, hey, guess what, we're not going to  build an action here. So this is probably the most complicated bit of the code. There's  not a lot going on in message router except for this. This is pretty gnarly stuff. But  again, what's great about this is the use of quotations has let me write strongly typed  F sharp in a way that is identical to if I wasn't doing reflective metaprogramming and  yet I'm still doing reflective metaprogramming. For folks who have worked with Lisp or other  languages that have strong macro support, it's somewhat similar conceptually. I won't  claim that there's any actual similarities because people who are into Lisp will get  all over me. But it's conceptually similar to this model of I'm writing what looks like  normal code but I'm not treating it like normal code. I'm treating it as expressions. And  the main reason we do this is to get strong type safety and to not deal with such clunky  reflection APIs. For instance, if you tried to do this using, so C sharp also supports  expression trees. It doesn't support quotations but it does support expression trees. If you  try to do this in C sharp or VB, what you would have is you would have this messy, messy  chain of method calls where you've got a lambda and inside the lambda you've got a call to  a let and inside the let you've got a call to a var and inside the var you've got a call  to an access. And like it just becomes very clunky and unreadable and not actually strongly  typed in the way we would like or in a way that's easy to reason about but F sharp quotations  let us do that. So questions about this?  Because the most performant and efficient way to do that is to build a delegate and hang  on to the delegate. Because ultimately what we have in memory is we have a concurrent  dictionary where the key is going to be our message type and the value is going to be  a delegate. We don't want to keep looking up the handler every time because that can  be expensive. We want to just say for a message type give me a delegate and so it turns out  just in the nature of.NET with reflection. Reflectively building a delegate and then  caching it is going to be a lot faster than reflectively looking up the handler every time.  And if we just held the reference to the handler directly, I suppose technically we could do  that but it would be a little bit more awkward to invoke because again it's a reflective  call so we'd have to invoke it in sort of a less strongly typed way. The delegate gives  us a nice strongly typed wrapper over that if that makes sense.  Yes, invoking it by reflection versus invoking the delegate there is over time especially  at volume there is an appreciable performance difference. At least on CLR version 4.5 on  Windows. Obviously benchmarks are going to vary a little bit. And then finally this is  where we use some of those things that we just saw. So you'll see here we use those  two methods that contain the quotations to get our interface, our strongly, our closed  generic interface. The actual handler itself or the actual, you know, the pointer to the  handler, the method info. And the actual, we passed that into here to build up the actual  action. Those are the two functions we just saw in the previous slide. We then, remember  we start with a list of handler types. These are the types we know about. The resolver  is just used to complete building the call. We basically choose any of the ones that implement  the interface. So now what we've gone from is matching against an open generic, sorry,  a marker interface that is tied to an open generic interface. We've gone and created  a concrete interface, a filled in instance of the interface if you will. And now we're  going through all our handlers and just finding the handlers that implement that interface.  And here we're doing this to build the event handlers. So we want all of the ones that  implement that interface. This build action here, this is a poor naming on my part. Method  to function actually takes in more arguments. We've gone ahead and partially applied it  to get this build action function, which will ultimately take in the, so if we back up one  slide, you'll see method to function takes in a resolver, an action and a handler type.  Here we've pre-applied the resolver because we're going to use the same resolver all the  time. We've pre-applied the handle, the action, you could say, the method info because we  got that from our previous call. And what that gives us is a function that we can then  pass an interface type to and it will build up our delegate for us. So we choose only  the types that match our strongly generically defined interface. We build up delegates  for them and then we wrap them into a container. This event handlers is just part of the union  that says we have event handlers. Here's the list of them. We have a command handler. We  have one of them. It's just a little wrapper to help identify what we've got in our list  here. For people not familiar with F-sharp, sequence is basically an Ionumerable. In fact,  not basically, it is an Ionumerable. So anywhere you see seek, just mentally insert Ionumerable  if that's more comfortable for you. We haven't talked about this much. This vertical bar  carrot sign, that's what's known as a pipe operator. This lets you take the trailing  argument of one function and put it at the front of the function. And what this lets  us do is this lets us build a nice linear pipeline. So this is conceptually very similar  to a fluent interface in like C-sharp or JavaScript. But in F-sharp, we do it with this nifty little  pipe operator because of a number of not important for this discussion details. So this is our  function that builds event handlers. The function to build command handlers looks very, very  similar except it calls get method for command instead of get method for event. And instead  of just choosing all of the ones that match and then mapping over all of them, it does  a little extra work to make sure we have exactly one that comes back. And then in extract handlers,  this is sort of our root level thing. We check the message type to say, remember these are  the other two active patterns we defined. Is this a command? Okay, build command handlers.  Is this an event? Okay, go build event handlers. And then we fall down the chain. If we get  a message that does not implement i command or i event, neither of our first two cases  will match. So we'll fall into our third case, which is we return an error saying, hey, this  message type is not a valid message type. Please try again. Questions? So there you  can see we're using the functions that have the quotations in them. Then we're using some  active patterns. So we got here a little faster than I thought we would. That's cool, though.  In conclusion, I want to just stress a few things. Designing from any languages does  require a bit more thought. It does require a bit more care, but it's totally worth it.  We had a tremendously positive reception, uptake, and usage from the teams that use  this library. And they know nothing about F-sharp. They don't know an action from an  actor from a function. They know their C-sharp. They know their VB. That's it. And they've  been very, very happy. So you can get that good experience as a library author, as a  library maintainer. You do have to do a bit more work, but it is absolutely worth it.  That's point one. Point two, and this is a good one to stress. It will be a recurring  theme if you hang out in room seven today. F-sharp builds simple solution, helps you  build simple solutions to complex problems. Not that routing messages is terribly a complex  problem, but managing concurrency, managing reflection, these things can be quite complex.  F-sharp gives us a lot of nice tools to help simplify all of this. And this last bullet  point is if you take nothing else away from this talk, this is the bullet point to take  away. This is real and happening in production now. This code is actually used at one of  the largest retail mortgage lenders in the United States on multiple projects and multiple  teams. So this is not a toy example. This is not a very niche example. This is not something  that's limited to a highly academic field. This is something that a large company is  put into production in the real world. So that, if you take nothing else away from this  talk, if everything I've said for the past 47 minutes is completely in one ear or not  the other, please hang on to this because someday someone will ask you, well, nobody  really uses F-sharp in production, do they? No, it turns out they do. And they don't  just do it at startups. They don't just do it for scientific research. They don't just  do it in green field projects. They do it as extensions to brown field projects. They  do it in places where they are not predominantly an F-sharp shop. They do it in polyglot scenarios.  This is real. If you want more information about F-sharp, F-sharp.org is the home site  for the F-sharp software foundation, which is a tremendous nonprofit that helps steer  the life and development of F-sharp as a language and as a community. I encourage you to join  if you're at all interested in F-sharp or at least visit the page. They have great information  for helping you get started. F-sharp for fun and profit is a terrific reference resource,  especially if you're an enterprise developer and you want to learn more about functional  programming and more about F-sharp. I highly recommend you go check that out. And the author  of that content is actually speaking later today. Say hi, Scott. TryF-sharp.net is a  great resource where if you're willing to deal with silver light being enabled, you  can actually play around with F-sharp 100% in your browser. So you don't even need to  know. There's a billion different ways to get F-sharp on your computer. F-sharp.org details  all of them. But if you're really, really like, I just don't want to install anything.  I don't want to have to think about anything. I don't want to download anything. Let me  just point my browser and play with some F-sharp. TryF-sharp lets you do that. And FS-SNP, which  is actually the brainchild of another speaker who's speaking later, I think, tomorrow. Or  no, actually later today. This is a great community-sourced collection of various F-sharp  code snippets. Categorized and tagged and with comments and you can see lots of different  things, get lots of ideas, find some different techniques. I highly recommend digging into  all of these. Again, I've been Paul Michael Belazzucci. I'm at PeopleLazzucci on Twitter.  I'm at PeopleLazzucci on github.com. If you want to find me on LinkedIn, I'm PeopleLazzucci  on LinkedIn. Pretty much anywhere you go on the web, I would be PeopleLazzucci, hopefully.  At this point, you guys have been awesome. We've got a few minutes. I'd like to open  up. Oh, before I open up to questions, I do have stickers. Come get F-sharp stickers because  as any F-sharp developer will tell you, you are not a real F-sharp developer unless you  take your corporate laptop, cover it with stickers in the hopes that you will piss off  the IT department when you hand it back in in a year. So plenty of free stickers. At  this point, I'd just like to open up to more questions. Yeah?  Yeah.  Absolutely. We have a...  Sure. Sure. The question was, can I talk through... We've talked through the library. We haven't  talked through an instance of using it. We haven't talked through an application that  actually uses it. In the source code itself, there's a very trivial application that uses  it in two domains. One domain would be doing basic arithmetic. The other domain would be  debits and credits to a checking account. But in real life, where we've actually used  it, and I will try to do this without violating my nondisclosure agreements and the intellectual  property agreements I had to sign, we have a system that takes in various pieces of supplemental  data about the market in real time. And it basically massages them and puts them into  a data store that are... So the group I work for supports the folks who buy and sell mortgages  in the financial markets to create the funding for them. So we have a system that takes in  all of this sort of supernumerary market data and massages it into a shape that our traders  can look at and puts it into a data store where they can run reports and get at the  information. And the way that works is we will have lots of external services that monitor  various external... Sorry. We have lots of services outside of the system that monitor  external data points when they get new data. They send a command into the system. The command  contains the new data. We run the command to store the data and then separately our traders  can go to a web page and issue queries against the system to see the latest data. Modulo  a second or two of latency because it is an eventually consistent system. We keep it running  pretty snappy but it's not real time. Additional questions? Awesome. You guys have been terrific.  Please come get stickers. Thank you.
The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_0_59.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_60_179.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_180_275.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_276_323.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_324_527.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_528_599.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_600_971.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_972_1079.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1080_1127.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1128_1271.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1272_1343.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1344_1427.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1428_1499.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1500_1571.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1572_1643.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1644_1703.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_1704_2075.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_2076_4199.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_4200_5279.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_5280_5495.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_5496_7331.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_7332_7391.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_7392_7439.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_7440_7487.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_7488_8303.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_8304_11699.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_11700_11771.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_11772_11819.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_11820_13079.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_13080_13679.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_13680_13739.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_13740_15827.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_15828_17903.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_17904_18635.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_18636_18683.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_18684_18731.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_18732_19127.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_19128_19175.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_19176_19223.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_19224_19283.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_19284_23627.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_23628_26603.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_26604_26771.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_26772_26891.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_26892_29531.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_29532_30203.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_30204_32075.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_32076_32399.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_32400_33779.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_33780_33839.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_33840_35699.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_35700_39419.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_39420_39527.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_39528_39791.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_39792_39875.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_39876_41867.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_41868_45683.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_45684_51647.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_51648_65219.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_65220_67367.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_67368_67511.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_67512_71195.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_71196_71243.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_71244_71399.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_71400_72191.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_72192_72959.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_72960_74351.png The_Beating_Heart_of_CQRS__or_Actor-Based_Message_Routing_in_F_Paul_Blasucci_HD_74352_80729.png