From HTTP to Kafka-based microservices
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Rząsa, Wojciech
HTTP or asynchronous communication in microservices? This question is frequently repeated and discussed. Obviously, HTTP-based communication is easier for developers and architects. Even if your developers have no prior experience with microservices, they will probably understand how to implement an HTTP service well. While asynchronous communication has a lot of advantages that allow to design and implement really robust microservices system, they also bring new challenges not so obvious for people who didn’t have a chance to work with such an environment before. In FLYR we mostly have HTTP-based Inter Process Communication (IPC) in our infrastructure. At some point, we realized that to provide the functionality required by our product we needed something more flexible and more… asynchronous. We designed and implemented a Python library to facilitate switching to asynchronous IPC, supporting one- or two-way or even single-request – multi-response communication. An important thing in the design process was to provide developers having HTTP experience a tool that would ease the process of switching to asynchronous communication. Consequently, to switch an HTTP server-side endpoint to asynchronous IPC is a straightforward task. We selected Kafka for our message broker, not surprisingly by comparing its performance reports with our very rough, but no less demanding performance requirements. But we also took care to hide the details of the broker logic from developers. Yes, we don’t use all Kafka capabilities, if you need e.g. Kafka Streams, you will have to use another solution. But we can decide what capabilities are used in our microservices and how we can make changes in the way our services communicate in a single place. There are also hooks, Kubernetes health checks and more with a lot of flexibility available out of the box. We plan to opensource our library. At the moment of writing it ‘opensourcing’ is still a work in progress and we didn’t have sufficient time to do it due to strict time constraints we have on delivering functionality to our customers, but we hope to be able to do it soon. In this talk I would like to describe how we solved particularly important problems, what solutions we developed, how we use them and what problems still need to be addressed by developers. In other words, I would like to describe you the journey we made from HTTP to Kafka-based microservices.
 Hi, nice to see you all here. How was the coffee? Great. Nice. Okay, so I'm a person that always wanted to be an IT guy, computer science guy and so on. So I was working for 15 years in academic world, doing science, teaching students and so on. In the meantime, I did my PhD, but always the practical things were most important for me, so even in doing some science, I was always trying to do it more practical, available for developers, so you can Google something about it also. So now I'm working for FLYR and doing microservices for them. I was always interested in distributed systems, so it's perfectly aligned with what I was doing before. I'm also one of the people that organized Ruby user group in my country, so please be friendly. Okay. Okay, what about FLYR? What do we do in FLYR? We do revenue management system for airlines. So actually, we take a lot of data from them, so we do big data, ETL pipelines and so on. Then we do machine learning on this data and we tell them what prices should they sell, the tickets to earn most. That's our goal. We have an office in San Francisco and we have office in Kraków, Poland, so you can work for FLYR from European time zone and have a work-life balance. Okay, so that's some things we use, not everything. And one more thing, my colleagues from Kraków told me that I must bring to you. We were recently on a Python conference, some of us were in Czech on Python conference, and we've met their very interesting guy, so we hired him. You know the guy in the middle of the photo? Who knows this guy? Kretek. Yeah, that's right. Kretek, in Polish it's Kretek. It's for the ones that didn't see it, it's a mole, and it's a character from cartoon in our childhood, very popular in our part of Europe. So we've met this guy, we took him on a conference, he was on a talk, on different talks. He even gave his own lightning talk. So if you are wondering if you should or not, he did it, you can do it. Okay, so he is actually an exceptional data digger as a mole, so that's why we hired him. But to the point, the history begins like a year ago, a little less than a year ago, when I started to work in Flyer, and by then all microservices in our product were communicating over HTTP, and Flyer felt quite comfortable with this solution, but they also have the feeling that for some applications it won't be the best solution, and that we will at some point probably have to switch, we already had some places where the services communicated over a Rabit MQ, but it was not perfectly implemented, and we knew it, so there was a feeling that we should find an asynchronous way of communication, and the thing that actually caused the decision was a new requirement required to our e-commerce use case, and the requirement can be described more or less like this. We had an UI interacting with the user, there was a backend for this UI, so there was some interaction, and at some point this interaction between the user and UI backend caused the UI backend issue a query to the service our team was maintaining, and to fulfill this request, to respond to this request, we had to, we knew that we will have to find out a number of requests to the other services, some of them, most of them external, so we knew it will be time consuming to get the responses, but we wanted to give the user anything to show, to have anything to show to the user whenever we have anything useful for him, and when we get anything better for him, so we will update whatever we are showing, so that was the use case, so we actually wanted to implement something like this, we get the original query, we found out the subqueries, and whenever we get the first response to a query, we send a partial response to the original query to show something to the user, and whenever we get another response to a subquery, we will send another partial response to update this information that is showed to the user, and so on. That was the use case we wanted to implement, so that's the first thing, second thing was related to performance, and I can't give you the precise numbers, but from what I can tell you was that when I looked at Rabit and Q, we wanted persistence in our communication, so when I looked at Rabit and Q at 5000 messages per second that can be handled when you turn on the persistence, it was definitely not enough, so the requirements were quite high, so that was the second important thing, so of course you can do it using HTTP, but we already felt that we will anyway need asynchronous communication, so that's the good point to start with, and how we did it, how we approached this situation, we decided yes, we need to do, we need to do a, well, but what and how. We have HTTP based infrastructure and it works, we have experience with it, we have developers experience with this way of communication, they have habits with implementing this way of communication, and you know competence is important, but old habits die hard, so it's the hardest thing to overcome in some situations, of course we knew we lacked experience with this kind of communication because we always did it using HTTP, so one more requirement, of course we must do it well and the first time do it well, well, hard to do it well first time, but maybe, and of course we knew we will get all these goodies when we switch to the asynchronous communication and even more, for example, more opportunities, anyone knows what's the first opportunity we get in this situation, anybody? What do you think? Sorry? Also? Sure, you can always refactor the code, but I have some planes for you, can you catch your plane? Thanks. Oh, sorry, but it's still perfectly operational. Anybody tries? It was probably on the previous slide. Okay, that's not along my line of thinking. Sure, yes, you do, you can have all these things, but also you have the perfect opportunity to make new mistakes. Isn't it true, completely new mistakes, completely new things can go wrong when you go to asynchronous communication. So we can have different concurrency issues, race conditions because we do asynchronous things, okay? And in the places we didn't have it before, there is a problem, you know, we should choose a broker. We have experience, we can read a lot, we can do research, but there is always a chance we will choose a wrong broker because we didn't research for the right thing. If we choose the right broker, there's probably more than one driver we can choose, okay? So which one should we choose? And on what basis? How should we decide? We can use the correct driver, but use it incorrectly, okay? If, you know, if it's just in your one simple service, that's basic, but if it spreads all over your system and then you realize, well, you have to do this and that to make this communication stable and now you have to find all these places, the other people just copied and pasted their incorrect code, that's hard to overcome. And finally, we can have correct driver and correct broker, but we can use the broker incorrectly. And a lot of different things may also go wrong. So we decided to contain all these horrible things in one place, okay? A library and call this library, I think, calls and there's another hard question. I have a cup for you. I won't be throwing it. I'll walk. I deliver it by, you know, not by plane. Why I think calls? Okay? Don't answer. Don't answer. Don't answer. I have a cup for you. No, we don't use as in IO below for some reasons. Okay, the reason is I just didn't want, I just wanted to give you the cup before you try to answer because the answer is so strange that you didn't have a chance, okay? As there's always this naming thing in computer science. Okay, so we wanted to create a library that meets our functional requirements. We wanted for developers, for this library to wherever possible, resemble what they already saw, what they knew. And of course we wanted this asynchronous communication below. So whenever it's possible, we wanted to join these three requirements and why the library, you know, for maintainers of a library or of this switch, you know, the Sauron, you know, the guy. One ring to roll them all. So, yes, one place to fix all the bugs. One place to change decisions. So it's much easier to change decisions. You don't have to trace it all over microservices implementation, just one place. And if we need to apply good patterns, it's not just that we teach all these people how to use this, like Kafka driver well, we just use it well in one place and we don't have to change what we thought to people because it's harder than just updating the code. But you know, Sauron had to sell these rings somehow to the people. So how do we sell it to developers? There must be something for them in this. Yeah, sure, and not no Kafka because it's hidden. But they do. The complexity will be hidden. If we do a good abstraction above it, so they won't have to think about all difficult things related to it. And lower entry barrier. That's a way different communication than using HTTP. So let's try to do it this way. So the decisions we had to take were easier because they no longer were final. We were more comfortable with the thought that we will have to change these decisions at some point. The decisions were that we will choose Kafka as message broker, not super singly for performance reasons, and for performance it offers just out of the box. And we will choose Confluent Kafka as a driver because the performance reasons. And we also hoped that something supported by Confluent will be really stable and well. So, well, that's what we use. We wanted to make it just a library, no framework approach, no put everything inside. Just a communication library make it simple. If we need something more complicated, maybe we will put another library above it. So that was first thing. We wanted to make it testable. First, okay, this Kafka is nice. How do I see you to this Kafka? Well, you don't. You can't issue HTTP to KafkaQ. So how do I test it? We need to give people a way to ad hoc send something to just let them test their service. And we wanted to make it testable automatically. So we wanted to provide some reasonable mocks for unit testing to make implementing tests easier. And if possible, maybe we could make it resemble flask just to easier, to let developers easier get used to the new approach. Okay, but it's like half of my time and I'm talking, talking, talking and it's developer conference so I probably, that's what you think. So yes, okay, I'm showing you the code. To use it. How do we use it? We create an object. Given that service name as a parameter, it's just an identifier of this service. It should be unique across your system. And when you have this object and you want to create a server endpoint, so an endpoint that will be asynchronously responding for some requests. So you just create a function and decorate it with async call server callback for. The parameter for the callback is the name of the endpoint. It resembles HTTP endpoint, but it doesn't have to. This slash is not necessary there. Just a convention. This function will get the request object as a parameter. You can do it the request whatever you wish. And you can use this request object to create a response, more than one response. But each of these responses can be sent back and they will be delivered to whoever sent the original request. So you can send zero or more responses to a request. Well, to create, to send a request, so to, oh, that's about identification. That's an ID of the service and address of the service. And this is name of the endpoint. So you send the request to the service ID and specific endpoint. You can have a lot of endpoints in a single service. To send a request, you use async call's client to create new message that will be sent. There's a destination ID in this message, target endpoint, and of course a payload. Maybe some more things. And you send this request. But wait, it's all asynchronous. The sending is asynchronous. It is not non-blocking. So how do we get a response for this request? Now before we send the request, we should define a callback to handle response we expect. So we define a function this time that the creator is async call's client, not async call's server. And we define callback for the service that will be sending responses to us. That's why that's the first name. And the endpoint that will be queried and will respond to our queries. So we can handle response this way. So that's just about how you can use it. Of course, in the most basic approach, of course, the last thing you should do is you should start listening. And in the server, actually, you can have client and server endpoints in a single service. So you can get some requests, send some responses to fulfill them, and then when you get a response, you can respond to the original request. It's all feasible. So you just call the async call's listening at the end of your program. It will make async call's receive messages, route them to correct callbacks. So what we have? We have server which is like HTTP server, even driven. The callbacks we know from Flask, for example. We also have client which is not like an HTTP client because it's not blocking. It's asynchronous, you send a request and just nothing happens. You should have a callback to handle a response. So if your request requires query from another service, you get the request, you send the query to another service, and your process is not blocked by waiting for a response. It can actually serve another request, waiting for the response for the previous request. A single process can be a server and a client, of course, and we can handle one request and a number of responses. So we can have more than one response to the original request, but we also can have no response to original request. We can just send notifications this way, and if the receiver expects just to handle the notifications, it's not a problem. You don't have to send a response. So okay, now when I showed you the basics, I could talk a long time about different things and the details that are inside, but I just want to tell about one single. For me, it's one of the most important things. How do you test it? Is there a way to easily test this as unit tests? So yes, I think calls have a testing mode. You need to enable testing mode, and you can use it in your unit tests. To enable testing mode, you just set the testing flag to true. Then you import your application, which defines all these callbacks. And finally, you must have a fixture that will reset the testing mode between the tests, because we need to clear some buffers. And when you have the testing mode enabled, you can start testing. So we can have different use cases that we want to test. The most basic is we have a server endpoint, and we want to check if it gives correct responses. So we want to send it a request and verify what responses we got and if it's correct. The most basic thing. So we need a way to send a test request to an arbitrary service. We can do it using that, because if you turn on the testing mode, you get something which is shown here. It's test client. And you can use this test client to create arbitrary messages and to send these arbitrary messages wherever you want in your unit tests. So that's how you send the request to a tested service. You send this request, and when you send it, the test client will receive the responses for you. So when you send the request, you immediately can check what responses were received. And then you can just assert for correctness of these responses. So you don't have to think about this Kafka below, about messages or anything. You just send a request according to business rules and verify the response according to business rules. That's all. That's the simplest thing to do. The more complicated is when we have another service. We are testing the service in the middle, a money broker. And this service is when it serves our original request, is expected to notify another service, a notification receiver, about some things. So we are testing the original, the money broker, but while testing it, we want to verify that correct notifications would be sent outside. Of course, we don't want to spin out all the infrastructure. We want to have it mocked in our unit tests. We don't want such a communication. So we need a way to mock this other service just to verify what it would receive if it worked, if it really was really started. So aside from test client, I think in testing mode gives you also a test server. So in this test server, we can, in this test server, we can register an endpoint for this service we want to mock. It was called notification receiver. There was a notify endpoint. And this test server will just receive these messages for us and will allow us to retrieve these messages and check if correct messages were retrieved. So we register it. Then we trigger the endpoint we want to test. We can do some assertions about responses as in the previous example. And we can use the test server received requests to obtain the requests that were received by the service. And the most complicated thing is when the third service is actually expected to respond to some queries, and these responses should be used to generate the response we are testing. So we want to mock all the service together with its responses to produce, to check that correct output will be produced. So here we need to mock the random service. And to do it, we just define a generator function. And when we register the endpoint in the test server, we just give it the fake responses generator and it will generate responses. And we will be able to verify that payload on the output are correct. It's correct. So we have testing tools out of the box. We discolve in testing mode, are made on stack, so they are deterministic. The tests are deterministic. We don't need message cube broker. We don't have to think about how this IPC is actually done below. Just think about the business level in testing. We have also many more features like before and receive, before send hooks, like endpoint context managers, if you want to measure how long performance of your endpoints so you can hook a context manager around this endpoint. We have error handlers for endpoints, Kubernetes health check because we run it on Kubernetes. We have sewer like client and a lot of more things. Of course, if we hide some complexity, we also hide some opportunities, not only to make errors. But so if you want Kafka streams, for example, so sorry, we won't be able to deliver it because we hid this specific of Kafka below the abstractions. Okay, so we still can have these problems, like concurrency issues we will have because we have asynchronous communication. We won't run away from it. But it's all for developers. It's all on the level of the business logic they are implementing. They don't have to think about Kafka usage pattern, driver usage pattern and so on. And if there are some problems below, that they can be solved in one place and actually were solved in one place without bothering a lot of developers. So switching from HTTP to async calls for server is straightforward. It's not a problem. It's a client, it's a little bit more complicated. We support one way communication. If we use more complex use cases, yes, it's a matter of doing it well. We have callbacks, so we always can have a callback hell. But we also know that there are patterns to do it well. So we can build something above it if we need. We have easily disabled services because we have tools to do it. And we have now a standard project-wide layer to make asynchronous communication between the services. Thank you. I have three more cups. I won't be throwing it. But please grab them and don't make me take them back by playing, you know, to my place. Any questions? We have time for a few questions. What do you maybe? Yeah, thank you for the talk. An interesting thing is how to design exceptions and errors. How did you approach that? We have, as I said, we have exception handlers. So you can register an exception handler for specific exceptions for all your service. So exception handler is a function that will be called when an exception in your endpoint occurs. Okay. For the HTTP developers, they don't see much of it, of the Kafka specifics. But there's like a layer, your library layer between throwing a specific errors. Exception handlers are for the exceptions that are going out of the callbacks. As for Kafka exceptions, well, they shouldn't see them because we should debug the library. If not, we must fix it. Yeah, like message length is fixed and stuff like that. You kind of have too much payload and stuff like that. You need to abstract that. Actually, you know, because in message given communication, the errors are handled in the different place rather on the level of receiver than the level of sender of the request. That's also the tricky thing to switch from HTTP. You don't have error 500 because your service is down. Your message is just waiting. When the server, the service is up after like 10 minutes, it will serve your stale request and maybe it will also send responses to you. Unfortunately, that's all the time we have, but maybe you can grab. I'm around. I'm around. So, questions after this? One big round of applause. Thank you very much.
 Hi, nice to see you all here.  How was the coffee?  Great.  Nice.  Okay, so I'm a person that always wanted to be an IT guy, computer science guy and so  on.  So I was working for 15 years in academic world, doing science, teaching students and so on.  In the meantime, I did my PhD, but always the practical things were most important for  me, so even in doing some science, I was always trying to do it more practical, available  for developers, so you can Google something about it also.  So now I'm working for FLYR and doing microservices for them.  I was always interested in distributed systems, so it's perfectly aligned with what I was  doing before.  I'm also one of the people that organized Ruby user group in my country, so please be  friendly.  Okay.  Okay, what about FLYR?  What do we do in FLYR?  We do revenue management system for airlines.  So actually, we take a lot of data from them, so we do big data, ETL pipelines and so on.  Then we do machine learning on this data and we tell them what prices should they sell,  the tickets to earn most.  That's our goal.  We have an office in San Francisco and we have office in Kraków, Poland, so you can  work for FLYR from European time zone and have a work-life balance.  Okay, so that's some things we use, not everything.  And one more thing, my colleagues from Kraków told me that I must bring to you.  We were recently on a Python conference, some of us were in Czech on Python conference,  and we've met their very interesting guy, so we hired him.  You know the guy in the middle of the photo?  Who knows this guy?  Kretek.  Yeah, that's right.  Kretek, in Polish it's Kretek.  It's for the ones that didn't see it, it's a mole, and it's a character from cartoon  in our childhood, very popular in our part of Europe.  So we've met this guy, we took him on a conference, he was on a talk, on different talks.  He even gave his own lightning talk.  So if you are wondering if you should or not, he did it, you can do it.  Okay, so he is actually an exceptional data digger as a mole, so that's why we hired him.  But to the point, the history begins like a year ago, a little less than a year ago,  when I started to work in Flyer, and by then all microservices in our product were communicating  over HTTP, and Flyer felt quite comfortable with this solution, but they also have the  feeling that for some applications it won't be the best solution, and that we will at  some point probably have to switch, we already had some places where the services communicated  over a Rabit MQ, but it was not perfectly implemented, and we knew it, so there was  a feeling that we should find an asynchronous way of communication, and the thing that actually  caused the decision was a new requirement required to our e-commerce use case, and the requirement  can be described more or less like this.  We had an UI interacting with the user, there was a backend for this UI, so there was some  interaction, and at some point this interaction between the user and UI backend caused the  UI backend issue a query to the service our team was maintaining, and to fulfill this  request, to respond to this request, we had to, we knew that we will have to find out  a number of requests to the other services, some of them, most of them external, so we  knew it will be time consuming to get the responses, but we wanted to give the user  anything to show, to have anything to show to the user whenever we have anything useful  for him, and when we get anything better for him, so we will update whatever we are showing,  so that was the use case, so we actually wanted to implement something like this, we get the  original query, we found out the subqueries, and whenever we get the first response to  a query, we send a partial response to the original query to show something to the user,  and whenever we get another response to a subquery, we will send another partial response  to update this information that is showed to the user, and so on.  That was the use case we wanted to implement, so that's the first thing, second thing was  related to performance, and I can't give you the precise numbers, but from what I can tell  you was that when I looked at Rabit and Q, we wanted persistence in our communication,  so when I looked at Rabit and Q at 5000 messages per second that can be handled when you turn  on the persistence, it was definitely not enough, so the requirements were quite high,  so that was the second important thing, so of course you can do it using HTTP, but we  already felt that we will anyway need asynchronous communication, so that's the good point to  start with, and how we did it, how we approached this situation, we decided yes, we need to  do, we need to do a, well, but what and how.  We have HTTP based infrastructure and it works, we have experience with it, we have  developers experience with this way of communication, they have habits with implementing this way  of communication, and you know competence is important, but old habits die hard, so it's  the hardest thing to overcome in some situations, of course we knew we lacked experience with  this kind of communication because we always did it using HTTP, so one more requirement,  of course we must do it well and the first time do it well, well, hard to do it well  first time, but maybe, and of course we knew we will get all these goodies when we switch  to the asynchronous communication and even more, for example, more opportunities, anyone  knows what's the first opportunity we get in this situation, anybody?  What do you think?  Sorry?  Also?  Sure, you can always refactor the code, but I have some planes for you, can you catch  your plane?  Thanks.  Oh, sorry, but it's still perfectly operational.  Anybody tries?  It was probably on the previous slide.  Okay, that's not along my line of thinking.  Sure, yes, you do, you can have all these things, but also you have the perfect opportunity  to make new mistakes.  Isn't it true, completely new mistakes, completely new things can go wrong when you go to asynchronous  communication.  So we can have different concurrency issues, race conditions because we do asynchronous  things, okay?  And in the places we didn't have it before, there is a problem, you know, we should choose  a broker.  We have experience, we can read a lot, we can do research, but there is always a chance  we will choose a wrong broker because we didn't research for the right thing.  If we choose the right broker, there's probably more than one driver we can choose, okay?  So which one should we choose?  And on what basis?  How should we decide?  We can use the correct driver, but use it incorrectly, okay?  If, you know, if it's just in your one simple service, that's basic, but if it spreads all  over your system and then you realize, well, you have to do this and that to make this  communication stable and now you have to find all these places, the other people just copied  and pasted their incorrect code, that's hard to overcome.  And finally, we can have correct driver and correct broker, but we can use the broker  incorrectly.  And a lot of different things may also go wrong.  So we decided to contain all these horrible things in one place, okay?  A library and call this library, I think, calls and there's another hard question.  I have a cup for you.  I won't be throwing it.  I'll walk.  I deliver it by, you know, not by plane.  Why I think calls?  Okay?  Don't answer.  Don't answer.  Don't answer.  I have a cup for you.  No, we don't use as in IO below for some reasons.  Okay, the reason is I just didn't want, I just wanted to give you the cup before you  try to answer because the answer is so strange that you didn't have a chance, okay?  As there's always this naming thing in computer science.  Okay, so we wanted to create a library that meets our functional requirements.  We wanted for developers, for this library to wherever possible, resemble what they already  saw, what they knew.  And of course we wanted this asynchronous communication below.  So whenever it's possible, we wanted to join these three requirements and why the library,  you know, for maintainers of a library or of this switch, you know, the Sauron, you  know, the guy.  One ring to roll them all.  So, yes, one place to fix all the bugs.  One place to change decisions.  So it's much easier to change decisions.  You don't have to trace it all over microservices implementation, just one place.  And if we need to apply good patterns, it's not just that we teach all these people how  to use this, like Kafka driver well, we just use it well in one place and we don't have  to change what we thought to people because it's harder than just updating the code.  But you know, Sauron had to sell these rings somehow to the people.  So how do we sell it to developers?  There must be something for them in this.  Yeah, sure, and not no Kafka because it's hidden.  But they do.  The complexity will be hidden.  If we do a good abstraction above it, so they won't have to think about all difficult  things related to it.  And lower entry barrier.  That's a way different communication than using HTTP.  So let's try to do it this way.  So the decisions we had to take were easier because they no longer were final.  We were more comfortable with the thought that we will have to change these decisions  at some point.  The decisions were that we will choose Kafka as message broker, not super singly for performance  reasons, and for performance it offers just out of the box.  And we will choose Confluent Kafka as a driver because the performance reasons.  And we also hoped that something supported by Confluent will be really stable and well.  So, well, that's what we use.  We wanted to make it just a library, no framework approach, no put everything inside.  Just a communication library make it simple.  If we need something more complicated, maybe we will put another library above it.  So that was first thing.  We wanted to make it testable.  First, okay, this Kafka is nice.  How do I see you to this Kafka?  Well, you don't.  You can't issue HTTP to KafkaQ.  So how do I test it?  We need to give people a way to ad hoc send something to just let them test their service.  And we wanted to make it testable automatically.  So we wanted to provide some reasonable mocks for unit testing to make implementing tests  easier.  And if possible, maybe we could make it resemble flask just to easier, to let developers easier  get used to the new approach.  Okay, but it's like half of my time and I'm talking, talking, talking and it's developer  conference so I probably, that's what you think.  So yes, okay, I'm showing you the code.  To use it.  How do we use it?  We create an object.  Given that service name as a parameter, it's just an identifier of this service.  It should be unique across your system.  And when you have this object and you want to create a server endpoint, so an endpoint  that will be asynchronously responding for some requests.  So you just create a function and decorate it with async call server callback for.  The parameter for the callback is the name of the endpoint.  It resembles HTTP endpoint, but it doesn't have to.  This slash is not necessary there.  Just a convention.  This function will get the request object as a parameter.  You can do it the request whatever you wish.  And you can use this request object to create a response, more than one response.  But each of these responses can be sent back and they will be delivered to whoever sent  the original request.  So you can send zero or more responses to a request.  Well, to create, to send a request, so to, oh, that's about identification.  That's an ID of the service and address of the service.  And this is name of the endpoint.  So you send the request to the service ID and specific endpoint.  You can have a lot of endpoints in a single service.  To send a request, you use async call's client to create new message that will be sent.  There's a destination ID in this message, target endpoint, and of course a payload.  Maybe some more things.  And you send this request.  But wait, it's all asynchronous.  The sending is asynchronous.  It is not non-blocking.  So how do we get a response for this request?  Now before we send the request, we should define a callback to handle response we expect.  So we define a function this time that the creator is async call's client, not async  call's server.  And we define callback for the service that will be sending responses to us.  That's why that's the first name.  And the endpoint that will be queried and will respond to our queries.  So we can handle response this way.  So that's just about how you can use it.  Of course, in the most basic approach, of course, the last thing you should do is you  should start listening.  And in the server, actually, you can have client and server endpoints in a single service.  So you can get some requests, send some responses to fulfill them, and then when you get a response,  you can respond to the original request.  It's all feasible.  So you just call the async call's listening at the end of your program.  It will make async call's receive messages, route them to correct callbacks.  So what we have?  We have server which is like HTTP server, even driven.  The callbacks we know from Flask, for example.  We also have client which is not like an HTTP client because it's not blocking.  It's asynchronous, you send a request and just nothing happens.  You should have a callback to handle a response.  So if your request requires query from another service, you get the request, you send the  query to another service, and your process is not blocked by waiting for a response.  It can actually serve another request, waiting for the response for the previous request.  A single process can be a server and a client, of course, and we can handle one request and  a number of responses.  So we can have more than one response to the original request, but we also can have no  response to original request.  We can just send notifications this way, and if the receiver expects just to handle the  notifications, it's not a problem.  You don't have to send a response.  So okay, now when I showed you the basics, I could talk a long time about different things  and the details that are inside, but I just want to tell about one single.  For me, it's one of the most important things.  How do you test it?  Is there a way to easily test this as unit tests?  So yes, I think calls have a testing mode.  You need to enable testing mode, and you can use it in your unit tests.  To enable testing mode, you just set the testing flag to true.  Then you import your application, which defines all these callbacks.  And finally, you must have a fixture that will reset the testing mode between the tests,  because we need to clear some buffers.  And when you have the testing mode enabled, you can start testing.  So we can have different use cases that we want to test.  The most basic is we have a server endpoint, and we want to check if it gives correct responses.  So we want to send it a request and verify what responses we got and if it's correct.  The most basic thing.  So we need a way to send a test request to an arbitrary service.  We can do it using that, because if you turn on the testing mode, you get something which  is shown here.  It's test client.  And you can use this test client to create arbitrary messages and to send these arbitrary  messages wherever you want in your unit tests.  So that's how you send the request to a tested service.  You send this request, and when you send it, the test client will receive the responses  for you.  So when you send the request, you immediately can check what responses were received.  And then you can just assert for correctness of these responses.  So you don't have to think about this Kafka below, about messages or anything.  You just send a request according to business rules and verify the response according to  business rules.  That's all.  That's the simplest thing to do.  The more complicated is when we have another service.  We are testing the service in the middle, a money broker.  And this service is when it serves our original request, is expected to notify another service,  a notification receiver, about some things.  So we are testing the original, the money broker, but while testing it, we want to verify  that correct notifications would be sent outside.  Of course, we don't want to spin out all the infrastructure.  We want to have it mocked in our unit tests.  We don't want such a communication.  So we need a way to mock this other service just to verify what it would receive if it  worked, if it really was really started.  So aside from test client, I think in testing mode gives you also a test server.  So in this test server, we can, in this test server, we can register an endpoint for this  service we want to mock.  It was called notification receiver.  There was a notify endpoint.  And this test server will just receive these messages for us and will allow us to retrieve  these messages and check if correct messages were retrieved.  So we register it.  Then we trigger the endpoint we want to test.  We can do some assertions about responses as in the previous example.  And we can use the test server received requests to obtain the requests that were received  by the service.  And the most complicated thing is when the third service is actually expected to respond  to some queries, and these responses should be used to generate the response we are testing.  So we want to mock all the service together with its responses to produce, to check that  correct output will be produced.  So here we need to mock the random service.  And to do it, we just define a generator function.  And when we register the endpoint in the test server, we just give it the fake responses  generator and it will generate responses.  And we will be able to verify that payload on the output are correct.  It's correct.  So we have testing tools out of the box.  We discolve in testing mode, are made on stack, so they are deterministic.  The tests are deterministic.  We don't need message cube broker.  We don't have to think about how this IPC is actually done below.  Just think about the business level in testing.  We have also many more features like before and receive, before send hooks, like endpoint  context managers, if you want to measure how long performance of your endpoints so you  can hook a context manager around this endpoint.  We have error handlers for endpoints, Kubernetes health check because we run it on Kubernetes.  We have sewer like client and a lot of more things.  Of course, if we hide some complexity, we also hide some opportunities, not only to  make errors.  But so if you want Kafka streams, for example, so sorry, we won't be able to deliver it because  we hid this specific of Kafka below the abstractions.  Okay, so we still can have these problems, like concurrency issues we will have because  we have asynchronous communication.  We won't run away from it.  But it's all for developers.  It's all on the level of the business logic they are implementing.  They don't have to think about Kafka usage pattern, driver usage pattern and so on.  And if there are some problems below, that they can be solved in one place and actually  were solved in one place without bothering a lot of developers.  So switching from HTTP to async calls for server is straightforward.  It's not a problem.  It's a client, it's a little bit more complicated.  We support one way communication.  If we use more complex use cases, yes, it's a matter of doing it well.  We have callbacks, so we always can have a callback hell.  But we also know that there are patterns to do it well.  So we can build something above it if we need.  We have easily disabled services because we have tools to do it.  And we have now a standard project-wide layer to make asynchronous communication between  the services.  Thank you.  I have three more cups.  I won't be throwing it.  But please grab them and don't make me take them back by playing, you know, to my place.  Any questions?  We have time for a few questions.  What do you maybe?  Yeah, thank you for the talk.  An interesting thing is how to design exceptions and errors.  How did you approach that?  We have, as I said, we have exception handlers.  So you can register an exception handler for specific exceptions for all your service.  So exception handler is a function that will be called when an exception in your endpoint  occurs.  Okay.  For the HTTP developers, they don't see much of it, of the Kafka specifics.  But there's like a layer, your library layer between throwing a specific errors.  Exception handlers are for the exceptions that are going out of the callbacks.  As for Kafka exceptions, well, they shouldn't see them because we should debug the library.  If not, we must fix it.  Yeah, like message length is fixed and stuff like that.  You kind of have too much payload and stuff like that.  You need to abstract that.  Actually, you know, because in message given communication, the errors are handled in the  different place rather on the level of receiver than the level of sender of the request.  That's also the tricky thing to switch from HTTP.  You don't have error 500 because your service is down.  Your message is just waiting.  When the server, the service is up after like 10 minutes, it will serve your stale request  and maybe it will also send responses to you.  Unfortunately, that's all the time we have, but maybe you can grab.  I'm around.  I'm around.  So, questions after this?  One big round of applause.  Thank you very much.
Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_0_119.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_120_239.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_240_1727.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_1728_3431.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_3432_4499.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_4500_6191.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_6192_8687.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_8688_9851.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_9852_10055.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_10056_11111.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_11112_11339.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_11340_12935.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_12936_15299.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_15300_15563.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_15564_17183.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_17184_17855.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_17856_18191.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_18192_20471.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_20472_21527.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_21528_23159.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_23160_23363.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_23364_23411.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_23412_25859.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_25860_27479.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_27480_28439.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_28440_29903.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_29904_30551.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_30552_31175.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_31176_31931.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_31932_34259.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_34260_35483.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_35484_36911.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_36912_37535.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_37536_38063.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_38064_38639.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_38640_39371.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_39372_39995.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_39996_40763.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_40764_42299.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42300_42371.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42372_42419.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42420_42467.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42468_42515.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42516_42563.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_42564_45275.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_45276_45395.png Wojciech_Rzasa_-_From_HTTP_to_Kafka-based_microservices_YsQ7Ps0qH7k_45396_45428.png