Kafka on Kubernetes
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Zelenin, Anatoly
In this talk we provide a short introduction to Apache Kafka and walk you through the steps to deploy Apache Kafka with Strimzi Kafka Operator on Kubernetes and show you how you can manage it using native Kubernetes tools. Apache Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, and can be used for Stream Processing, as a Storage or Messaging System and more. Running and operating Stateful apps on Kubernetes is not easy, at least if you’re going to deal with replication and have to take care of syncing and re-balancing your streaming data on different nodes and / or different clusters in different regions. Kubernetes is about Resiliency and Scale, Kafka too! Kafka is Stateful, Kubernetes' support for Statefulsets has reached a mature state!
 So, hi, I'm Anatoly and you joined me today to explore the world of Kafka on Kubernetes. But first start with who am I? I'm adventurer and a computer scientist and so the obvious job choice for me is to be an IT adventurer. So, currently I'm giving Kafka training and also doing some Kafka and Kubernetes consulting as a freelancer. But today we want to talk about Kafka and Kubernetes. Who of you heard about something about Kubernetes? Who has Kubernetes running in production? Okay, who of you knows something about Kafka? Quite a bit, good. A few runs Kafka in production. Wow. Four people. More than expected. But then, nevertheless, let's first talk about what is Kubernetes. I think most of you know what it is, but let's start with very, very rough and short overview. So for me, Kubernetes is a declarative container orchestration tool. I'm a computer scientist and so the most interesting thing for me is the declarative part. But let's first look at the words. What is a container? I think who does not know what a container is? Good. Okay. So, all of you know that guy. And for us it's enough if we say that a container is a mini VM with one application per container. So we can have Postgres running in a container, GitLab, or Nginx. So far so good. So I think you all know that. So what is a container orchestration tool then? So the question is basically, okay, I have a runtime and I have a container. And how can I put the container in the runtime? For one runtime and for one container, that's easy. You just let the container run on the runtime you have. What if you have one runtime and N containers? Again, containers are very, very tiny VMs and it's not a problem to run multiple containers on one runtime. Now the question arises, what if we have N runtimes and M containers? How can we spread the containers across the runtimes given some heuristics or the way we want to have it? So maybe this could be a way to do that. But I assure that this one is the best way. Maybe there's a better one. Or what do you do if the service wants to talk to each other? What do you do if you want to have encrypted communication between the containers? And what do you do when some of the failed rails is on fire or somebody's in your network or the network is again broken on AWS? So maybe you should think about network, security, FortoLand and so on. And this is something as most of you know Kubernetes helps us a lot with that. And finally, the last word in this phrase is probably declarative. So declarative for me is I describe how the world should be like and how I want to have it. So I declare which applications should run, which names do the services have, which security policies I want. And then Kubernetes does all the plumbing and fulfills my wishes. I like to say the computer, please, I want to have that word and the computer will do the rest for me. I'm lazy guy, so I don't want to do it myself. So I think that's enough for Kafka, at least of course this is not a deep dive obviously. So Kubernetes not Kafka. And let's talk about Kafka again. So for me again, I'm a theoretical computer scientist. For me Kafka is a distributed log. What is a distributed log? Let's talk about log first. So a log is an append only list. We can have here six elements for example, a few characters and a white space and we can add more data to the end of the list. And what can we do with that thing, with that list, with that log? We can basically, we can append data and we can read the full log starting at a specific position. So if we talk about appending data, that's pretty easy. We have a function, let's call it push. And let's say it takes a list of As, so As some type, it doesn't matter which one. And it takes another element of the same type and gives us back a new list containing of the types. So for us it's just appending new elements. For example, we just appended a Y. And if we append elements, we get a new position or we call it offset. So the list gets longer and we can address the elements in the list by the offsets. So we can add another element and so on and so on. And then we can read the full log starting at a specific position. So we have a function which is called read from offset. It takes again a list of As. We don't care what is A and an integer or some number. And then it will return us the log starting at that position. For example, if we say that our current offset is zero and we call that function with that list and zero, we will get back the whole log. So let's append a new element to the log. And we have to remember somewhere this current offset and then call the function again with a new offset. And then we will get only a U back, only one element, or a list containing the U. So what does that mean for us? In a log, we don't have any deletion. We don't have any random access. So we could tweak somehow the random access, but it's not intended to be used as random access. And there's no magic in it. So we can describe a log just by the data structure and the two functions. That it. No magic. The magic comes when we move away from the theoretical part and go to the practice. And if you're interested in more things about logs, there's a good article from LinkedIn about the log and what every software engineer should know about logs. So distributed. Kafka distributes this log. And how does Kafka do this distributed thing? What is rule number one of distributed systems? Any ideas? Down. But still we need somehow. We have a lot of distributed systems right now. Maybe you have heard of this thing called microservices, distributed systems. I think in every IT or computer science studies, there will be a lecture on distributed systems. So distributed systems. No, we don't want to do that. But still we need to. One of the two reasons are for them we have two match data for one computer and computers like to crash. I hope that my one took a rush today. So if we have too much data, I was okay. Bam. So how can we solve that? So Kafka approaches for too much data. Kafka is using partitioning and for computer crashes, Kafka is using replication to come up with that. So again, we are not trying to build better and better hardware which don't crash. But we say, okay, we cope with that. This is a problem we have. Theoretically, we are not able to solve that. How can we cope with that? So what is partitioning? So a Kafka cluster, logically, with just a set of computers. And we store multiple topics on the cluster. Also on a database, you also have not only one table, but database cluster or database instance, mostly multiple. And then each topic, a topic is not a log. A topic is basically a set of logs. And we are partitioning each topic, for example, that way. So topic A has three partitions. And if we read them as the partition one of topic A, SEA14, what does it mean? Maybe it does mean something, maybe not. But we just partitioned our data. So for example, the input for that was some data 12345 and a fluid. So the partitioning was just one problem. We put the S in the first partition, the O in the second partition, the M in the third partition, and then started again. Okay. That's partitioning. So how do we do that? Basically, we have a producer which writes something and the producer itself decides to which partition to write. That's important. This is in a traditional message queue, the producer writes to something and then the system itself says where to put it. And in Kafka, you have to decide yourself where to put it. There are different strategies. How do you can do that? You can just write all data to one partition, but then probably bad things will happen. You can randomly assign data to partitions and then multiple other ways. We will have some time in the demo to explore that. And when we have multiple producers, the picture stays simple but a bit chaotic. Every producer just writes to the partitions that need to write to. So reading, writing is quite straightforward. Reading is a bit more complicated. So if we have just one consumer and we want to guarantee that each message is read once and we would like to have it also once and only once to read, then we send all data or the consumer fetches all data from all partitions. It's nice, but what if your consumer crashes or if the one consumer is not fast enough, then you can just add more consumers. And then Kafka will spread the partitions across the computers. Or not Kafka is doing that, but the libraries, the consumer libraries will do that for you. Okay, we have two consumers and three partitions. And we send the first two partitions to the consumer one and the third partition to consumer two. What do we do when we have three consumers? We assign just one partition to each consumer and we are happy. Could we scale further? Yes, no? Does it make sense to have a fourth consumer? If we have a fourth consumer, then the fourth consumer won't get any data because we cannot split this further. If we want to guarantee that each message is read by exactly one consumer. But still this can be reasonable. For example, if one consumer crashes, then we will have another consumer which can take over the work and continue. Especially if you're running your favorite Java virtual machine and it needs some time to warm up, it may be very useful to have a warm standby replica. And it's recommended in many cases to have that. Okay, that's about partitioning. What about replication? So we talked already about clusters. If we have a cluster, we have multiple brokers. In that example, we have three brokers. Now if you remember, we have a topic with multiple partitions. And we didn't put all partitions on one broker because why should we have multiple brokers? But we have just, we put partition one, for example, on broker one, partition two on broker two, partition three on broker three and so forth. And partition four, then again on broker one because we don't have a fourth broker to put it on. Okay, nice about replication. We have just read our partitions across the brokers. But if broker one, for example, fails, we use partition one and partition four. So how can we cope with that? We replicate data. So we put, for example, partition one, another replica to broker two, partition two to broker three and so forth. And if we lose broker one, partition one is still available. It's on partition, on broker two, partition four is also on broker two and partition three is on broker three. So we will survive the crash of a broker. Okay, let's say this is a very short introduction to Kafka. So what about Kafka on Kubernetes? Why would we want to do that? Is it because of the password bingo you have heard? Or is it the so-called conference-driven development? Or is there something behind that that may be useful or make sense? So we have heard a bit about what is Kubernetes, what is Kafka, what could be that Kafka on Kubernetes? So for me, I see chances to have that. So what I like about Kubernetes is this declarative thing. I can just say I want to have an application running, an engine X for example. I want to have three replicas of my application. And if they don't have any state, everything is fine. I like about Kubernetes that it simplifies automation quite a lot because it can help us with failover plans and so forth. And if all your applications are already running on Kubernetes, then why not put everything on Kubernetes? Maybe it makes sense, maybe not, but let's see. The challenges about Kubernetes, especially Kafka on Kubernetes is that Kafka itself is very stateful. That's obvious. Kafka is a kind of a database. And Kubernetes at least till the last few versions was not very happy about stateful applications and it was quite challenging to deploy stateful applications in the character of Kubernetes. Kafka itself, if you give Kafka RAM or memory, it will take it all if it can. So the recommendation for a small Kafka cluster are, yeah, have a free machine with 30 gig RAM each for a small cluster for the beginning. And then if you need more, then get more of it. It also dislikes sharing CPU and so on. And Kubernetes is all about resource sharing. So you just often you try to put as much containers on one machine as possible to save compute costs in your cloud environment. And most importantly, Kafka itself requires very careful consideration where to deploy it. Kafka is very fast, but this comes at a cost. The cost is you should be careful and you should know what you do. Kafka itself does not commit data to disk. It waits until the operating system sometimes decides to commit data to disk. So if you have running multiple Kafka workers on one machine and that machine dies, you probably will lose data. You shouldn't put two Kafka workers on one machine. On one physical machine, maybe you have two virtual machines and they are all on the same physical machines. You will have issues. You shouldn't put Kafka clusters in one rack like that. If your top-mount switch dies and you lose electricity or you lose network connection, bad things will happen. So you should be very careful where to deploy Kafka. It doesn't matter if it's on bare metal virtual VMs, virtual machines or Kubernetes. You should be always very careful. And Kubernetes can have us with that, but we should be just aware. So I would say no matter what the infrastructure is below or what platform you use, Kafka requires a very good infrastructure and a team that can handle it. If you lack one of that, you should not use a do Kafka yourself. So my wish list for Kafka on Kubernetes is we want to have stateful sets for Kafka and Zookeepers. Stateful sets are one of the ways how to manage stateful applications on Kubernetes. You really want fast storage and you really want fast network. If you don't have fast storage or don't have fast network, you will get troubles or at least Kafka will be just very slow. So Kafka is very happy if it has fast storage and you can get a good performance out of it, but if your storage sucks, Kafka will suck for you too. And for Kafka and Kubernetes, I also wish to have services for finding clusters and brokers. And then of course, nice to have would be some way to have declarative resources for everything. So if you have multiple brokers, I don't want to create to define all of them. I want to say, hey, I would like to have a Kafka cluster with three brokers, four brokers, five brokers, how many I want. Who of you likes configuring SSL keys? Me not. So I would like to have to say, I would like to have TLS encryption between all my services. Do it. I don't want to do that. I want to declaratively define which users I want to have or which authentication method I want to use and which users are authorized to do what. And finally, we have multiple topics in Kafka. And it would be great to have a declarative way to define them because if I do Kafka create new topic and then I create another one and then I create another one, I would maybe lose some time, my overview. And you know what happens if, yeah, we have this topic. I don't remember what the name was and who is using that, but somewhere we have the information about that. So declarative way for all of that would be great to have. It's about infrastructure as a code, the bus water here. And Kubernetes provides us more or less for free. So why shouldn't we do that? So these are things that Kubernetes operators providing us. So what is a Kubernetes operator? So Kubernetes operator is very similar to a Kubernetes controller. And the controller is the thing that looks what you have defined and what is the current state of the Kafka Kubernetes cluster and makes things happen such that the Kubernetes cluster becomes the work we would like to have. And same for operator. I'm sometimes wondering why they have called it operators, why not just Kubernetes controllers, but that's another topic. And the basic idea about a Kafka Kubernetes operator is that it codifies human knowledge. The knowledge of an operator, that's why the name. And most often, especially for Kafka, the operators come with their own custom resources. So you can extend Kubernetes to have custom resources where you can say, I would like to have a Kafka cluster. I would like to have a Postgres database with that configuration. Please do that for me. I don't want to do it myself. And at the counter, there are three Kafka operators out there. One is Strimsy. Strimsy is deployed by Redhead and is the oldest of these three. A recent very interesting operator is the Banzai Cloud Kafka operator. It's relatively new. It just was introduced a few weeks ago. And so if you start a new project, you should definitely look at the Banzai Cloud Kafka operator. It's very clever. They do. And if you have a lot of money and want to spend it and want real good enterprise support, you should probably use. You will probably use a Confluent operator if you have Confluent enterprise licenses. If you don't use their other services. And we will talk today about Strimsy. So Strimsy you can find on the sounders. Okay. So Strimsy is developed by Redhead and Strimsy is the open source version of that operator. And if you want to give Redhead a lot of money, then you should buy AMQ streams, which is the same product, but you get enterprise support for it. And Strimsy consists basically of three operators. One is the cluster operator, which manages all the cluster things, the topic operator and the user operator. I didn't tell you the whole story. Kafka does not consist only of Kafka, but it has also a so-called zookeeper. Who knows what zookeeper is? Some of you. Okay. Zookeeper is basically a helper tool for Kafka to do cluster coordination and it stores some data data. Luckily, it will go away soon or later because nobody likes zookeeper in the Kafka community. Zookeeper limits your performance, zookeeper limits the amount of topics you can have and so on and so on. So they want to get rid of it. It will happen soon, hopefully. So the cluster operator just does the provisioning for the zookeeper ensemble. That's the provisioning for the Kafka cluster and also for the topic operator and user operator. Zookeeper itself comes without any transport encryption. Not nice. But you can use a TLS proxy and the cluster operator configures everything fast. This way zookeeper is talking to the TLS proxy and we have encrypted traffic in our network. I like that. And you also do global configuration for Kafka and zookeeper with the cluster operator. The topic operator, the name says, does the topic management. We can create new topics. We can delete topics. We can adapt the configuration the way we need. And the interesting thing is that it has also two ways. What does it mean? So if we create a Kubernetes resource for a topic, the topic operator will create a cluster Kafka topic, but if we have application that wants to create its own topics in Kafka directly, then the topic operator will create a Kubernetes resource. This way we will have always an up-to-date state of the topics in our Kubernetes. We don't need to think about I created the topic in Kafka or Kubernetes. We will have just a consistent way. And similarly, the user operator does the user management, it does the authentication authorization, and it stores the keys and passwords in Kubernetes secrets. This is not the most recommended way because Kubernetes secrets are not really secret. They are stored in plain text. So you should think about a key management system, but if you have a good Kubernetes cluster, you already have probably a key management system for that. And then you just say, hey, Strimsie, please use that key management system and it works. So that's it for that part. Time for a demo. So let's see if that works. No, that one. Okay, so I have here a Kubernetes cluster. It's on Google Cloud Engine. You can't see anything right now, but give me a second. So you can say QBZTL.namespaces. So Kubernetes consists of namespaces where you can just logically group stuff in it, and it's relatively empty. So if you create a new Kubernetes plus on Google Cloud platform, you will get that. So we want to create our own namespace. I will just say QBZTL create an S Kafka and switch to that one. Okay, it says we are now in the namespace Kafka. Let's install. First, we have to install Strimsie and one of the easiest way to do that is using Helm. So you just say Helm install, then you give a name and say I want to install this Strimsie Kafka operator. Take some time. And now it says yes, we are ready. And then we can do a QBZTL getProt. And we see that we have one on pot that is running and the readiness state is 0.1. So we have probably to wait a second. And I like I have a command which is just, who knows about the watch comment on Linux. Awesome tool. So watch is doing basically you say watch, please execute the following command every second and please show me a graphical diff for that. And then you will see, oh, look, we just execute QBZTL getProt every second and we see here the diff, we see that the age of the container changes every second. Awesome. Okay, now we have the cluster operator running. What should we do next? Let's create a Kafka cluster. And for that, we just use, oops, that's the fun. One one, we are using, so you are configuring Kubernetes using YAML files and there's Strimsie installs and own custom resource definition. And this way we can say, yeah, I would like to have something in the API version Kafka.simsie.ios like B1, Beta 1. So we may think that Simsie is still better. And please do something with this kind Kafka. Then we give it a name and then we specify. So I would like to have a Kafka in version 2.3.0. I would like to have three replicas. I would like to have a listener, a plain text listener. We could also say, please do a TLS version, but for now it's just a demo environment. It doesn't matter for us. Storage, I would like to have a FMRL storage. You should not do that in production, obviously. FMRL storage means that there's no persistent storage and if we kill the pot, the storage is lost. That's okay for our environment here. I would like to have a ZuckiPir ensemble with three replicas and again, FMRL storage. And then I say, yes, I would like to have the topic operator and the user operator. Then I just do a cube CTL, apply minus F Kafka. And then we can have a look at our watch. What will happen? Something changed. So we got three ZuckiPirs. They are fast today. They are already running. But we see the readiness date is zero of two. So we just wait a few seconds when they are ready. Oh, they are already one of two. That's nice. Two of two. And then we have a ZuckiPir ensemble running. And you see what the, now the Kubernetes cluster operator looks. Oh, ZuckiPirs up. Now I can start Kafka because it doesn't make any sense to start Kafka without a running ZuckiPir ensemble because Kafka depends on it. So when Kafka puts up, it looks, oh, where's my ZuckiPir? But if it's ZuckiPir, it's not alive, it will crash. And when Kafka is running, we create the cluster operator, create a new entity operator. And the entity operator is basically one part for the topic and the user operator. Good. So now everything is running. And now I would like to talk with Kafka and do some things. So there are basically commands called Kafka, console consumer, or console producer, and so on. And the best things about this is sometimes, so you have to provide a bootstrap server to them and some say, yeah, the command, the flag for that is console, bootstrap server, some of the commands say, it's Kafka brokers, some of the old commands need ZuckiPir, and so on and so on. And this is quite annoying. So there's, I created a so-called toolbox. We just configured the whole information using environment variables and then you do not need to remember the parameters. So we have here most of you who haven't seen something like that before. Okay. So this is basically a definition of a Kafka pot, a Kubernetes pot. A pot is the smallest entity in Kubernetes. And I say I would like to have a pot with the name Kafka Toolbox and with following containers inside. The containers are just normal Docker containers. So I would like to have a container called Kafka Toolbox with that image, oops, with that image and with the following environment variables. And the command the container should execute is just sleep infinity because it shouldn't do anything. I just want to log in to it and do things with that. And I can create that pot by using kubectl apply minus f Kafka Toolbox. It's created and now we can have a look at our watch and it's already running. And now I can connect to that. I'll just say kubectl xx minus it minus i is interactive and minus t says please I'll need a tty and then I just say Kafka Toolbox and I would like to execute bin bash. And now I'm in that pot. So we can say who am I? Okay. I'm Kafka. And now we can do funny things here. So for example, we can say Kafka topics minus, minus list. And Java says some things and it gives us an empty list because we have not created something. So let's create a topic. Creating a topic is relatively straightforward. Again, this is just a community's yaml. It says API version is SwimZ, kind is Kafka topic, meta data, please name it my topic, please assign it to that cluster and I would like to have three partitions and one replica. We have one replica. That's okay for us. That means if one of the, one replica goes down, we don't have, we lose that partition. But that's okay for our showcase. So I just say kubectl xx minus f topic. And then I can say kubectl get Kafka topic and I will get my topic, which was named my topic with three partitions and one application factor. If I do the same in my Kafka toolbox pot, I just say Kafka topics minus, minus list. And it will say, hey, I know about that topic, my topic. You can also say my top Kafka topics minus, minus describe minus minus topic, my topic. And we will get the configuration for that. Okay, we have partition count of three. Obviously replication factor four. And then we see which partition is on which, on which replicas, which brokers. So for example, partition zero is on our broker number two, partition one and zero, partition two and one. What else can we do? So let's write some stuff to it. So for that we use Kafka on zero, producer minus, minus topic. My topic is, now I can write stuff, whatever we want to. So let's read some things from that. I will just open it in another side-by-side. We again execute, I'll log into my Kafka toolbox. So I'm in and then I'll say Kafka console, consumer minus, minus topic, my topic. We can't see anything. What's happening here? Because by default it just starts, it just reads new elements from that. And I can say minus, minus from beginning. And if you're lucky, yes, we see the stuff I have written to it. Okay. And now I can write stuff to it. And sometime later, I don't know, I'm so slow here, it doesn't matter for me too much. Bloop, test, and so on. We remember, we have three partitions, yes? And we can't see anything here. Maybe you realize, look, bar was sent after the string and after foo, and we received it first. That's weird. Why is that? Any ideas? Why is bar before the others? Yes, so he said that if we have multiple partitions, we do not have any guarantee of the ordering between the partitions. So Kafka guarantees us ordering only in size of one partition, not across the partitions. So if you send two, one, and two to the same partition, we will get the same order. If you send them to different partitions, nobody guarantees us the order. And this is what we are experiencing here. So to see that, oops, we can say to the console consumer, I would like to have partition zero. And then we see some things. So we see that partition zero had that long string and bus in it. And we can have a look. Let's connect to another one and Kafka console consumer minus minus topic. So for partition one, we will see bar and test are inside. Let's do, oops. Okay, and let's have a look now. Oops. Sorry. Let's have a look now at the partition number two. Kafka. I'm now to read it better. So we see that foo and bloop are inside. So let's empty everything. Okay. Now we can produce some more data. So for example, it appears in the partition number zero. Let's call one again. Oh, why is it partition two? Let's do it again. Now it's in that partition. Hmm. Any ideas what is happening? Just doing round-warbin. We start with that one, then to that, then to that, and so on. That's a partitioning scheme. So we are just sending data and it appears somewhere. Let's do more with some strategy. I need to copy that command because it's a bit longer. Key. So we now use the console producer. So again, I haven't told you the who's, do we? Kafka is not a lock, not simply a lock or partition not simply a lock. But it's a lock of key value pairs. And you can just put key value pairs in that. And one of the reasons why we have keys and values is to have a way to distinguish to which lock to write. So we can have, say, let's write all messages with key one to partition one and so on. And this way we can go in T. For example, if you have a user database and you just are talking about this, containing users, if you want to say, hmm, I have a lot of users, but I would like to guarantee that the data for each user is in one partition, such that the order in that partition or for that user is always preserved. Because if you spread the data for one user across multiple partitions, there's no guarantee that you will get the correct order. And then bad things can happen. So we just do that. We just say, please provide us a way where we can insert key value. So I say just one, one. So it just says the key is one and the value is also one. Then I write it. It's maybe a better example. Let's clean that. And now every time I use the key one, it will appear in the partition number zero. The value doesn't matter. And you see that by default, the consumer doesn't show us the key. Most of the time, it doesn't matter too much. Other value, and it's always sent to partition zero. If we say two and put some data in it, then it appears there. You see? And that's guaranteed. So the console consumer just uses, in that case, the default partitioner and the default partitioner uses a hash algorithm to just decide to which partition to send the data. So and then let's say three and three again, the key, three will appear in partition number two again, not in one, because the hash algorithm is designed in that way. Let's try where we will appear four. And now we have, we know that key one will appear in partition zero, key two in partition two, key three in partition two, and key four in partition one. What else can I show you? I guess. Now let's exit that. And if you create here a topic, so in the default configuration, Kafka creates us a topic automatically if we write data to it. So we can say Kafka console. I can clean that, that. Okay. Kafka console producer. Give me an. No. Kafka console producer minus, minus topic. New topic, which just sounds like, exist. Just minus. Yeah, that's it. And now we just, oh, we get an error. Leader is not available. That's weird. That's relatively obvious. So we're just sending a request for a topic that doesn't exist currently. And that's why Kafka is creating now a topic for us. And we can have a look and if we continue writing something to that, we won't get any errors anymore. And now if we are using Kafka console, consumer minus, minus topic, what was the name? New topic. Beginning. We should get the values. Yes. So we can add the stuff. And now the interesting thing is we remember that the topic operator works in both directions. So if we do kubectl get Kafka topic, we are getting now the new topic too. And it has default configurations of one partition and replication factor of one. In a production environment, you probably should disable the functions that topics are created implicitly and automatically. That's not a good way to do that. And we see that there's also this, another topic that's called consumer offsets. And the consumer offsets topic is an internal topic where Kafka is storing the offsets. This way Kafka, the consumers know which messages they have for it and which they have to read. And then they go to the topic, say, please give me all consumer offsets and then they can decide where to start, where to continue reading. Okay. So that's it for the demo. I hope it will bring back the correct screen. Yes. Okay. So what have we explored today? We have explored a bit Kubernetes. Kubernetes is for me. It's a declarative container acquisition tool. Kafka is a distributed log. I think if somebody else got Kafka, Kafka is a distributed log and then you can go away because the rest are obvious. Not. And SwimZ is a way to run Kafka Kubernetes. There are also other ways, but for now that's one of the ways I explored and others I haven't did so far. So thank you for your presence here. And if you want to learn more about Kafka, I will give my next Kafka training in September in the Linux hotel in Essen and it's only in German. But if you want to get there, ask me or if you have more questions, you can just send me a message the way you'd like to. Thank you. So we have some time for questions. Any questions so far? You showed only the FMRL storage or persistence for Kafka on Kubernetes. How would the persistence storage work with the pods and all the volatile breaking down things and so on? I will leave that. The persistence storage works as you in Kubernetes, so how fluent are you in Kubernetes? Not bad. So Kubernetes has a concept called persistent volume claims and there you can say I would like to have a persistent volume with that properties. So for example, let's in Google Cloud, so it depends on your cloud environment or on your environment. So if you go on a cloud like Google, you can say Kafka, Qtl, get storage classes, which is short for SC and then there's a standard storage class and then you can create your own and so on. So you would probably want to have fast SSD storage in your Kafka cluster and you can just define I would like to have SSD with that much storage and that's it. So it's relatively straightforward. Again you should be relatively careful how to choose that. And as long as I have the mic, can you talk a little more about how all the components that found each other, like the toolbox you had, how did you find the QoBrowcust you previously set up? Okay. So the thing to find things in Kubernetes are services. Services create DNS names for your thing. So you can define for example, Kafka service or my hello world service and then you have a name for that. So if you say Qtl, get service, oops, we will see that we have four services here. So we have a Kafka bootstrap service, we have a Kafka VOKA service, we have a Kafka Zookeeper client service and Zookeeper node service. And then you see that the bootstrap service has a cluster IP. You can either use that IP to contact the bootstrap service or you can use that DNS name. And if we have, we can have just a look there, Qtl, get services, or describe. And then you see that we have multiple endpoints to that here, this is what we care about. And these are basically the Kafka VOKA's. And then you have Kubernetes will create IP tables rules. If it sees the DNS or if you contact the IP address, it's a virtual IP address and it will just randomly assign you the IP and go to one of the VOKA's. And we defined in our toolbox that the Kafka bootstrap server value is that one. And then we can just use the nice DNS name to do that. Basically normally you enter a list of your bootstrap servers or all your VOKA's, but we don't need to do that in Kubernetes because we have the service and it will just randomly assign you one VOKA where you contact it. And then it will give you back that piece on the names of the other VOKA's. Any other questions? Okay. If not, then thank you. Thank you for your attention. Thank you. Thank you. Thank you.
 So, hi, I'm Anatoly and you joined me today to explore the world of Kafka on Kubernetes.  But first start with who am I?  I'm adventurer and a computer scientist and so the obvious job choice for me is to be  an IT adventurer.  So, currently I'm giving Kafka training and also doing some Kafka and Kubernetes consulting  as a freelancer.  But today we want to talk about Kafka and Kubernetes.  Who of you heard about something about Kubernetes?  Who has Kubernetes running in production?  Okay, who of you knows something about Kafka?  Quite a bit, good.  A few runs Kafka in production.  Wow.  Four people.  More than expected.  But then, nevertheless, let's first talk about what is Kubernetes.  I think most of you know what it is, but let's start with very, very rough and short overview.  So for me, Kubernetes is a declarative container orchestration tool.  I'm a computer scientist and so the most interesting thing for me is the declarative part.  But let's first look at the words.  What is a container?  I think who does not know what a container is?  Good.  Okay.  So, all of you know that guy.  And for us it's enough if we say that a container is a mini VM with one application per container.  So we can have Postgres running in a container, GitLab, or Nginx.  So far so good.  So I think you all know that.  So what is a container orchestration tool then?  So the question is basically, okay, I have a runtime and I have a container.  And how can I put the container in the runtime?  For one runtime and for one container, that's easy.  You just let the container run on the runtime you have.  What if you have one runtime and N containers?  Again, containers are very, very tiny VMs and it's not a problem to run multiple containers  on one runtime.  Now the question arises, what if we have N runtimes and M containers?  How can we spread the containers across the runtimes given some heuristics or the way  we want to have it?  So maybe this could be a way to do that.  But I assure that this one is the best way.  Maybe there's a better one.  Or what do you do if the service wants to talk to each other?  What do you do if you want to have encrypted communication between the containers?  And what do you do when some of the failed rails is on fire or somebody's in your network  or the network is again broken on AWS?  So maybe you should think about network, security, FortoLand and so on.  And this is something as most of you know Kubernetes helps us a lot with that.  And finally, the last word in this phrase is probably declarative.  So declarative for me is I describe how the world should be like and how I want to have  it.  So I declare which applications should run, which names do the services have, which security  policies I want.  And then Kubernetes does all the plumbing and fulfills my wishes.  I like to say the computer, please, I want to have that word and the computer will do  the rest for me.  I'm lazy guy, so I don't want to do it myself.  So I think that's enough for Kafka, at least of course this is not a deep dive obviously.  So Kubernetes not Kafka.  And let's talk about Kafka again.  So for me again, I'm a theoretical computer scientist.  For me Kafka is a distributed log.  What is a distributed log?  Let's talk about log first.  So a log is an append only list.  We can have here six elements for example, a few characters and a white space and we  can add more data to the end of the list.  And what can we do with that thing, with that list, with that log?  We can basically, we can append data and we can read the full log starting at a specific  position.  So if we talk about appending data, that's pretty easy.  We have a function, let's call it push.  And let's say it takes a list of As, so As some type, it doesn't matter which one.  And it takes another element of the same type and gives us back a new list containing of  the types.  So for us it's just appending new elements.  For example, we just appended a Y.  And if we append elements, we get a new position or we call it offset.  So the list gets longer and we can address the elements in the list by the offsets.  So we can add another element and so on and so on.  And then we can read the full log starting at a specific position.  So we have a function which is called read from offset.  It takes again a list of As.  We don't care what is A and an integer or some number.  And then it will return us the log starting at that position.  For example, if we say that our current offset is zero and we call that function with that  list and zero, we will get back the whole log.  So let's append a new element to the log.  And we have to remember somewhere this current offset and then call the function again with  a new offset.  And then we will get only a U back, only one element, or a list containing the U.  So what does that mean for us?  In a log, we don't have any deletion.  We don't have any random access.  So we could tweak somehow the random access, but it's not intended to be used as random  access.  And there's no magic in it.  So we can describe a log just by the data structure and the two functions.  That it.  No magic.  The magic comes when we move away from the theoretical part and go to the practice.  And if you're interested in more things about logs, there's a good article from LinkedIn  about the log and what every software engineer should know about logs.  So distributed.  Kafka distributes this log.  And how does Kafka do this distributed thing?  What is rule number one of distributed systems?  Any ideas?  Down.  But still we need somehow.  We have a lot of distributed systems right now.  Maybe you have heard of this thing called microservices, distributed systems.  I think in every IT or computer science studies, there will be a lecture on distributed systems.  So distributed systems.  No, we don't want to do that.  But still we need to.  One of the two reasons are for them we have two match data for one computer and computers  like to crash.  I hope that my one took a rush today.  So if we have too much data, I was okay.  Bam.  So how can we solve that?  So Kafka approaches for too much data.  Kafka is using partitioning and for computer crashes, Kafka is using replication to come  up with that.  So again, we are not trying to build better and better hardware which don't crash.  But we say, okay, we cope with that.  This is a problem we have.  Theoretically, we are not able to solve that.  How can we cope with that?  So what is partitioning?  So a Kafka cluster, logically, with just a set of computers.  And we store multiple topics on the cluster.  Also on a database, you also have not only one table, but database cluster or database  instance, mostly multiple.  And then each topic, a topic is not a log.  A topic is basically a set of logs.  And we are partitioning each topic, for example, that way.  So topic A has three partitions.  And if we read them as the partition one of topic A, SEA14, what does it mean?  Maybe it does mean something, maybe not.  But we just partitioned our data.  So for example, the input for that was some data 12345 and a fluid.  So the partitioning was just one problem.  We put the S in the first partition, the O in the second partition, the M in the third  partition, and then started again.  Okay.  That's partitioning.  So how do we do that?  Basically, we have a producer which writes something and the producer itself decides  to which partition to write.  That's important.  This is in a traditional message queue, the producer writes to something and then the  system itself says where to put it.  And in Kafka, you have to decide yourself where to put it.  There are different strategies.  How do you can do that?  You can just write all data to one partition, but then probably bad things will happen.  You can randomly assign data to partitions and then multiple other ways.  We will have some time in the demo to explore that.  And when we have multiple producers, the picture stays simple but a bit chaotic.  Every producer just writes to the partitions that need to write to.  So reading, writing is quite straightforward.  Reading is a bit more complicated.  So if we have just one consumer and we want to guarantee that each message is read once  and we would like to have it also once and only once to read, then we send all data or  the consumer fetches all data from all partitions.  It's nice, but what if your consumer crashes or if the one consumer is not fast enough,  then you can just add more consumers.  And then Kafka will spread the partitions across the computers.  Or not Kafka is doing that, but the libraries, the consumer libraries will do that for you.  Okay, we have two consumers and three partitions.  And we send the first two partitions to the consumer one and the third partition to consumer two.  What do we do when we have three consumers?  We assign just one partition to each consumer and we are happy.  Could we scale further?  Yes, no?  Does it make sense to have a fourth consumer?  If we have a fourth consumer, then the fourth consumer won't get any data because we cannot  split this further.  If we want to guarantee that each message is read by exactly one consumer.  But still this can be reasonable.  For example, if one consumer crashes, then we will have another consumer which can take  over the work and continue.  Especially if you're running your favorite Java virtual machine and it needs some time  to warm up, it may be very useful to have a warm standby replica.  And it's recommended in many cases to have that.  Okay, that's about partitioning.  What about replication?  So we talked already about clusters.  If we have a cluster, we have multiple brokers.  In that example, we have three brokers.  Now if you remember, we have a topic with multiple partitions.  And we didn't put all partitions on one broker because why should we have multiple brokers?  But we have just, we put partition one, for example, on broker one, partition two on broker  two, partition three on broker three and so forth.  And partition four, then again on broker one because we don't have a fourth broker to put  it on.  Okay, nice about replication.  We have just read our partitions across the brokers.  But if broker one, for example, fails, we use partition one and partition four.  So how can we cope with that?  We replicate data.  So we put, for example, partition one, another replica to broker two, partition two to broker  three and so forth.  And if we lose broker one, partition one is still available.  It's on partition, on broker two, partition four is also on broker two and partition three  is on broker three.  So we will survive the crash of a broker.  Okay, let's say this is a very short introduction to Kafka.  So what about Kafka on Kubernetes?  Why would we want to do that?  Is it because of the password bingo you have heard?  Or is it the so-called conference-driven development?  Or is there something behind that that may be useful or make sense?  So we have heard a bit about what is Kubernetes, what is Kafka, what could be that Kafka on  Kubernetes?  So for me, I see chances to have that.  So what I like about Kubernetes is this declarative thing.  I can just say I want to have an application running, an engine X for example.  I want to have three replicas of my application.  And if they don't have any state, everything is fine.  I like about Kubernetes that it simplifies automation quite a lot because it can help  us with failover plans and so forth.  And if all your applications are already running on Kubernetes, then why not put everything  on Kubernetes?  Maybe it makes sense, maybe not, but let's see.  The challenges about Kubernetes, especially Kafka on Kubernetes is that Kafka itself is  very stateful.  That's obvious.  Kafka is a kind of a database.  And Kubernetes at least till the last few versions was not very happy about stateful  applications and it was quite challenging to deploy stateful applications in the character  of Kubernetes.  Kafka itself, if you give Kafka RAM or memory, it will take it all if it can.  So the recommendation for a small Kafka cluster are, yeah, have a free machine with 30 gig  RAM each for a small cluster for the beginning.  And then if you need more, then get more of it.  It also dislikes sharing CPU and so on.  And Kubernetes is all about resource sharing.  So you just often you try to put as much containers on one machine as possible to save  compute costs in your cloud environment.  And most importantly, Kafka itself requires very careful consideration where to deploy  it.  Kafka is very fast, but this comes at a cost.  The cost is you should be careful and you should know what you do.  Kafka itself does not commit data to disk.  It waits until the operating system sometimes decides to commit data to disk.  So if you have running multiple Kafka workers on one machine and that machine dies, you  probably will lose data.  You shouldn't put two Kafka workers on one machine.  On one physical machine, maybe you have two virtual machines and they are all on the same  physical machines.  You will have issues.  You shouldn't put Kafka clusters in one rack like that.  If your top-mount switch dies and you lose electricity or you lose network connection,  bad things will happen.  So you should be very careful where to deploy Kafka.  It doesn't matter if it's on bare metal virtual VMs, virtual machines or Kubernetes.  You should be always very careful.  And Kubernetes can have us with that, but we should be just aware.  So I would say no matter what the infrastructure is below or what platform you use, Kafka requires  a very good infrastructure and a team that can handle it.  If you lack one of that, you should not use a do Kafka yourself.  So my wish list for Kafka on Kubernetes is we want to have stateful sets for Kafka and  Zookeepers.  Stateful sets are one of the ways how to manage stateful applications on Kubernetes.  You really want fast storage and you really want fast network.  If you don't have fast storage or don't have fast network, you will get troubles or at  least Kafka will be just very slow.  So Kafka is very happy if it has fast storage and you can get a good performance out of  it, but if your storage sucks, Kafka will suck for you too.  And for Kafka and Kubernetes, I also wish to have services for finding clusters and  brokers.  And then of course, nice to have would be some way to have declarative resources for  everything.  So if you have multiple brokers, I don't want to create to define all of them.  I want to say, hey, I would like to have a Kafka cluster with three brokers, four brokers,  five brokers, how many I want.  Who of you likes configuring SSL keys?  Me not.  So I would like to have to say, I would like to have TLS encryption between all my services.  Do it.  I don't want to do that.  I want to declaratively define which users I want to have or which authentication method  I want to use and which users are authorized to do what.  And finally, we have multiple topics in Kafka.  And it would be great to have a declarative way to define them because if I do Kafka create  new topic and then I create another one and then I create another one, I would maybe lose  some time, my overview.  And you know what happens if, yeah, we have this topic.  I don't remember what the name was and who is using that, but somewhere we have the information  about that.  So declarative way for all of that would be great to have.  It's about infrastructure as a code, the bus water here.  And Kubernetes provides us more or less for free.  So why shouldn't we do that?  So these are things that Kubernetes operators providing us.  So what is a Kubernetes operator?  So Kubernetes operator is very similar to a Kubernetes controller.  And the controller is the thing that looks what you have defined and what is the current  state of the Kafka Kubernetes cluster and makes things happen such that the Kubernetes cluster  becomes the work we would like to have.  And same for operator.  I'm sometimes wondering why they have called it operators, why not just Kubernetes controllers,  but that's another topic.  And the basic idea about a Kafka Kubernetes operator is that it codifies human knowledge.  The knowledge of an operator, that's why the name.  And most often, especially for Kafka, the operators come with their own custom resources.  So you can extend Kubernetes to have custom resources where you can say, I would like  to have a Kafka cluster.  I would like to have a Postgres database with that configuration.  Please do that for me.  I don't want to do it myself.  And at the counter, there are three Kafka operators out there.  One is Strimsy.  Strimsy is deployed by Redhead and is the oldest of these three.  A recent very interesting operator is the Banzai Cloud Kafka operator.  It's relatively new.  It just was introduced a few weeks ago.  And so if you start a new project, you should definitely look at the Banzai Cloud Kafka  operator.  It's very clever.  They do.  And if you have a lot of money and want to spend it and want real good enterprise support,  you should probably use.  You will probably use a Confluent operator if you have Confluent enterprise licenses.  If you don't use their other services.  And we will talk today about Strimsy.  So Strimsy you can find on the sounders.  Okay.  So Strimsy is developed by Redhead and Strimsy is the open source version of that operator.  And if you want to give Redhead a lot of money, then you should buy AMQ streams, which is  the same product, but you get enterprise support for it.  And Strimsy consists basically of three operators.  One is the cluster operator, which manages all the cluster things, the topic operator  and the user operator.  I didn't tell you the whole story.  Kafka does not consist only of Kafka, but it has also a so-called zookeeper.  Who knows what zookeeper is?  Some of you.  Okay.  Zookeeper is basically a helper tool for Kafka to do cluster coordination and it stores  some data data.  Luckily, it will go away soon or later because nobody likes zookeeper in the Kafka community.  Zookeeper limits your performance, zookeeper limits the amount of topics you can have and  so on and so on.  So they want to get rid of it.  It will happen soon, hopefully.  So the cluster operator just does the provisioning for the zookeeper ensemble.  That's the provisioning for the Kafka cluster and also for the topic operator and user operator.  Zookeeper itself comes without any transport encryption.  Not nice.  But you can use a TLS proxy and the cluster operator configures everything fast.  This way zookeeper is talking to the TLS proxy and we have encrypted traffic in our network.  I like that.  And you also do global configuration for Kafka and zookeeper with the cluster operator.  The topic operator, the name says, does the topic management.  We can create new topics.  We can delete topics.  We can adapt the configuration the way we need.  And the interesting thing is that it has also two ways.  What does it mean?  So if we create a Kubernetes resource for a topic, the topic operator will create a  cluster Kafka topic, but if we have application that wants to create its own topics in Kafka  directly, then the topic operator will create a Kubernetes resource.  This way we will have always an up-to-date state of the topics in our Kubernetes.  We don't need to think about I created the topic in Kafka or Kubernetes.  We will have just a consistent way.  And similarly, the user operator does the user management, it does the authentication  authorization, and it stores the keys and passwords in Kubernetes secrets.  This is not the most recommended way because Kubernetes secrets are not really secret.  They are stored in plain text.  So you should think about a key management system, but if you have a good Kubernetes cluster,  you already have probably a key management system for that.  And then you just say, hey, Strimsie, please use that key management system and it works.  So that's it for that part.  Time for a demo.  So let's see if that works.  No, that one.  Okay, so I have here a Kubernetes cluster.  It's on Google Cloud Engine.  You can't see anything right now, but give me a second.  So you can say QBZTL.namespaces.  So Kubernetes consists of namespaces where you can just logically group stuff in it, and  it's relatively empty.  So if you create a new Kubernetes plus on Google Cloud platform, you will get that.  So we want to create our own namespace.  I will just say QBZTL create an S Kafka and switch to that one.  Okay, it says we are now in the namespace Kafka.  Let's install.  First, we have to install Strimsie and one of the easiest way to do that is using Helm.  So you just say Helm install, then you give a name and say I want to install this Strimsie  Kafka operator.  Take some time.  And now it says yes, we are ready.  And then we can do a QBZTL getProt.  And we see that we have one on pot that is running and the readiness state is 0.1.  So we have probably to wait a second.  And I like I have a command which is just, who knows about the watch comment on Linux.  Awesome tool.  So watch is doing basically you say watch, please execute the following command every  second and please show me a graphical diff for that.  And then you will see, oh, look, we just execute QBZTL getProt every second and we see here  the diff, we see that the age of the container changes every second.  Awesome.  Okay, now we have the cluster operator running.  What should we do next?  Let's create a Kafka cluster.  And for that, we just use, oops, that's the fun.  One one, we are using, so you are configuring Kubernetes using YAML files and there's Strimsie  installs and own custom resource definition.  And this way we can say, yeah, I would like to have something in the API version Kafka.simsie.ios  like B1, Beta 1.  So we may think that Simsie is still better.  And please do something with this kind Kafka.  Then we give it a name and then we specify.  So I would like to have a Kafka in version 2.3.0.  I would like to have three replicas.  I would like to have a listener, a plain text listener.  We could also say, please do a TLS version, but for now it's just a demo environment.  It doesn't matter for us.  Storage, I would like to have a FMRL storage.  You should not do that in production, obviously.  FMRL storage means that there's no persistent storage and if we kill the pot, the storage  is lost.  That's okay for our environment here.  I would like to have a ZuckiPir ensemble with three replicas and again, FMRL storage.  And then I say, yes, I would like to have the topic operator and the user operator.  Then I just do a cube CTL, apply minus F Kafka.  And then we can have a look at our watch.  What will happen?  Something changed.  So we got three ZuckiPirs.  They are fast today.  They are already running.  But we see the readiness date is zero of two.  So we just wait a few seconds when they are ready.  Oh, they are already one of two.  That's nice.  Two of two.  And then we have a ZuckiPir ensemble running.  And you see what the, now the Kubernetes cluster operator looks.  Oh, ZuckiPirs up.  Now I can start Kafka because it doesn't make any sense to start Kafka without a running  ZuckiPir ensemble because Kafka depends on it.  So when Kafka puts up, it looks, oh, where's my ZuckiPir?  But if it's ZuckiPir, it's not alive, it will crash.  And when Kafka is running, we create the cluster operator, create a new entity operator.  And the entity operator is basically one part for the topic and the user operator.  Good.  So now everything is running.  And now I would like to talk with Kafka and do some things.  So there are basically commands called Kafka, console consumer, or console producer, and  so on.  And the best things about this is sometimes, so you have to provide a bootstrap server to  them and some say, yeah, the command, the flag for that is console, bootstrap server,  some of the commands say, it's Kafka brokers, some of the old commands need ZuckiPir, and  so on and so on.  And this is quite annoying.  So there's, I created a so-called toolbox.  We just configured the whole information using environment variables and then you do not  need to remember the parameters.  So we have here most of you who haven't seen something like that before.  Okay.  So this is basically a definition of a Kafka pot, a Kubernetes pot.  A pot is the smallest entity in Kubernetes.  And I say I would like to have a pot with the name Kafka Toolbox and with following containers  inside.  The containers are just normal Docker containers.  So I would like to have a container called Kafka Toolbox with that image, oops, with  that image and with the following environment variables.  And the command the container should execute is just sleep infinity because it shouldn't  do anything.  I just want to log in to it and do things with that.  And I can create that pot by using kubectl apply minus f Kafka Toolbox.  It's created and now we can have a look at our watch and it's already running.  And now I can connect to that.  I'll just say kubectl xx minus it minus i is interactive and minus t says please I'll  need a tty and then I just say Kafka Toolbox and I would like to execute bin bash.  And now I'm in that pot.  So we can say who am I?  Okay.  I'm Kafka.  And now we can do funny things here.  So for example, we can say Kafka topics minus, minus list.  And Java says some things and it gives us an empty list because we have not created  something.  So let's create a topic.  Creating a topic is relatively straightforward.  Again, this is just a community's yaml.  It says API version is SwimZ, kind is Kafka topic, meta data, please name it my topic,  please assign it to that cluster and I would like to have three partitions and one replica.  We have one replica.  That's okay for us.  That means if one of the, one replica goes down, we don't have, we lose that partition.  But that's okay for our showcase.  So I just say kubectl xx minus f topic.  And then I can say kubectl get Kafka topic and I will get my topic, which was named  my topic with three partitions and one application factor.  If I do the same in my Kafka toolbox pot, I just say Kafka topics minus, minus list.  And it will say, hey, I know about that topic, my topic.  You can also say my top Kafka topics minus, minus describe minus minus topic, my topic.  And we will get the configuration for that.  Okay, we have partition count of three.  Obviously replication factor four.  And then we see which partition is on which, on which replicas, which brokers.  So for example, partition zero is on our broker number two, partition one and zero, partition  two and one.  What else can we do?  So let's write some stuff to it.  So for that we use Kafka on zero, producer minus, minus topic.  My topic is, now I can write stuff, whatever we want to.  So let's read some things from that.  I will just open it in another side-by-side.  We again execute, I'll log into my Kafka toolbox.  So I'm in and then I'll say Kafka console, consumer minus, minus topic, my topic.  We can't see anything.  What's happening here?  Because by default it just starts, it just reads new elements from that.  And I can say minus, minus from beginning.  And if you're lucky, yes, we see the stuff I have written to it.  Okay.  And now I can write stuff to it.  And sometime later, I don't know, I'm so slow here, it doesn't matter for me too much.  Bloop, test, and so on.  We remember, we have three partitions, yes?  And we can't see anything here.  Maybe you realize, look, bar was sent after the string and after foo, and we received  it first.  That's weird.  Why is that?  Any ideas?  Why is bar before the others?  Yes, so he said that if we have multiple partitions, we do not have any guarantee of the ordering  between the partitions.  So Kafka guarantees us ordering only in size of one partition, not across the partitions.  So if you send two, one, and two to the same partition, we will get the same order.  If you send them to different partitions, nobody guarantees us the order.  And this is what we are experiencing here.  So to see that, oops, we can say to the console consumer, I would like to have partition zero.  And then we see some things.  So we see that partition zero had that long string and bus in it.  And we can have a look.  Let's connect to another one and Kafka console consumer minus minus topic.  So for partition one, we will see bar and test are inside.  Let's do, oops.  Okay, and let's have a look now.  Oops.  Sorry.  Let's have a look now at the partition number two.  Kafka.  I'm now to read it better.  So we see that foo and bloop are inside.  So let's empty everything.  Okay.  Now we can produce some more data.  So for example, it appears in the partition number zero.  Let's call one again.  Oh, why is it partition two?  Let's do it again.  Now it's in that partition.  Hmm.  Any ideas what is happening?  Just doing round-warbin.  We start with that one, then to that, then to that, and so on.  That's a partitioning scheme.  So we are just sending data and it appears somewhere.  Let's do more with some strategy.  I need to copy that command because it's a bit longer.  Key.  So we now use the console producer.  So again, I haven't told you the who's, do we?  Kafka is not a lock, not simply a lock or partition not simply a lock.  But it's a lock of key value pairs.  And you can just put key value pairs in that.  And one of the reasons why we have keys and values is to have a way to distinguish to  which lock to write.  So we can have, say, let's write all messages with key one to partition one and so on.  And this way we can go in T. For example, if you have a user database and you just are  talking about this, containing users, if you want to say, hmm, I have a lot of users,  but I would like to guarantee that the data for each user is in one partition, such that  the order in that partition or for that user is always preserved.  Because if you spread the data for one user across multiple partitions, there's no guarantee  that you will get the correct order.  And then bad things can happen.  So we just do that.  We just say, please provide us a way where we can insert key value.  So I say just one, one.  So it just says the key is one and the value is also one.  Then I write it.  It's maybe a better example.  Let's clean that.  And now every time I use the key one, it will appear in the partition number zero.  The value doesn't matter.  And you see that by default, the consumer doesn't show us the key.  Most of the time, it doesn't matter too much.  Other value, and it's always sent to partition zero.  If we say two and put some data in it, then it appears there.  You see?  And that's guaranteed.  So the console consumer just uses, in that case, the default partitioner and the default  partitioner uses a hash algorithm to just decide to which partition to send the data.  So and then let's say three and three again, the key, three will appear in partition number  two again, not in one, because the hash algorithm is designed in that way.  Let's try where we will appear four.  And now we have, we know that key one will appear in partition zero, key two in partition  two, key three in partition two, and key four in partition one.  What else can I show you?  I guess.  Now let's exit that.  And if you create here a topic, so in the default configuration, Kafka creates us a topic automatically  if we write data to it.  So we can say Kafka console.  I can clean that, that.  Okay.  Kafka console producer.  Give me an.  No.  Kafka console producer minus, minus topic.  New topic, which just sounds like, exist.  Just minus.  Yeah, that's it.  And now we just, oh, we get an error.  Leader is not available.  That's weird.  That's relatively obvious.  So we're just sending a request for a topic that doesn't exist currently.  And that's why Kafka is creating now a topic for us.  And we can have a look and if we continue writing something to that, we won't get any  errors anymore.  And now if we are using Kafka console, consumer minus, minus topic, what was the name?  New topic.  Beginning.  We should get the values.  Yes.  So we can add the stuff.  And now the interesting thing is we remember that the topic operator works in both directions.  So if we do kubectl get Kafka topic, we are getting now the new topic too.  And it has default configurations of one partition and replication factor of one.  In a production environment, you probably should disable the functions that topics are  created implicitly and automatically.  That's not a good way to do that.  And we see that there's also this, another topic that's called consumer offsets.  And the consumer offsets topic is an internal topic where Kafka is storing the offsets.  This way Kafka, the consumers know which messages they have for it and which they have to read.  And then they go to the topic, say, please give me all consumer offsets and then they  can decide where to start, where to continue reading.  Okay.  So that's it for the demo.  I hope it will bring back the correct screen.  Yes.  Okay.  So what have we explored today?  We have explored a bit Kubernetes.  Kubernetes is for me.  It's a declarative container acquisition tool.  Kafka is a distributed log.  I think if somebody else got Kafka, Kafka is a distributed log and then you can go away  because the rest are obvious.  Not.  And SwimZ is a way to run Kafka Kubernetes.  There are also other ways, but for now that's one of the ways I explored and others I haven't  did so far.  So thank you for your presence here.  And if you want to learn more about Kafka, I will give my next Kafka training in September  in the Linux hotel in Essen and it's only in German.  But if you want to get there, ask me or if you have more questions, you can just send  me a message the way you'd like to.  Thank you.  So we have some time for questions.  Any questions so far?  You showed only the FMRL storage or persistence for Kafka on Kubernetes.  How would the persistence storage work with the pods and all the volatile breaking down  things and so on?  I will leave that.  The persistence storage works as you in Kubernetes, so how fluent are you in Kubernetes?  Not bad.  So Kubernetes has a concept called persistent volume claims and there you can say I would  like to have a persistent volume with that properties.  So for example, let's in Google Cloud, so it depends on your cloud environment or on  your environment.  So if you go on a cloud like Google, you can say Kafka, Qtl, get storage classes, which  is short for SC and then there's a standard storage class and then you can create your  own and so on.  So you would probably want to have fast SSD storage in your Kafka cluster and you can  just define I would like to have SSD with that much storage and that's it.  So it's relatively straightforward.  Again you should be relatively careful how to choose that.  And as long as I have the mic, can you talk a little more about how all the components  that found each other, like the toolbox you had, how did you find the QoBrowcust you previously  set up?  Okay.  So the thing to find things in Kubernetes are services.  Services create DNS names for your thing.  So you can define for example, Kafka service or my hello world service and then you have  a name for that.  So if you say Qtl, get service, oops, we will see that we have four services here.  So we have a Kafka bootstrap service, we have a Kafka VOKA service, we have a Kafka Zookeeper  client service and Zookeeper node service.  And then you see that the bootstrap service has a cluster IP.  You can either use that IP to contact the bootstrap service or you can use that DNS name.  And if we have, we can have just a look there, Qtl, get services, or describe.  And then you see that we have multiple endpoints to that here, this is what we care about.  And these are basically the Kafka VOKA's.  And then you have Kubernetes will create IP tables rules.  If it sees the DNS or if you contact the IP address, it's a virtual IP address and it  will just randomly assign you the IP and go to one of the VOKA's.  And we defined in our toolbox that the Kafka bootstrap server value is that one.  And then we can just use the nice DNS name to do that.  Basically normally you enter a list of your bootstrap servers or all your VOKA's, but  we don't need to do that in Kubernetes because we have the service and it will just randomly  assign you one VOKA where you contact it.  And then it will give you back that piece on the names of the other VOKA's.  Any other questions?  Okay.  If not, then thank you.  Thank you for your attention.  Thank you.  Thank you.  Thank you.
2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_0_59.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_60_179.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_180_1751.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_1752_6539.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_6540_14531.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_14532_33815.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_33816_34775.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_34776_38699.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_38700_39575.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_39576_39743.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_39744_42995.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_42996_43163.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_43164_45167.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_45168_45371.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_45372_48299.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48300_48431.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48432_48551.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48552_48599.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48600_48647.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48648_48743.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48744_48947.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_48948_49151.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_49152_49451.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_49452_49847.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_49848_50063.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50064_50171.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50172_50723.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50724_50771.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50772_50915.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50916_50975.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_50976_51359.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_51360_52163.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_52164_52415.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_52416_52847.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_52848_53123.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_53124_68351.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_68352_68459.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_68460_68723.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_68724_71939.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_71940_72695.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_72696_72815.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_72816_72959.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_72960_73007.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_73008_74087.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_74088_74171.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_74172_74495.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_74496_77195.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_77196_77243.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_77244_77447.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_77448_77867.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_77868_78551.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_78552_78707.png 2019_-_Kafka_on_Kubernetes_HyfJosBuVSs_78708_78947.png