Ceph storage with Rook
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Trost, Alexander
The talk will show how simple it is to use Rook for Ceph storage on Kubernetes. In addition to showing how simple it is to use the storage, the ease of adding and removing nodes will also be shown in a demo. Target audience are people interested in Ceph and Kubernetes. Topics: What is Rook? Architecture of Rook Kubernetes native integration What can Rook help you do (better) with Ceph? Demo of Rook's capabilities Creating a Ceph cluster Showing ease of consuming storage using an example application Adding and removing a new (Ceph) cluster node Why Rook?
 Okay. Welcome everybody to our first roof talk and welcome, Alexander the roof maintainer. Yeah. So my name is Alexander Trost. I'm a well, roof maintainer. I'm just a side of that. I'm a DevOps engineer at cloud ability. Today I'm kind of going over using roof, yeah, using roof in Kubernetes to run SAP in Kubernetes and kind of going into like what are the advantages of doing that. And you have from a genocide kind of like what is through a bit of the architecture of roof a bit into like the integration with Kubernetes that well, in the end, most of the time you want to use the self storage you're running with roof inside of your Kubernetes cluster for your applications. Let's say I don't know like it my skill database or something. And then well, the advantage is kind of like how can it help you doing that. And one part which probably falls flat due to technical issues on with my laptop. Well, I'll see if I can get some information on some video or something up that we kind of get a demo of the creation of a cluster and also how easy it is done in Kubernetes to consume the storage there. And while you're adding and removing a new note to the cluster to the self cluster in the end, well, I have to see. Yeah, and in the end, just like a wire route, a few more points there and yeah. So, Rook is a cloud native storage or a straight up cloud native. Everybody talks about it's it's well, those this cool cloud thing, you know, it's specifically made for being run in a Kubernetes there. Yeah, which simply implies that it's in the container. So it's basically a cloud native container search orchestrator. And through this through this orchestration. Rook well can run, for example, safe, but also other storage softwares, which I will come back later on. For you in your Kubernetes cluster, the special point which yeah, Kubernetes allows through the extensibility of the Kubernetes API is that you have the possibility of create your own custom types and also well in the end, you have the control us which react to these custom types in Kubernetes. More on it later and in general the goal here is that for the orchestration part, we try to have the deployment automated as best as possible bootstrapping of the software of the for example, in this case, a self cluster, the configuration provisioning, scaling, upgrading, migration disaster recovery monitoring and resource management. All points which we try to cover not all of them. Those points are covered yet, but well, it's open source. We are happy to see people help us reach the goal of full automation of your, for example, self cluster in Kubernetes. As I already said a bit earlier, Rook is not only self, self is one of the many storage providers that Rook can run for you. For example, MinIO, which is an object storage, maybe you've heard about it, then there's also from an excenter guys, it was HFS, it's well kind of in the direction of like self, which provides block storage, file storage and also object storage, but in a more like, well, I think they have a huge point in like, geo replication by default kind of in it, definitely worth to check it out. And some other tool which I have laid on a slide, which are currently on in Rook. It's well, we had for some, it's open source. It's especially what is good for us, it's hosted by the Cloud Native Computing Foundation. It's currently in the incubating state, incubator state of the CNCF project level. Yeah. Rook framework. As Rook is not only there for just running self, but also for MinIO, HFS and so on, Rook tries to get generalization of certain types which are common across those multiple storage back ends. So for example, to say it with for example, yeah, self and HFS, the selection of the devices on nodes is the same. There are no two types which are different. This is one type where we basically have a list with some more details which we go in later too, of which devices to use. And only already with those common specifications back used between multiple storage providers, it's most of the time pretty easy to, well, not that I would recommend it too much to instead of like, distro hopping as you storage software hop all the time, but in the end something like that would be possible because you could just copy and paste the device list from, well, self for example, then to HFS or well, back again. Yeah. With the framework part of Rook there, we not only try to have this common ground of specifications, we simply also try to have combined testing efforts that makes it easier for all those platforms which have their own operator in Rook to be, well, tested together, have policies and stuff, just share the code that is not like every operator which for example would run something like self, like HFS and other possible software backends, software storage backends, that they don't need to write that from scratch all the time. So that's one of the goals with the Rook framework basically there. Yeah. The architecture of Rook, it's going a bit into Kubernetes right now here as, we're kind of a general look right now here. We have the Kubernetes API, we have our client utility which we can use to talk with the Kubernetes API and Kubernetes uses Etsy for the data store of the API objects. And if we basically start from here, we have our client utility with which we tell the Kubernetes API, we tell them about these new objects. So that in the end we end up with objects like self cluster, self block pool, self object service, self file system and well for the other operators, if you install them too, obviously those two. When we have created those objects, we can access them again, just run it like for those knowing about kubectl like kubectl gets self cluster and we get the object as it is currently in the API and in more common, more current releases of Kubernetes, I think with one something 12, 13 or so, you can get some more information about the object as it is, if you simply run the kubectl gets to, well retrieve the object from the Kubernetes API so that you have a quicker overview of if it has been created successfully, how many monitors am I running and just some general information. Moving on to the operators, as said with the multiple storage back ends, for example we have the self operator there then, with minni operator and so on for each storage back end to own operator and the operator basically takes care of creating the deployments, the deum set and all that is needed to run the storage software. So for self, it would be creating deployments for the monitors, creating deployments for the OSDs and all other components like manager, MDS, RGB and then also for example configure certain aspects in the manager like enable the dashboard and for example enable SSL on it, disable SSL on it, stuff like that. And it's making it simple by, this is possible again because we have one common object, the self cluster object here. With this one object with which the root operator knows how to handle it and has all the options we support in it, which you can just go ahead and add it for not all options right now, but for a good amount of options you can go ahead like toggle the dashboard on the fly, if you have it on false when you create a decluster initially and just go edit it, put through and apply it against the Kubernetes API and the root operator will pick up this change and do its thing and enable the dashboard. Now moving on to this part here, this is basically like we are on the node right now. The demons are obviously for let's say for example SAF, we have a monitor in OSD or one of the SAF demons which is placed on a node, if a mon is placed and do you want to use the disks on a node also for storage you would obviously also have OSD demons running on that. But all containerized so there's no like, well conflict with other stuff running on the system, it's well container so well there are certain differences between which container runtime you use from the isolation level but they are isolated at least. So the container part there is also the point where the upgrading for RUG is also making it easier because if we just go ahead in the deployment which for example run in OSD there, we just go ahead to the Kubernetes API and say hey, change the image from SAF version 12 something to version 13 something. And for the other you basically the same update aspect comes in place too because Kubernetes does this management for us. Then there's also the RUG agent. The RUG agent is in our current case the so-called RUG flex volume provider, plug-in provider. This is, well flex volume is kind of like the predecessor of CSI kind of in Kubernetes where CSI is the container storage interface which well kind of as the name implies like a common interface for storage not only a thing for containers but in general for storage. And flex volume is kind of like that too but more on a limited scale as I know especially for Kubernetes there. And we have it, we have it running right now still with flex volume or just more or less to the point because of Kubernetes with having CSI support on stable, beta or stable with version 113. And we're currently separating like the last three versions of Kubernetes so we still have to have the flex volume. But to the flex volume here if I have a pot which wants to use a volume on this node it would, the Q-Blood, the node component of Kubernetes would call out to a socket on a node of flex volume socket and then the RUG agent would receive this request and then well go ahead and get the information of the volume and do the RBD mapping and then the RBD mounting. One quick note, the Q-Blood I think is in most cases doing the formatting of the volume. So, yeah, I think the formatting is done by the Q-Blood which, yeah? I'm not sure if you're going to tell us more about this or not, but for instance OSDs on SAP, if you need access to the block devices, is this accomplished with flex volume or will it be accomplished with the new CSI stuff? I'm trying to answer questions. So, you mean the block device, if I mount the map the volume? No, so the OSDs themselves, they are backed by the disks where they write the objects and all that stuff. So, the containers will need to have those block devices exposed to the parent, right? Okay, so if I understand correctly the question is that how are the OSDs able or getting the access to the block devices? Well, right now it is simply done by mounting the slash depth of the host into the container. There are ways, I think, especially through, what was it, local storage, local storage from Kubernetes which has been introduced, I think, like 1.9.1.1.1.1.1.1, I think, which could potentially be used for this exactly purpose. We are still kind of looking into how we can make it, use it like this, but we're kind of like right now a slash depth is basically working for us. But we are also already at least looking into restricting the slash depth access to just the device and or the partitions that are on this disk than 40 SAP OSDs. There's still a good amount of work to do to get there. Yeah. Thank you. So, yeah. So, if I create the RookSafe operator and then go ahead and create a RookSafe cluster object too, well, I have, as already said in the previous example here, we have the agent pod which is running in all nodes and the operator then, as said, takes care of creating the SAP component pods like the OSDs, the monitors, manager, MDS, RGV is also there. And, well, in the end, you have a SAP cluster with that. Yeah. So, to summarize this part, we have this custom object with kind SAP cluster which Rook operator, the RookSafe operator has defined. You create it and what you currently with that would get is, now, we're a SAP cluster running SAP version 13.2.4 something, something. And ignoring your data, the, well, data, the host path in Rook is right now where most config is stored and monitor data, but we're working on trying to, well, make it really just for configuration, maybe still one data but not OSDs. Because right now the behavior would be that if you don't specify like disks in anything else, you would get an OSD using a directory of, well, basically a file, directory OSD which, well, if you have disks from performance devices, way better. The disk one is way better. Yeah. But coming back here, we have like the dashboard part here where we say, yeah, we want the dashboard. So we have enabled through. We can control a bit about the monitors like how many do we want. If we want multiple monitors on one node, this is if you have, well, only three nodes, you may or may not want this to happen. Or if you have multiple nodes, you normally want to disable those modern three nodes, for example, because then you always have room to move a monitor from a node which failed to another node and, well, create a new monitor there, basically. This part with the storage here is here, for example, use all nodes, use all devices is kind of like controlling there's more to control. But those two options are kind of allowing you to select where storage will be used and what storage will be used. With the use all nodes, it's kind of, well, use all nodes that the Rook operator can place OSDs on it, which are pretty like that. Valid in Kubernetes aspect. Use all nodes with implied that all empty devices will be used for an OSD. And if we, I didn't upload it last time. If we go, if we skip a few slides ahead, we come back again to this storage configuration part where we have, again, those use all nodes and use all devices. Instead of, for example, having use all nodes true that all nodes which are applicable will be used, we could also specify a list of nodes which will have information on the next slide. We can also have a simple device filter where we go, yeah, use all this with SD and, well, Wildcard, basically, and then Rook will take all disks with this matching ragex besides disks that are not empty. There's also possibility to specify certain configuration parameters, not only in a cluster-wide level, but also on a node and even OSD part level, where in this case, for example, we have, like, an option which normally is only used for NVMe or, well, faster than SSDs devices is the, how many OSDs should be created per device. If we go just one further, we are at the nodes list here, and there you can see that, for example, you have this node, you say, well, use a directory, have those resource limits in place from Kubernetes site that contain the OSD container for this node as it's on the node level here, we'll have those resource limits in place. Then, for example, here with a device, you have the node name and, in this case, specifically say, hey, use SDB, but you can also, again, if it would be, for example, NVMe, as it's the second device here, you could then specify, like, config and then OSDs per device, like, free, or, well, insert a math for NVMe devices, how many OSDs here, basically. There we even have it, OSDs per device, and we just have it per device and our node level on cluster-wide level, so that's not all. Just as we find where we left off here. Yeah. To shortly go into the further Kubernetes-native integration part, besides the so-called customer-assource definition, those custom-types objects we can create, we already have in Kubernetes storage classes, which allow us, the admin most of the time, to specify which provisioner and certain parameters for this provisioner to be called, the provisioner will be called if you create a volume now, a persistent volume claim. If it matches this storage class, if it has this storage class in it, Kubernetes will take care of talking with the provisioner and saying, hey, somebody requested 20 gigs of storage, for example, if you go for one further. Here, just example with 20 gigabytes of storage, and here we specify the storage class names, or especially use the one we just saw. If we create this, this will result in 20 gigabytes of storage. That's kind of where Kubernetes takes over in point of storage management, as Kubernetes has all the persistent volume claims, which I hope for everyone using them, always reflect the claims an application has made, like, hey, I need 500 gigs of storage. So it's always this application is using 500 gigs of storage. And in the background, it's kind of like a not on Perl application level, you have the persistent volumes, which are, again, also having the information, like, which storage class has been used to create this volume, how much storage is this actually, what access mode is used, like, Kubernetes has three access modes, where the first is read, write, once, which is, well, only one can read and write to it. The second is read, write, many, where, well, many can read and write at any time at the same time. And there is reads only many, which, well, many can read only at the same time. This way here. Yeah. So what can rule kind of do to make it, well, if you have a self-class or especially because, well, you need to run a self-class, make it better for you to, well, run a self-class in Kubernetes, especially, as I said, if you have one, if you don't have one and you're like, yeah, let's go, it might be a bit problematic because you need to know a bit of stuff with Kubernetes. It's, well, the simple fact that if you start one higher with a platform and then put it on, it's, you should have knowledge about a platform. Part one, what Rook does is, as we heard, health checking for monitors. So if you have, let's say, five nodes and you currently, just for simple example reasons, have your monitors one on the first node, second on the second, and third on the third, and the third node would fail. It would move this monitor, fail it in the self-cluster, and move it over to, let's say, node four or five, which are available for that. The simple management, simply through the Kubernetes API, in a second, through the Kubernetes API in point that you have those YAML manifests, kind of, say and put it like this, with infrastructure as a code kind of. It's, yeah, more and more, well, not only infrastructure as a code, but kind of application deployment as a code, depending on how far you see it. You have those YAML files which define how your self-cluster should look, which pools should be created. You just have a YAML file which has here a pool object with these and these names, how many replications, should you use the ratio code, and should it, which failure domain, which, as a YAML object is, well, you just, you've got to create it to the Kubernetes API, and you have it. You have your pool created in a few seconds, and it makes it easier on that side to, well, manage those things, too. And files, for example, same here, too. It's not like, well, I want you to use file system. Let me buy five servers and install seven, well, set up the, yes. With that, with RUG, well, you would need to put Kubernetes on it, and then just create a file system object which has size numbers and everything, like for the pools, again, for this file system, what else are the numbers like, how many active MDS do you want, should there be a standby MDS to, and just gives you a bit of, well, playground, not to say like about a good amount of room to have certain options and have that automatically happen. Same for RGV. You have an object, control it through that, have certain options to control the behavior of the pools or how they should look. Yeah. And the third point I would like to bring up here is that, as we saw with the storage selection, again, there, you have a self-cluster object. You have either a list of nodes or a more generalized device filter or something where you say, use those disks in the folder cluster or even directories right now, which just as a side note for the storage selection, we have Travis Nielsen, which is also working on the Rook project, looking into enabling LVM kind of with the self-volume, the self-volume part, to be created, well, through self-code, well, no, self-code, and not as it was done since the release of 0.9. Now it's done not for all cases like that anymore, for Rook's own, well, it was the preparation, partitioning, formatting, and so on, process. Yeah, well, we already went through that. Well, the Y-Rook part is kind of covered by what you would get as benefits if you have a Kubernetes cluster for something like that, but so, in general, it simplifies certain tasks for the storage back end. There are assets, still some work, like, for example, replacing a node right now. Still a bit of a manual process, but we're getting there. It's more or less a bit of a clash, like, should we listen on the nodes that are in the cluster and then react if one goes missing, for example, or is already just not ready, or just some points where, for whatever reason, maybe an SD issue or something, there's an empty node, so we can go, ah, nothing to run, so just grab it all, kind of. But, well, as we're happy to just, well, see people maybe just chime in on the discussions on those topics. Yeah, self-managing part is kind of that it's, well, we don't have lifeless props and readiness props for all self-components, but we think we have it, at least for the manager right now, and for the monitor, we do kind of external half-checking from the operator side to see, is this monitor still in form, or do we need to fail it over to a new node? Yeah. And the part with the dynamic provisioning there that I create a claim, and a few seconds later, my block volume in-serve has been created and linked with Kubernetes through the persistent volume object is, well, it's a breeze to, if you want to run with applications, where you don't need to, like, well, fill in a ticket and say, hey, I need 20 gigs of storage, can you please provide it to me? And someone starts running through the data center and tries to find the data, they just go for something for you. But it's simply dynamic there, which is pretty good for, well, especially for developers. We are also interested with kind of like with the guys from cross-plane I.O., Jared Watson, Basam Tabara, if I got his name correctly. We're looking into also trying to get more abstraction in, like, for example, and a developer would say, yeah, I need a bucket, so it just creates a bucket object. And for example, we also have support for CockroachDB. So we would obviously, as a database, also want to look into having a database object which the developer would just create and then in some way, which is kind of still up for discussion, should it be a service broker or should we do some magic, like, with Kubernetes, well, mounting a directory into then the part which contains the credentials, for example, then to this database or so. Again, there, if you want to kind of jam in there, we're happy to have discussion on that. The last point here in this list is, well, it's depending on how you see it. One problem, I think, which is still existing for AWS, with the there, what was it? The EBS block storage from Amazon. I think what was it? The failover time was like, when the node failed, it would take, I think, like five minutes or so to failover to another node. And, well, I hope you're only running cloud native applications or there shouldn't be a problem as you have more than one replica. But it's still kind of a not too good number as you use, even though it's cloud native and it should be okay with a failure of a part or even multiple parts in your cluster or multiple applications. It's still kind of a bummer to have, like, wait five minutes in the case of a failure of a part. So that's kind of where I'm kind of saying it's the last vendor login. Because, well, you would run a self-class, which, yeah, on the one hand, could also create more maintenance instead of, for example, taking care of just the EBS volumes of Amazon. But the failover is, in this case, for example, faster or you just don't have to use their storage. Yeah. Yeah. As kind of, yeah, said in the beginning, it's not only self. There's not only a root-self operator. There's also a root-mini operator, which, well, is object-searcher. We have a minio-object-search object. We have also Cockroach-Gb, which, well, right now we only have a Cockroach-Gb object. But, again, we are open for people communicating and discussing with us how we can go to best path if it's more of a service broker approach or not approach. HFS has been implemented by the next vendor guys. Just there again, also a big shout-out to them, as it's a huge work. It's not like a minio. We have two or three replicas of onus, where it's full-set only. But there's, again, kind of like self, more complexity simply behind it. And it's amazing to see them have implemented it in Rook. And so they're using certain parts of their Rook framework. And an FS server has minio and Cockroach-Gb. And, well, HFS too implemented thanks to the community. So, yeah, well, I don't have to demo right now. Let me get to this page first. If you want to get involved, we are on GitHub. I think some people are still on GitHub, right? Nothing to do with GitHub. We have a page which we are currently reworking. So if you're right now like, where do you find this and this, we're working on it. We know of the problem. And, well, we have a Slack channel, a Slack, no, a whole Slack for our own. Yeah, if you want to join, we also have a conference channel. And, well, if you have questions, a channel, a channel, and so on. Well, Twitter, we still have a mailing list, I think. And there's community meetings every second week. Yeah. I would just jump to the questions right now. OK, so two questions. On one of your slides you have there that you would allow multiple monitors per node. Is that correct or am I mis-revering? So the question was that there's an option to run multiple monitors on the same node, right? Yes. Yes? I mean, that's for development purposes only. There are certain people which probably run Rook with just one node and monitor count free. So. Development purposes only. Yes. It's officially a development purpose only, yes. The second question is about the monitors failover. You're saying that if a monitor dies, it will be failed over to somewhere else. I'm fully aware that the monitors themselves are able to sync from an existing, at least as long as we still have one available monitor, it will be able to sync. But now I'm curious, what's the storage back end for those monitors? Just your plain disk. So. Is that disk mounted to the container? Is Kubernetes itself or Rook managing to copy that data to some extent, or is it relying on the monitors synchronization mechanisms? Let me repeat the question. So the question is, for the month failover, kind of how does the data get to the other node, or does the data even get copied from the node which failed to the other node which is then the new monitor? No. It doesn't get copied. It's a brand new monitor, which, well, just a new directory which, well, what was it, like just a monhost sent here, talked to them, and as you said, then the synchronization basically kicks in for monitors, and the new monitor should be up to speed in a few seconds, I hope, at the longest. You will always need a please one monitor in life, otherwise everyone's cool. OK. Yeah. Just make sure. OK. I have a follow-up question for this. What could happen with the ports that need storage? Will they wait for the new monitor to be up, or they would be redistributed to other nodes that already have the storage? So the question is that if a monitor fails, what will kind of happen to the ports than the juice storage? So from self-sight, the monitor list is always kind of, well, to be like fluid. So if one monitor fails, well, it's still in the list of the conflicts most of the time. So if you bring up a new monitor, and it's back in the, we're talking with the other monitors, this new MonMap is basically where the new monitors in it will be kind of distributed to you, or, well, the applications again, at least as far as I'm concerned. OK. So you are literally creating a new monitor and not just reusing the same ID and ID. Yes. We're creating a whole new. We basically disfriended the special palet or maybe. Another question is a big question. Yeah. So you showed us how to deploy storage, but with software defined storage, the network layer, like find, find, public, cluster, network. OK. So the question is, how does the network layer, or, well, how does the network kind of, well, what would be the network layer if you, for example, then put self on it and you have, like, 5,000 other applications running in the cluster, right? Kind of? Yeah. Like, how do you find, find, public, cluster, networks? OK. So to kind of extend on what I just said on the fourth question, it's going into the direction of separating the self traffic, the replication traffic, and the client traffic. Right now, you would go and use host network mode, which, well, currently Kubernetes state is that you only have one interface per pod. Well, if you use host network, you get a nodes network which has other advantages and or is advantages, so to say, like this. But at least to go a bit further into the CNI pod, there for Kubernetes, there are multiple, I think, two or three projects right now, which allow you to define multiple CNI plugins in your, in the config somewhere, and then have multiple interfaces in your pods, which I think Intel, at least, well, I heard it from Intel at like container days where they showed it off for like, vibe stuff and so on. And for them, it seemed to work pretty good. But right now, host network would be the way to go. But yeah. Go. OK. Let's give it up. Thank you.
 Okay. Welcome everybody to our first roof talk and welcome, Alexander the roof maintainer.  Yeah. So my name is Alexander Trost. I'm a well, roof maintainer. I'm just a side of that. I'm a DevOps engineer at cloud ability.  Today I'm kind of going over using roof, yeah, using roof in Kubernetes to run SAP in Kubernetes and kind of going into like what are the advantages of doing that.  And you have from a genocide kind of like what is through a bit of the architecture of roof a bit into like the integration with Kubernetes that well, in the end, most of the time you want to use the self storage you're running with roof inside of your Kubernetes cluster for your applications.  Let's say I don't know like it my skill database or something. And then well, the advantage is kind of like how can it help you doing that.  And one part which probably falls flat due to technical issues on with my laptop.  Well, I'll see if I can get some information on some video or something up that we kind of get a demo of the creation of a cluster and also how easy it is done in Kubernetes to consume the storage there.  And while you're adding and removing a new note to the cluster to the self cluster in the end, well, I have to see.  Yeah, and in the end, just like a wire route, a few more points there and yeah.  So, Rook is a cloud native storage or a straight up cloud native.  Everybody talks about it's it's well, those this cool cloud thing, you know,  it's specifically made for being run in a Kubernetes there.  Yeah, which simply implies that it's in the container.  So it's basically a cloud native container search orchestrator.  And through this through this orchestration.  Rook well can run, for example, safe, but also other storage softwares, which I will come back later on.  For you in your Kubernetes cluster, the special point which yeah,  Kubernetes allows through the extensibility of the Kubernetes API is that you have the possibility of create your own custom types and also well in the end,  you have the control us which react to these custom types in Kubernetes.  More on it later and in general the goal here is that for the orchestration part,  we try to have the deployment automated as best as possible bootstrapping of the software of the for example,  in this case, a self cluster, the configuration provisioning, scaling, upgrading,  migration disaster recovery monitoring and resource management.  All points which we try to cover not all of them.  Those points are covered yet, but well, it's open source.  We are happy to see people help us reach the goal of full automation of your,  for example, self cluster in Kubernetes.  As I already said a bit earlier,  Rook is not only self, self is one of the many storage providers that Rook can run for you.  For example, MinIO, which is an object storage, maybe you've heard about it,  then there's also from an excenter guys, it was HFS,  it's well kind of in the direction of like self,  which provides block storage, file storage and also object storage,  but in a more like, well, I think they have a huge point in like,  geo replication by default kind of in it, definitely worth to check it out.  And some other tool which I have laid on a slide, which are currently on in Rook.  It's well, we had for some, it's open source.  It's especially what is good for us,  it's hosted by the Cloud Native Computing Foundation.  It's currently in the incubating state, incubator state of the CNCF project level.  Yeah.  Rook framework.  As Rook is not only there for just running self,  but also for MinIO, HFS and so on,  Rook tries to get generalization of certain types  which are common across those multiple storage back ends.  So for example, to say it with for example, yeah,  self and HFS, the selection of the devices on nodes is the same.  There are no two types which are different.  This is one type where we basically have a list with some more details  which we go in later too, of which devices to use.  And only already with those common specifications  back used between multiple storage providers,  it's most of the time pretty easy to, well,  not that I would recommend it too much to instead of like,  distro hopping as you storage software hop all the time,  but in the end something like that would be possible  because you could just copy and paste the device list from,  well, self for example, then to HFS or well, back again.  Yeah.  With the framework part of Rook there,  we not only try to have this common ground of specifications,  we simply also try to have combined testing efforts  that makes it easier for all those platforms  which have their own operator in Rook to be,  well, tested together, have policies and stuff,  just share the code that is not like every operator  which for example would run something like self,  like HFS and other possible software backends,  software storage backends,  that they don't need to write that from scratch all the time.  So that's one of the goals with the Rook framework basically there.  Yeah.  The architecture of Rook,  it's going a bit into Kubernetes right now here as,  we're kind of a general look right now here.  We have the Kubernetes API,  we have our client utility which we can use to talk with the  Kubernetes API and Kubernetes uses Etsy for the data store  of the API objects.  And if we basically start from here,  we have our client utility with which we tell the Kubernetes API,  we tell them about these new objects.  So that in the end we end up with objects like  self cluster, self block pool, self object service,  self file system and well for the other operators,  if you install them too, obviously those two.  When we have created those objects,  we can access them again, just run it like for those knowing about  kubectl like kubectl gets self cluster and we get the object  as it is currently in the API and in more common,  more current releases of Kubernetes,  I think with one something 12, 13 or so,  you can get some more information about the object as it is,  if you simply run the kubectl gets to,  well retrieve the object from the Kubernetes API  so that you have a quicker overview of if it has been created  successfully, how many monitors am I running and just some  general information.  Moving on to the operators, as said with the multiple storage  back ends, for example we have the self operator there then,  with minni operator and so on for each storage back end  to own operator and the operator basically takes care of  creating the deployments, the deum set and all that is needed  to run the storage software.  So for self, it would be creating deployments for the monitors,  creating deployments for the OSDs and all other components  like manager, MDS, RGB and then also for example  configure certain aspects in the manager like enable the  dashboard and for example enable SSL on it,  disable SSL on it, stuff like that.  And it's making it simple by,  this is possible again because we have one common object,  the self cluster object here.  With this one object with which the root operator knows how to  handle it and has all the options we support in it,  which you can just go ahead and add it for not all options  right now, but for a good amount of options you can go ahead  like toggle the dashboard on the fly, if you have it on  false when you create a decluster initially and just go  edit it, put through and apply it against the Kubernetes  API and the root operator will pick up this change and do its  thing and enable the dashboard.  Now moving on to this part here,  this is basically like we are on the node right now.  The demons are obviously for let's say for example  SAF, we have a monitor in OSD or one of the SAF demons  which is placed on a node, if a mon is placed and do you want  to use the disks on a node also for storage you would  obviously also have OSD demons running on that.  But all containerized so there's no like,  well conflict with other stuff running on the system,  it's well container so well there are certain differences  between which container runtime you use from the  isolation level but they are isolated at least.  So the container part there is also the point where  the upgrading for RUG is also making it easier because  if we just go ahead in the deployment which for example  run in OSD there, we just go ahead to the Kubernetes API  and say hey, change the image from SAF version 12  something to version 13 something.  And for the other you basically the same update aspect  comes in place too because Kubernetes does this  management for us.  Then there's also the RUG agent.  The RUG agent is in our current case the so-called  RUG flex volume provider, plug-in provider.  This is, well flex volume is kind of like the predecessor  of CSI kind of in Kubernetes where CSI is the container  storage interface which well kind of as the name implies  like a common interface for storage not only a thing  for containers but in general for storage.  And flex volume is kind of like that too but more on  a limited scale as I know especially for Kubernetes there.  And we have it, we have it running right now still with  flex volume or just more or less to the point because  of Kubernetes with having CSI support on stable,  beta or stable with version 113.  And we're currently separating like the last three  versions of Kubernetes so we still have to have the  flex volume.  But to the flex volume here if I have a pot which wants  to use a volume on this node it would,  the Q-Blood, the node component of Kubernetes would call out  to a socket on a node of flex volume socket and then the  RUG agent would receive this request and then well go  ahead and get the information of the volume and do the  RBD mapping and then the RBD mounting.  One quick note, the Q-Blood I think is in most cases doing  the formatting of the volume.  So, yeah, I think the formatting is done by the Q-Blood  which, yeah?  I'm not sure if you're going to tell us more about this or not,  but for instance OSDs on SAP, if you need access to the  block devices, is this accomplished with flex volume  or will it be accomplished with the new CSI stuff?  I'm trying to answer questions.  So, you mean the block device, if I mount the map the volume?  No, so the OSDs themselves, they are backed by the disks  where they write the objects and all that stuff.  So, the containers will need to have those block devices  exposed to the parent, right?  Okay, so if I understand correctly the question is that  how are the OSDs able or getting the access to the block devices?  Well, right now it is simply done by mounting the  slash depth of the host into the container.  There are ways, I think, especially through, what was it,  local storage, local storage from Kubernetes which has been  introduced, I think, like 1.9.1.1.1.1.1.1, I think,  which could potentially be used for this exactly purpose.  We are still kind of looking into how we can make it,  use it like this, but we're kind of like right now  a slash depth is basically working for us.  But we are also already at least looking into restricting  the slash depth access to just the device and or the  partitions that are on this disk than 40 SAP OSDs.  There's still a good amount of work to do to get there.  Yeah.  Thank you.  So, yeah.  So, if I create the RookSafe operator and then go ahead  and create a RookSafe cluster object too, well, I have,  as already said in the previous example here,  we have the agent pod which is running in all nodes  and the operator then, as said, takes care of creating  the SAP component pods like the OSDs, the monitors,  manager, MDS, RGV is also there.  And, well, in the end, you have a SAP cluster with that.  Yeah.  So, to summarize this part, we have this custom object  with kind SAP cluster which Rook operator,  the RookSafe operator has defined.  You create it and what you currently with that would get is,  now, we're a SAP cluster running SAP version 13.2.4  something, something.  And ignoring your data, the, well,  data, the host path in Rook is right now where most  config is stored and monitor data, but we're working on  trying to, well, make it really just for configuration,  maybe still one data but not OSDs.  Because right now the behavior would be that if you don't  specify like disks in anything else, you would get an OSD  using a directory of, well, basically a file,  directory OSD which, well, if you have disks from  performance devices, way better.  The disk one is way better.  Yeah.  But coming back here, we have like the dashboard part here  where we say, yeah, we want the dashboard.  So we have enabled through.  We can control a bit about the monitors like how many do we want.  If we want multiple monitors on one node, this is if you have,  well, only three nodes, you may or may not want this to happen.  Or if you have multiple nodes, you normally want to disable  those modern three nodes, for example, because then you  always have room to move a monitor from a node which  failed to another node and, well, create a new monitor there,  basically.  This part with the storage here is here, for example,  use all nodes, use all devices is kind of like controlling  there's more to control.  But those two options are kind of allowing you to select where  storage will be used and what storage will be used.  With the use all nodes, it's kind of, well, use all nodes  that the Rook operator can place OSDs on it, which are  pretty like that.  Valid in Kubernetes aspect.  Use all nodes with implied that all empty devices will be  used for an OSD.  And if we, I didn't upload it last time.  If we go, if we skip a few slides ahead, we come back again  to this storage configuration part where we have, again,  those use all nodes and use all devices.  Instead of, for example, having use all nodes true that all  nodes which are applicable will be used, we could also  specify a list of nodes which will have information on the  next slide.  We can also have a simple device filter where we go, yeah,  use all this with SD and, well, Wildcard, basically,  and then Rook will take all disks with this matching  ragex besides disks that are not empty.  There's also possibility to specify certain configuration  parameters, not only in a cluster-wide level, but also  on a node and even OSD part level, where in this case,  for example, we have, like, an option which normally is only  used for NVMe or, well, faster than SSDs devices is the,  how many OSDs should be created per device.  If we go just one further, we are at the nodes list here,  and there you can see that, for example, you have this node,  you say, well, use a directory, have those resource limits  in place from Kubernetes site that contain the OSD container  for this node as it's on the node level here,  we'll have those resource limits in place.  Then, for example, here with a device, you have the node name  and, in this case, specifically say, hey, use SDB,  but you can also, again, if it would be, for example, NVMe,  as it's the second device here, you could then specify,  like, config and then OSDs per device, like, free,  or, well, insert a math for NVMe devices,  how many OSDs here, basically.  There we even have it, OSDs per device,  and we just have it per device and our node level  on cluster-wide level, so that's not all.  Just as we find where we left off here.  Yeah.  To shortly go into the further Kubernetes-native integration  part, besides the so-called customer-assource definition,  those custom-types objects we can create,  we already have in Kubernetes storage classes,  which allow us, the admin most of the time,  to specify which provisioner and certain parameters  for this provisioner to be called,  the provisioner will be called if you create a volume now,  a persistent volume claim.  If it matches this storage class, if it has this storage  class in it, Kubernetes will take care of talking  with the provisioner and saying, hey, somebody requested  20 gigs of storage, for example, if you go for one further.  Here, just example with 20 gigabytes of storage,  and here we specify the storage class names,  or especially use the one we just saw.  If we create this, this will result in 20 gigabytes of storage.  That's kind of where Kubernetes takes over in point of storage  management, as Kubernetes has all the persistent volume  claims, which I hope for everyone using them,  always reflect the claims an application has made,  like, hey, I need 500 gigs of storage.  So it's always this application is using 500 gigs of storage.  And in the background, it's kind of like a not on Perl  application level, you have the persistent volumes,  which are, again, also having the information,  like, which storage class has been used to create this volume,  how much storage is this actually, what access mode is used,  like, Kubernetes has three access modes,  where the first is read, write, once, which is, well,  only one can read and write to it.  The second is read, write, many, where, well, many can read  and write at any time at the same time.  And there is reads only many, which, well,  many can read only at the same time.  This way here.  Yeah.  So what can rule kind of do to make it, well,  if you have a self-class or especially because, well,  you need to run a self-class, make it better for you  to, well, run a self-class in Kubernetes,  especially, as I said, if you have one,  if you don't have one and you're like, yeah, let's go,  it might be a bit problematic because you need to know  a bit of stuff with Kubernetes.  It's, well, the simple fact that if you start one higher  with a platform and then put it on, it's, you should have  knowledge about a platform.  Part one, what Rook does is, as we heard,  health checking for monitors.  So if you have, let's say, five nodes and you currently,  just for simple example reasons, have your monitors one  on the first node, second on the second, and third on the third,  and the third node would fail.  It would move this monitor, fail it in the self-cluster,  and move it over to, let's say, node four or five,  which are available for that.  The simple management, simply through the Kubernetes API,  in a second, through the Kubernetes API in point that you  have those YAML manifests, kind of, say and put it like this,  with infrastructure as a code kind of.  It's, yeah, more and more, well, not only infrastructure  as a code, but kind of application deployment as a code,  depending on how far you see it.  You have those YAML files which define how your self-cluster  should look, which pools should be created.  You just have a YAML file which has here a pool object  with these and these names, how many replications,  should you use the ratio code, and should it, which failure  domain, which, as a YAML object is, well, you just,  you've got to create it to the Kubernetes API,  and you have it.  You have your pool created in a few seconds,  and it makes it easier on that side to, well,  manage those things, too.  And files, for example, same here, too.  It's not like, well, I want you to use file system.  Let me buy five servers and install seven,  well, set up the, yes.  With that, with RUG, well, you would need to put  Kubernetes on it, and then just create a file system object  which has size numbers and everything,  like for the pools, again, for this file system,  what else are the numbers like, how many active  MDS do you want, should there be a standby MDS to,  and just gives you a bit of, well, playground,  not to say like about a good amount of room  to have certain options and have that automatically happen.  Same for RGV.  You have an object, control it through that,  have certain options to control the behavior of the pools  or how they should look.  Yeah.  And the third point I would like to bring up here is that,  as we saw with the storage selection, again, there,  you have a self-cluster object.  You have either a list of nodes or a more generalized  device filter or something where you say,  use those disks in the folder cluster  or even directories right now, which just as a side note  for the storage selection, we have Travis Nielsen,  which is also working on the Rook project,  looking into enabling LVM kind of with the self-volume,  the self-volume part, to be created, well, through self-code,  well, no, self-code, and not as it was done  since the release of 0.9.  Now it's done not for all cases like that anymore,  for Rook's own, well, it was the preparation,  partitioning, formatting, and so on, process.  Yeah, well, we already went through that.  Well, the Y-Rook part is kind of covered by what you would get  as benefits if you have a Kubernetes cluster for something  like that, but so, in general, it simplifies certain tasks  for the storage back end.  There are assets, still some work, like, for example,  replacing a node right now.  Still a bit of a manual process, but we're getting there.  It's more or less a bit of a clash, like,  should we listen on the nodes that are in the cluster  and then react if one goes missing, for example,  or is already just not ready, or just some points where,  for whatever reason, maybe an SD issue or something,  there's an empty node, so we can go,  ah, nothing to run, so just grab it all, kind of.  But, well, as we're happy to just, well,  see people maybe just chime in on the discussions  on those topics.  Yeah, self-managing part is kind of that it's,  well, we don't have lifeless props and readiness props  for all self-components, but we think we have it,  at least for the manager right now, and for the monitor,  we do kind of external half-checking from the operator  side to see, is this monitor still in form,  or do we need to fail it over to a new node?  Yeah.  And the part with the dynamic provisioning there  that I create a claim, and a few seconds later,  my block volume in-serve has been created  and linked with Kubernetes through the persistent volume  object is, well, it's a breeze to,  if you want to run with applications,  where you don't need to, like, well, fill in a ticket  and say, hey, I need 20 gigs of storage,  can you please provide it to me?  And someone starts running through the data center  and tries to find the data, they just go for something for you.  But it's simply dynamic there, which is pretty good for,  well, especially for developers.  We are also interested with kind of like with the guys  from cross-plane I.O., Jared Watson,  Basam Tabara, if I got his name correctly.  We're looking into also trying to get more abstraction in,  like, for example, and a developer would say,  yeah, I need a bucket, so it just creates a bucket object.  And for example, we also have support for CockroachDB.  So we would obviously, as a database,  also want to look into having a database object  which the developer would just create and then in some way,  which is kind of still up for discussion,  should it be a service broker or should we do some magic,  like, with Kubernetes, well, mounting a directory into  then the part which contains the credentials,  for example, then to this database or so.  Again, there, if you want to kind of jam in there,  we're happy to have discussion on that.  The last point here in this list is, well,  it's depending on how you see it.  One problem, I think, which is still existing for AWS,  with the there, what was it?  The EBS block storage from Amazon.  I think what was it?  The failover time was like, when the node failed,  it would take, I think, like five minutes or so  to failover to another node.  And, well, I hope you're only running cloud native applications  or there shouldn't be a problem as you have more than one replica.  But it's still kind of a not too good number as you use,  even though it's cloud native and it should be okay  with a failure of a part or even multiple parts in your cluster  or multiple applications.  It's still kind of a bummer to have, like,  wait five minutes in the case of a failure of a part.  So that's kind of where I'm kind of saying it's  the last vendor login.  Because, well, you would run a self-class,  which, yeah, on the one hand, could also create more maintenance  instead of, for example, taking care of just the EBS volumes  of Amazon.  But the failover is, in this case, for example,  faster or you just don't have to use their storage.  Yeah.  Yeah.  As kind of, yeah, said in the beginning, it's not only self.  There's not only a root-self operator.  There's also a root-mini operator, which, well,  is object-searcher. We have a minio-object-search object.  We have also Cockroach-Gb, which, well,  right now we only have a Cockroach-Gb object.  But, again, we are open for people  communicating and discussing with us how we can go to best path  if it's more of a service broker approach or not approach.  HFS has been implemented by the next vendor guys.  Just there again, also a big shout-out to them,  as it's a huge work. It's not like a minio.  We have two or three replicas of onus,  where it's full-set only.  But there's, again, kind of like self, more complexity  simply behind it.  And it's amazing to see them have implemented it in Rook.  And so they're using certain parts of their Rook framework.  And an FS server has minio and Cockroach-Gb.  And, well, HFS too implemented thanks to the community.  So, yeah, well, I don't have to demo right now.  Let me get to this page first.  If you want to get involved, we are on GitHub.  I think some people are still on GitHub, right?  Nothing to do with GitHub.  We have a page which we are currently reworking.  So if you're right now like, where do you find this and this,  we're working on it.  We know of the problem.  And, well, we have a Slack channel, a Slack,  no, a whole Slack for our own.  Yeah, if you want to join, we also have a conference channel.  And, well, if you have questions, a channel, a channel,  and so on.  Well, Twitter, we still have a mailing list, I think.  And there's community meetings every second week.  Yeah.  I would just jump to the questions right now.  OK, so two questions.  On one of your slides you have there that you would allow  multiple monitors per node.  Is that correct or am I mis-revering?  So the question was that there's an option to run multiple  monitors on the same node, right?  Yes.  Yes?  I mean, that's for development purposes only.  There are certain people which probably run Rook with just one  node and monitor count free.  So.  Development purposes only.  Yes.  It's officially a development purpose only, yes.  The second question is about the monitors failover.  You're saying that if a monitor dies, it will be failed  over to somewhere else.  I'm fully aware that the monitors themselves are able to sync  from an existing, at least as long as we still have one  available monitor, it will be able to sync.  But now I'm curious, what's the storage back end for those  monitors?  Just your plain disk.  So.  Is that disk mounted to the container?  Is Kubernetes itself or Rook managing to copy that data  to some extent, or is it relying on the monitors  synchronization mechanisms?  Let me repeat the question.  So the question is, for the month failover, kind of how does  the data get to the other node, or does the data even get  copied from the node which failed to the other node which  is then the new monitor?  No.  It doesn't get copied.  It's a brand new monitor, which, well, just a new directory  which, well, what was it, like just a monhost sent here,  talked to them, and as you said, then the synchronization  basically kicks in for monitors, and the new monitor should  be up to speed in a few seconds, I hope, at the longest.  You will always need a please one monitor in life,  otherwise everyone's cool.  OK.  Yeah.  Just make sure.  OK.  I have a follow-up question for this.  What could happen with the ports that need storage?  Will they wait for the new monitor to be up,  or they would be redistributed to other nodes that already  have the storage?  So the question is that if a monitor fails, what will kind  of happen to the ports than the juice storage?  So from self-sight, the monitor list is always kind of,  well, to be like fluid.  So if one monitor fails, well, it's still in the list of the  conflicts most of the time.  So if you bring up a new monitor, and it's back in the,  we're talking with the other monitors, this new MonMap is  basically where the new monitors in it will be kind of  distributed to you, or, well, the applications again,  at least as far as I'm concerned.  OK.  So you are literally creating a new monitor and not just  reusing the same ID and ID.  Yes.  We're creating a whole new.  We basically disfriended the special palet or maybe.  Another question is a big question.  Yeah.  So you showed us how to deploy storage, but with software  defined storage, the network layer, like find,  find, public, cluster, network.  OK.  So the question is, how does the network layer,  or, well, how does the network kind of, well,  what would be the network layer if you, for example,  then put self on it and you have, like,  5,000 other applications running in the cluster, right?  Kind of?  Yeah.  Like, how do you find, find, public, cluster,  networks?  OK.  So to kind of extend on what I just said on the fourth question,  it's going into the direction of separating the self  traffic, the replication traffic, and the client traffic.  Right now, you would go and use host network mode, which,  well, currently Kubernetes state is that you only have one  interface per pod.  Well, if you use host network, you get a nodes network  which has other advantages and or is advantages,  so to say, like this.  But at least to go a bit further into the CNI pod,  there for Kubernetes, there are multiple, I think,  two or three projects right now, which allow you to define  multiple CNI plugins in your, in the config somewhere,  and then have multiple interfaces in your pods,  which I think Intel, at least, well,  I heard it from Intel at like container days where they showed  it off for like, vibe stuff and so on.  And for them, it seemed to work pretty good.  But right now, host network would be the way to go.  But yeah.  Go.  OK.  Let's give it up.  Thank you.
Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_0_155.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_156_1091.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_1092_2951.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_2952_3011.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_3012_5735.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_5736_7511.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_7512_10499.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_10500_10595.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_10596_14183.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_14184_14231.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_14232_14279.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_14280_23039.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_23040_23087.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_23088_23411.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_23412_23495.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_23496_24011.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_24012_27407.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_27408_27455.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_27456_27503.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_27504_27611.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_27612_29303.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_29304_30695.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_30696_30743.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_30744_30803.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_30804_30851.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_30852_31247.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_31248_32183.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_32184_32495.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_32496_32555.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_32556_34415.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_34416_34463.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_34464_34583.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_34584_40487.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_40488_40559.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_40560_47483.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_47484_49295.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_49296_49403.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_49404_49451.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_49452_50795.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_50796_59099.png Ceph_storage_with_Rook_Running_Ceph_on_Kubernetes_cfwIcHU604Y_59100_59213.png