A Google Monitoring System, Monarch… in Open Source?
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Plotka, Bartek
Ye, Ben
Recently Google published a paper on their monitoring system Monarch, which happened to have similar design choices to the existing CNCF Incubated project: Thanos! During this talk, two of Thanos maintainers will explain why Thanos could be claimed as an unintentional open source evolution of Google Monitoring Systems like Monarch. Google is known for building massive, global scale solutions. While organizations rarely have a similar scale, we can learn from those solutions while developing in Open Source. We all see the successes of Kubernetes which was inspired by Google Borg, and Prometheus based on Google Borgmon. During this talk, two of Thanos maintainers will explain why Thanos could be claimed as an unintentional open source evolution of Google Monitoring Systems like Monarch. By following this comparison, the audience will learn the general pattern of using and scaling the Thanos system. This talk will also reveal the exciting direction of the project including the map-reduce pushdown for PromQL queries for massive cost reduction (don’t tell anyone!)
 Okay, hello everyone and welcome to our talk on FOSDEM 2021, which is our first talk in the air, hopefully not the last one. We are very excited to bring you some quite unique experience, some quite unique content of going through the Google monitoring system, which is called Monarch, and comparing it with what we have in open source really. But before that, let's have a short introduction. I'm here with Ben. Hello everyone, my name is Ben. I was an intern at Red Hat Observability. I'm one of the TANOS maintainers and also contributors to many open source projects. I like reading books, listening to podcasts, and playing soccer in my spare time. Thanks Ben. My name is Bartek and I'm principal software engineer at Red Hat. I work for the Observability team and I'm co-author of the TANOS project and maintainer of Chrome Tues. I love community and working together in open source building the system, especially in Go. I'm also active in the CNCF's Seek Observability, where I'm actually there. So you're welcome to join us. Okay, let's start. I think in the beginning of the year, it's always good to step back a little bit and take a fresh look on things. For example, how monitoring is going to shape in 2021. Is there still demand for monitoring, for example? In 2020, industry was slowly replacing humans with Kubernetes operator, obviously, but still someone has to monitor those operators and things they are operating. So at the end, observability and monitoring are not going anywhere. I think what we are saying is actually opposite. The demand is constantly growing to the increased complexity of systems, including serverless applications, mentioned operators, and more automation overall. So we know monitoring is crucial. What about metric itself? Are metrics still related, kind of needed and required? Other signals exist there as well and they are irreplaceable. In fact, in 2020, alone, popularity of even-based signals like logging and tracing has grown a lot. It's also much more accessible than before with emerging lightweight cloud systems like Loki or Tempo or many, many others. Really interested in systems like open telemetry and overall signal correlations went high as well. After all of those and more, there are many voices questioning, do we still need metrics? We could calculate metrics from events. As a team member responsible for Red Hat monitoring and OpenShift, we are periodically asking the same question as it would be some complexity improvement if you could just drop one signal. But still, many sources, our own experience and customers that we have confirm the importance of the metrics. And the reason is simple. Metric systems are generally much cheaper, if not more, simpler to use and operate for monitoring purposes. This is because metrics cannot by design store all the context of everything due to cardinality reasons. So users naturally have to focus on collecting aggregated information, focus on what matters. For example, service level indicators. This limits the data you need to collect and ingest. And it's kind of compressible with time as well. Of course, it would be amazing, we could just collect every event around us, around my system operations and persist all the context around it using just logs and tracing. But as January 21, if you want to have a solid monitoring without overspending tons of money on infrastructure and engineering time, metric is your first option. To this in mind, let's just have a very fun experiment today. We agreed that we need kind of a metric system, a monitoring system, ID, scalable and cheap. And there is a common pattern in our industry in open source to build on the shoulders of giants, to not reinvent the wheel. For example, let's say Google systems. They overall have to build systems that are planet scale, super fast and super reliable. There was a kind of reason why Borgmon, all their Google monitoring system, was an inspiration for open source prometeuse system, a default monitoring metric system in the CNCF. In the same way communities is based on Borg, Google orchestration system, ZooKeeper is model after Google's Chubby system, log service, similar with protocol buffers and GRPC. There is some similar technology existing in Google as well. That was kind of inspiration for that. So is there anything else we can learn from Google on monitoring drought? That's the question. Yes, recently in summer 2020, Google published amazing paper about Monarch, a newer version of Google planet scale monitoring system that was meant to support or replace Borgmon in some cases. So the previous monitoring system. So question is, is there anything similar in open source? And we would like to put one that is loud. We believe that in many ways you can treat Thanos system as the open source Monarch. Similarly as we treat prometeuse being modeled after Borgmon. The funny thing is that we had no idea about Monarch when we were designing Thanos. But some of our ideas and decisions ended up being exactly the same or similar to Google's one. I think this confirms that there are, you know, after all not that many possible ways of solving scalability problems, especially with metrics. And before we start this claimer, we are really admiring the engineering on Google site and overall engineering experience, but we have never worked at Google. So everything we tell you here is based on the official data published by Google, especially for from research papers and Google, for example, SRE book. So can we find the Monarch in the open source world? Actually as Thanos developers, after reading the Monarch paper, we find out that the Thanos architecture and design is very similar to Monarch. Next I will go through some parts of the paper and by comparing the two systems, I will show you the similarities and the differences between them. And behind this experiment, we have one goal in our mind, which is to expand our horizons and learn what can we do better to improve the Thanos project. So let's first start from the use cases. Monarch as a monitoring system, the use cases are very common. The first one is for alerting. The second one is for displaying dashboards for services status. And the last one is for doing some ad hoc queries for debugging problems and exploration. So what about Thanos? And Thanos as an open source and highly available pre-missive setup, it definitely has the same common use cases. So what we can say is that they are both systems that allow you to monitor, debug and alert on workload state. So in the paper, we can find four main reasons why Monarch was created in the first place. Because as we know, a Borgman, a previous monitoring system at Google, was already existing at the time. Overall, the main motivations we listed here are two limitations of Borgman that are standing out. First of all, the fact that Borgman was centralized, or actually is centralized, sorry, decentralized, similar to ProPointeus actually, this means that all our critical monitoring is within single binary that is running in your cluster together with your applications. And this has a powerful advantage of having everything simple and reliable because everything is kind of together, completed in the same cluster. However, the problem is with the user experience. You need to operate that binary on yourself. You need to scale it on yourself. You need to deal with problems on flying alone as a team. So it might sound simple, but in practice, it just takes time. It consumes your resources that you could use on developing your own application. So at the time, the centralized came as a major problem. And additionally to that, it was extremely hard for people to aggregate the data on the global level. So when you have multiple clusters and multiple Borgmots, then how do you get this data on the multi-cluster level as well and alert on that level? Secondly, it was hard to scale ingestion capabilities of Borgmon. You need to kind of create two, and then how do you again have this aggregated view across those charts, how to manually scale up and down. Everything has to be manually. That was not perfect in an ideal situation. And funny enough, motivation for creating tunnels were exactly similar to those for Monarch, except instead of Borgmon, we had a Prometheus that was a little bit limited in some various scalability cases. So the same as Borgmon, in Borgmon versus Monarch, tunnels enables global view and optional centralized storage backed by object storage that was hard to achieve with Prometheus alone. With tunnels, it's also much easier to work on automatic ingestion and scraping charting mechanisms, and even allow basic use cases like multi-replicar Prometheus for, for example, a higher availability for higher availability of ingestion. Tunnels solves this problem, and that was the motivation behind that. So next, let's go through some high-level architectures between the two systems. So first, let's take a look at Monarch. This is the Monarch architecture, and you can find out that it has multi-layer architecture, whereas there are global and zone components here. So the components color with red, and they are responsible for dead ingestion. And the component color with green, and they are responsible for data query. And we will skip the blue components here because they are mainly for configurations. So for the right path, the ingested data will be routed by some global ingestion routers, and they will be routed to leaf routers in each zone. And finally, going to the leaf nodes for storage, and for the read path. So there are some root mixers, which query data in a fun-of-way. So the queries will be routed to some zone mixers, and finally going to leaf nodes to fetch some data. So there are also one index servers component, which gather some indexes, indexes information, and this is very useful for guiding distributed query execution to improve the performance. So what about tunnels architecture? So you can see here, the architecture of tunnels is very, very similar to Mono. It also has multi-layer architecture by having a global query and multiple zone queries. And for the write path, it has one global routing receiver for routing data requests. And the actual data is ingested at the zone receiver level. And for the query path, the global query will do some final query, and it will hit the receivers in the end. So for the data storage, the most recent data will be stored in memory at the receiver component, and the long-term data will be uploaded to the object storage. So tunnels provides a stock gateway component for their purposes. So that's the design. And next, let's talk about the data model. So there are some differences here. So in Mono, it stores some time series data in schematized table, which means that you need to define the table columns before you ingest the data. So in this diagram, you can find out that it has multiple key columns and only one value column. So for the key columns, it can have two sources. One is called target schema, and it's related to the monitored entity. So you can see here, it contains a job column and a class column. So it's related to the entity itself or maybe some locations. And there's another schema called metric schema, which is used for describing the metric itself. So in this example, it's RPC service latency. So it has two key columns for the service name and the command, which is useful for describing this latency value. So let's talk about tunnels. So tunnels uses exactly the same data model as premises. So it's schema-less. So you can find out here that it contains one metric name and several labels. Also one value tag, which is float number. But actually in tunnels, we have one concept called external labels, which is very, very similar to the target schema in Mono. And it's useful for grouping all the target data together. And here in this example, it's so on which represents a location. So at the end, our conclusion for demo is that although they have some similarities, and they are different in the design. So next let's talk about examples. So Mono has some built-in supports for distribution, which is histogram, infamissives. And one distribution can have multiple buckets, and each bucket can have one example. So in this heat map diagram, there are multiple rate points, and they are examples. So when you want to debug some high latency requests or outliers, you can just click the examples here and go to the trace view for debugging, and yeah, it's very useful. So what about the tunnels ecosystem? So actually in the open metric standards, it already have examples support. And in some premises client libraries, examples are also supported as well. But for storing the data in premises itself, there are some working progress PRs right now. So it's not merged yet, but it's very close. And Grafana already have some very fancy example UIs here, which is very promising. And for tunnels, once these working progress PRs in premises are merged, we can just query the examples from premises, and the example data can be propagated via remote write protocol. So at that time, we can implement the examples APIs at the store API level in tunnels, and then the users can use examples in tunnels. Okay, going with the order of Monarch Paper deep dive, we came to the metric collection. It's definitely more complex than it sounds, but in principle, applications in Monarch have some libraries, ready libraries that are pushing metric samples in required frequency to the ingestion router, which is co-located in the same cluster. And then this ingestion router is responsible for discovery and making sure, making the decision where to forward to which zone or destination cluster forward given data. So this feels great, but in tunnels, we kind of leverage the existing from it use stack. And there is a reason for that. First of all, let's explain how it works. So in client, on client side, we can reuse existing instrumentation from use instrumentation and use essentially pull model, which is in our opinion, much, much better model for general type of infrastructure. It's much easier to ask application to expose open metrics protocol and control discovery and scrape frequency and remote right complexity from the single agent purpose that we deliver instead of having this implemented in each of the client implementations. And while this is possible on Google site, it's impossible to reproduce the same in the community because the push mechanism are just much more complex. And we are talking here about red rice, limits, buffers and all potential languages and instrumentation libraries in the open source would need to implement that. And it's just hard to implement like in one place. What about the whole world? Because of compatibility and simplicity, we chosen to leverage from it use and open metrics for client metric collection. Funny enough, when you look on the both diagrams on the left for Monach and right on tunnels and from use ecosystem, they kind of look at the end the same. They have kind of the same amount of components. And well, this is proving that whatever you choose, push and pull, they are at the end scaling in the same way. It's really about the small trade or smaller, bigger trade-offs that you are balancing between. It's worth to mention also that even with open metrics, there are possibilities to push metrics in very edge cases like serverless, that might be not edge case anymore, because it's more and more popular, but also bad jobs and stuff like that. You can totally do that, but it has to be by design minority of your requests. Now let's talk about the server side where things are even more tricky because you need to make sure the receive part is reliable and scalable. So how Monarch is doing this? So first of all, Monarch is taking the right request from the ingestion routers into leaf router and it forwards that further down into target leaves responsible for given target ranges. So everything is sharded by targets. This leaf is writing this data into memory and then recovery persistent log. And this data in memory is then used for querying as well. In the same time, it is performing a light compression, which is interesting that it shares timestamps from the same targets and also performs delta encoding. This is really similar to what Thanos is doing. We have receiver instead of leaf and generally it essentially have the forwarding capabilities. So you have that in your receiving cluster, you have receiver that can forward to the proper receiver based on the hashing configuration. And then the receiver can ingest this data and write to the local Prometheus, this DB. We raise the same code as Prometheus, which means it is stored in memory and have some write ahead log as well. At the end, this DB also encodes time series using Corilla and coding, which is a certain version of delta encoding at the end. Monok is mainly designed for storing data in memory. But as the paper mentioned, it also have some durability support. So it uses some log files for durability and a long term repository for storage. So let's take a look at the Thanos system. So in Thanos, we have a Thanos receiver. And the key difference here is that the Thanos receiver will not keep all the data in memory forever. So once the data is written to the storage, it will return to the write ahead log first. And every two hours, the data will be compacted to a block and uploaded to a remote object storage. And in practice, there's one more pattern. And we can apply this, the same pattern to Prometheus with the sidecar. So this is extremely useful because if you don't want to set up the push model, you can just add one sidecar to your Prometheus and it works perfectly well. Next, let's talk about the query language they use. They use their own query language. I will now introduce this in detail, but it's really like human readable. So it uses some like filter operators to select some columns. And it also does one join, a group by, and finally aggregate on the latency. So yeah, you can find the results here. It contains one key column for label and one label column for the latency. And in the tunnels ecosystem, tunnels just use reuses the same like from standard prom cure. So it's simple, but also powerful. It also supports aggregations and joins. And but the most important thing is that it has 100% prom cure compatibility. So it can use the awesome like monitoring mixings. So users can have like auto box, Grafana dashboards and alerting rules. So if you are interested in this, you can check out this website. And next, let's talk about the different query types. Monoc has two, like you usually have two use cases for query. One is ad hoc queries. It's very easy to understand. It's just users using some like web UIs for like debugging or exploration. Another type of query is called standing queries. They are just periodically a periodic query whose results will be like written back to the monoc system. And this is one example diagram here. The evaluator sends out sending queries periodically. And finally, the results will be written back to the storage node, which is the lead. And in tunnels, actually, if you are familiar with the promisius ecosystem, you will find out that it's exactly the same thing as recording rows and alerts. So in tunnels, the tunnels ruler does the same job as the evaluator. It queries the query and writes the results to its local TSDB. And by recently, we have this proposal to improve the scalability of ruler. And we are redesigning the ruler to have remote receiver clusters as its storage, which means the results will be written back to the receiver clusters. And after we read out the monoc paper, we find out that that's exactly the same design. Now, I hope you are ready for the last but not the least feature that monoc has. This is something I'm incredibly excited about. Estanos is literally unintentionally designed for this kind of amazing optimization. And Google actually describes query execution to be scalable thanks to the primarily this feature, which is query pushdown capabilities. How it works? Well, let's look on the diagram on the right. With such monoc architecture, let's imagine naive query that sums some metrics over time and groups them together. In a naive approach, we would need to fetch all of the samples matching from each of separate leaves nodes, then pass through the mixers and then process those on the root level. Lots of data pushed through the network, probably across different zones. This will cost a lot. Now with query pushdown, we are pushing the query evaluation deeper into the stack, actually as close as possible to the underlying data, which is amazingly enormously kind of beneficial to do. Instead of samples on the leaf node, we return kind of only evaluated results, so a sum of subset data, then the mixers has to merge that into bigger partial results and then at the root, you are merging just a couple of some results and present that to the user. Obviously, it reduces the network usage significantly, as well as allowing full or partial concurrency, which is super amazing. Worth to mention that obviously you cannot potentially use query pushdown for all kind of aggregations, like for example, top K. However, for most of the typical queries, we can do this optimization. Now, why it's amazing? Because we can apply exactly the same logic on TANOS side, where query will push down computation to underlying leaves. It's much more complex than it looks on the implementation side, because as I mentioned, it has to work with different kind of aggregations. It has to be precise, correct, consistent, and it has to work across different replications, but it's something we actively work on and you can actually read through the proposal and help us doing this. We didn't cover all the differences, for example, multi-tenancy, down sampling versus monar aggregation, replication on both sides, but at the end, we learned a lot. We learned that monar is indeed kind of, I hope you agree, similar, it has more similarities and differences between each of those systems. We also learned what we can improve in future. We could create, you know, we can improve TANOS prometeuse encoding, for example, by sharing the timestamp of the same target, which we learned. We could play a little bit with this schema-less model. It's superior for the users. It's much more flexible. However, it is kind of a problem to make sure everything works on the storage layer. That's why there would be kind of nice way, nice optimization if we would know what are the target labels, what are the metric labels. Maybe there's a good way to optimize that. And last but not the least, query pushdown and intermediate indexes would be enormous gain for query latency and efficiency, and that's our goal for 2021. Thank you for listening to us and please visit our website and also interactive tutorials if you are interested. Thank you. Thank you.
 Okay, hello everyone and welcome to our talk on FOSDEM 2021, which is our first talk in  the air, hopefully not the last one.  We are very excited to bring you some quite unique experience, some quite unique content  of going through the Google monitoring system, which is called Monarch, and comparing it  with what we have in open source really.  But before that, let's have a short introduction.  I'm here with Ben.  Hello everyone, my name is Ben.  I was an intern at Red Hat Observability.  I'm one of the TANOS maintainers and also contributors to many open source projects.  I like reading books, listening to podcasts, and playing soccer in my spare time.  Thanks Ben.  My name is Bartek and I'm principal software engineer at Red Hat.  I work for the Observability team and I'm co-author of the TANOS project and maintainer of Chrome  Tues.  I love community and working together in open source building the system, especially in  Go.  I'm also active in the CNCF's Seek Observability, where I'm actually there.  So you're welcome to join us.  Okay, let's start.  I think in the beginning of the year, it's always good to step back a little bit and  take a fresh look on things.  For example, how monitoring is going to shape in 2021.  Is there still demand for monitoring, for example?  In 2020, industry was slowly replacing humans with Kubernetes operator, obviously, but still  someone has to monitor those operators and things they are operating.  So at the end, observability and monitoring are not going anywhere.  I think what we are saying is actually opposite.  The demand is constantly growing to the increased complexity of systems, including serverless  applications, mentioned operators, and more automation overall.  So we know monitoring is crucial.  What about metric itself?  Are metrics still related, kind of needed and required?  Other signals exist there as well and they are irreplaceable.  In fact, in 2020, alone, popularity of even-based signals like logging and tracing has grown  a lot.  It's also much more accessible than before with emerging lightweight cloud systems like  Loki or Tempo or many, many others.  Really interested in systems like open telemetry and overall signal correlations went high  as well.  After all of those and more, there are many voices questioning, do we still need metrics?  We could calculate metrics from events.  As a team member responsible for Red Hat monitoring and OpenShift, we are periodically asking the  same question as it would be some complexity improvement if you could just drop one signal.  But still, many sources, our own experience and customers that we have confirm the importance  of the metrics.  And the reason is simple.  Metric systems are generally much cheaper, if not more, simpler to use and operate for  monitoring purposes.  This is because metrics cannot by design store all the context of everything due to cardinality  reasons.  So users naturally have to focus on collecting aggregated information, focus on what matters.  For example, service level indicators.  This limits the data you need to collect and ingest.  And it's kind of compressible with time as well.  Of course, it would be amazing, we could just collect every event around us, around my system  operations and persist all the context around it using just logs and tracing.  But as January 21, if you want to have a solid monitoring without overspending tons of money  on infrastructure and engineering time, metric is your first option.  To this in mind, let's just have a very fun experiment today.  We agreed that we need kind of a metric system, a monitoring system, ID, scalable and cheap.  And there is a common pattern in our industry in open source to build on the shoulders of  giants, to not reinvent the wheel.  For example, let's say Google systems.  They overall have to build systems that are planet scale, super fast and super reliable.  There was a kind of reason why Borgmon, all their Google monitoring system, was an inspiration  for open source prometeuse system, a default monitoring metric system in the CNCF.  In the same way communities is based on Borg, Google orchestration system, ZooKeeper is  model after Google's Chubby system, log service, similar with protocol buffers and GRPC.  There is some similar technology existing in Google as well.  That was kind of inspiration for that.  So is there anything else we can learn from Google on monitoring drought?  That's the question.  Yes, recently in summer 2020, Google published amazing paper about Monarch, a newer version  of Google planet scale monitoring system that was meant to support or replace Borgmon in  some cases.  So the previous monitoring system.  So question is, is there anything similar in open source?  And we would like to put one that is loud.  We believe that in many ways you can treat Thanos system as the open source Monarch.  Similarly as we treat prometeuse being modeled after Borgmon.  The funny thing is that we had no idea about Monarch when we were designing Thanos.  But some of our ideas and decisions ended up being exactly the same or similar to Google's  one.  I think this confirms that there are, you know, after all not that many possible ways  of solving scalability problems, especially with metrics.  And before we start this claimer, we are really admiring the engineering on Google site and  overall engineering experience, but we have never worked at Google.  So everything we tell you here is based on the official data published by Google, especially  for from research papers and Google, for example, SRE book.  So can we find the Monarch in the open source world?  Actually as Thanos developers, after reading the Monarch paper, we find out that the Thanos  architecture and design is very similar to Monarch.  Next I will go through some parts of the paper and by comparing the two systems, I will show  you the similarities and the differences between them.  And behind this experiment, we have one goal in our mind, which is to expand our horizons  and learn what can we do better to improve the Thanos project.  So let's first start from the use cases.  Monarch as a monitoring system, the use cases are very common.  The first one is for alerting.  The second one is for displaying dashboards for services status.  And the last one is for doing some ad hoc queries for debugging problems and exploration.  So what about Thanos?  And Thanos as an open source and highly available pre-missive setup, it definitely has the same  common use cases.  So what we can say is that they are both systems that allow you to monitor, debug and alert  on workload state.  So in the paper, we can find four main reasons why Monarch was created in the first place.  Because as we know, a Borgman, a previous monitoring system at Google, was already existing  at the time.  Overall, the main motivations we listed here are two limitations of Borgman that are standing  out.  First of all, the fact that Borgman was centralized, or actually is centralized, sorry, decentralized,  similar to ProPointeus actually, this means that all our critical monitoring is within  single binary that is running in your cluster together with your applications.  And this has a powerful advantage of having everything simple and reliable because everything  is kind of together, completed in the same cluster.  However, the problem is with the user experience.  You need to operate that binary on yourself.  You need to scale it on yourself.  You need to deal with problems on flying alone as a team.  So it might sound simple, but in practice, it just takes time.  It consumes your resources that you could use on developing your own application.  So at the time, the centralized came as a major problem.  And additionally to that, it was extremely hard for people to aggregate the data on the  global level.  So when you have multiple clusters and multiple Borgmots, then how do you get this data on  the multi-cluster level as well and alert on that level?  Secondly, it was hard to scale ingestion capabilities of Borgmon.  You need to kind of create two, and then how do you again have this aggregated view across  those charts, how to manually scale up and down.  Everything has to be manually.  That was not perfect in an ideal situation.  And funny enough, motivation for creating tunnels were exactly similar to those for  Monarch, except instead of Borgmon, we had a Prometheus that was a little bit limited  in some various scalability cases.  So the same as Borgmon, in Borgmon versus Monarch, tunnels enables global view and optional  centralized storage backed by object storage that was hard to achieve with Prometheus alone.  With tunnels, it's also much easier to work on automatic ingestion and scraping charting  mechanisms, and even allow basic use cases like multi-replicar Prometheus for, for example,  a higher availability for higher availability of ingestion.  Tunnels solves this problem, and that was the motivation behind that.  So next, let's go through some high-level architectures between the two systems.  So first, let's take a look at Monarch.  This is the Monarch architecture, and you can find out that it has multi-layer architecture,  whereas there are global and zone components here.  So the components color with red, and they are responsible for dead ingestion.  And the component color with green, and they are responsible for data query.  And we will skip the blue components here because they are mainly for configurations.  So for the right path, the ingested data will be routed by some global ingestion routers,  and they will be routed to leaf routers in each zone.  And finally, going to the leaf nodes for storage, and for the read path.  So there are some root mixers, which query data in a fun-of-way.  So the queries will be routed to some zone mixers, and finally going to leaf nodes to  fetch some data.  So there are also one index servers component, which gather some indexes, indexes information,  and this is very useful for guiding distributed query execution to improve the performance.  So what about tunnels architecture?  So you can see here, the architecture of tunnels is very, very similar to Mono.  It also has multi-layer architecture by having a global query and multiple zone queries.  And for the write path, it has one global routing receiver for routing data requests.  And the actual data is ingested at the zone receiver level.  And for the query path, the global query will do some final query, and it will hit the receivers  in the end.  So for the data storage, the most recent data will be stored in memory at the receiver  component, and the long-term data will be uploaded to the object storage.  So tunnels provides a stock gateway component for their purposes.  So that's the design.  And next, let's talk about the data model.  So there are some differences here.  So in Mono, it stores some time series data in schematized table, which means that you  need to define the table columns before you ingest the data.  So in this diagram, you can find out that it has multiple key columns and only one value  column.  So for the key columns, it can have two sources.  One is called target schema, and it's related to the monitored entity.  So you can see here, it contains a job column and a class column.  So it's related to the entity itself or maybe some locations.  And there's another schema called metric schema, which is used for describing the metric itself.  So in this example, it's RPC service latency.  So it has two key columns for the service name and the command, which is useful for  describing this latency value.  So let's talk about tunnels.  So tunnels uses exactly the same data model as premises.  So it's schema-less.  So you can find out here that it contains one metric name and several labels.  Also one value tag, which is float number.  But actually in tunnels, we have one concept called external labels, which is very, very  similar to the target schema in Mono.  And it's useful for grouping all the target data together.  And here in this example, it's so on which represents a location.  So at the end, our conclusion for demo is that although they have some similarities,  and they are different in the design.  So next let's talk about examples.  So Mono has some built-in supports for distribution, which is histogram, infamissives.  And one distribution can have multiple buckets, and each bucket can have one example.  So in this heat map diagram, there are multiple rate points, and they are examples.  So when you want to debug some high latency requests or outliers, you can just click the  examples here and go to the trace view for debugging, and yeah, it's very useful.  So what about the tunnels ecosystem?  So actually in the open metric standards, it already have examples support.  And in some premises client libraries, examples are also supported as well.  But for storing the data in premises itself, there are some working progress PRs right  now.  So it's not merged yet, but it's very close.  And Grafana already have some very fancy example UIs here, which is very promising.  And for tunnels, once these working progress PRs in premises are merged, we can just query  the examples from premises, and the example data can be propagated via remote write protocol.  So at that time, we can implement the examples APIs at the store API level in tunnels, and  then the users can use examples in tunnels.  Okay, going with the order of Monarch Paper deep dive, we came to the metric collection.  It's definitely more complex than it sounds, but in principle, applications in Monarch  have some libraries, ready libraries that are pushing metric samples in required frequency  to the ingestion router, which is co-located in the same cluster.  And then this ingestion router is responsible for discovery and making sure, making the  decision where to forward to which zone or destination cluster forward given data.  So this feels great, but in tunnels, we kind of leverage the existing from it use stack.  And there is a reason for that.  First of all, let's explain how it works.  So in client, on client side, we can reuse existing instrumentation from use instrumentation  and use essentially pull model, which is in our opinion, much, much better model for general  type of infrastructure.  It's much easier to ask application to expose open metrics protocol and control discovery  and scrape frequency and remote right complexity from the single agent purpose that we deliver  instead of having this implemented in each of the client implementations.  And while this is possible on Google site, it's impossible to reproduce the same in the  community because the push mechanism are just much more complex.  And we are talking here about red rice, limits, buffers and all potential languages and instrumentation  libraries in the open source would need to implement that.  And it's just hard to implement like in one place.  What about the whole world?  Because of compatibility and simplicity, we chosen to leverage from it use and open metrics  for client metric collection.  Funny enough, when you look on the both diagrams on the left for Monach and right on tunnels  and from use ecosystem, they kind of look at the end the same.  They have kind of the same amount of components.  And well, this is proving that whatever you choose, push and pull, they are at the end  scaling in the same way.  It's really about the small trade or smaller, bigger trade-offs that you are balancing between.  It's worth to mention also that even with open metrics, there are possibilities to push  metrics in very edge cases like serverless, that might be not edge case anymore, because  it's more and more popular, but also bad jobs and stuff like that.  You can totally do that, but it has to be by design minority of your requests.  Now let's talk about the server side where things are even more tricky because you need  to make sure the receive part is reliable and scalable.  So how Monarch is doing this?  So first of all, Monarch is taking the right request from the ingestion routers into leaf  router and it forwards that further down into target leaves responsible for given target  ranges.  So everything is sharded by targets.  This leaf is writing this data into memory and then recovery persistent log.  And this data in memory is then used for querying as well.  In the same time, it is performing a light compression, which is interesting that it  shares timestamps from the same targets and also performs delta encoding.  This is really similar to what Thanos is doing.  We have receiver instead of leaf and generally it essentially have the forwarding capabilities.  So you have that in your receiving cluster, you have receiver that can forward to the  proper receiver based on the hashing configuration.  And then the receiver can ingest this data and write to the local Prometheus, this DB.  We raise the same code as Prometheus, which means it is stored in memory and have some  write ahead log as well.  At the end, this DB also encodes time series using Corilla and coding, which is a certain  version of delta encoding at the end.  Monok is mainly designed for storing data in memory.  But as the paper mentioned, it also have some durability support.  So it uses some log files for durability and a long term repository for storage.  So let's take a look at the Thanos system.  So in Thanos, we have a Thanos receiver.  And the key difference here is that the Thanos receiver will not keep all the data in memory  forever.  So once the data is written to the storage, it will return to the write ahead log first.  And every two hours, the data will be compacted to a block and uploaded to a remote object  storage.  And in practice, there's one more pattern.  And we can apply this, the same pattern to Prometheus with the sidecar.  So this is extremely useful because if you don't want to set up the push model, you can  just add one sidecar to your Prometheus and it works perfectly well.  Next, let's talk about the query language they use.  They use their own query language.  I will now introduce this in detail, but it's really like human readable.  So it uses some like filter operators to select some columns.  And it also does one join, a group by, and finally aggregate on the latency.  So yeah, you can find the results here.  It contains one key column for label and one label column for the latency.  And in the tunnels ecosystem, tunnels just use reuses the same like from standard prom  cure.  So it's simple, but also powerful.  It also supports aggregations and joins.  And but the most important thing is that it has 100% prom cure compatibility.  So it can use the awesome like monitoring mixings.  So users can have like auto box, Grafana dashboards and alerting rules.  So if you are interested in this, you can check out this website.  And next, let's talk about the different query types.  Monoc has two, like you usually have two use cases for query.  One is ad hoc queries.  It's very easy to understand.  It's just users using some like web UIs for like debugging or exploration.  Another type of query is called standing queries.  They are just periodically a periodic query whose results will be like written back to  the monoc system.  And this is one example diagram here.  The evaluator sends out sending queries periodically.  And finally, the results will be written back to the storage node, which is the lead.  And in tunnels, actually, if you are familiar with the promisius ecosystem, you will find  out that it's exactly the same thing as recording rows and alerts.  So in tunnels, the tunnels ruler does the same job as the evaluator.  It queries the query and writes the results to its local TSDB.  And by recently, we have this proposal to improve the scalability of ruler.  And we are redesigning the ruler to have remote receiver clusters as its storage, which means  the results will be written back to the receiver clusters.  And after we read out the monoc paper, we find out that that's exactly the same design.  Now, I hope you are ready for the last but not the least feature that monoc has.  This is something I'm incredibly excited about.  Estanos is literally unintentionally designed for this kind of amazing optimization.  And Google actually describes query execution to be scalable thanks to the primarily this  feature, which is query pushdown capabilities.  How it works?  Well, let's look on the diagram on the right.  With such monoc architecture, let's imagine naive query that sums some metrics over time  and groups them together.  In a naive approach, we would need to fetch all of the samples matching from each of separate  leaves nodes, then pass through the mixers and then process those on the root level.  Lots of data pushed through the network, probably across different zones.  This will cost a lot.  Now with query pushdown, we are pushing the query evaluation deeper into the stack, actually  as close as possible to the underlying data, which is amazingly enormously kind of beneficial  to do.  Instead of samples on the leaf node, we return kind of only evaluated results, so a sum of  subset data, then the mixers has to merge that into bigger partial results and then  at the root, you are merging just a couple of some results and present that to the user.  Obviously, it reduces the network usage significantly, as well as allowing full or partial concurrency,  which is super amazing.  Worth to mention that obviously you cannot potentially use query pushdown for all kind  of aggregations, like for example, top K. However, for most of the typical queries, we can do  this optimization.  Now, why it's amazing?  Because we can apply exactly the same logic on TANOS side, where query will push down  computation to underlying leaves.  It's much more complex than it looks on the implementation side, because as I mentioned,  it has to work with different kind of aggregations.  It has to be precise, correct, consistent, and it has to work across different replications,  but it's something we actively work on and you can actually read through the proposal  and help us doing this.  We didn't cover all the differences, for example, multi-tenancy, down sampling versus monar  aggregation, replication on both sides, but at the end, we learned a lot.  We learned that monar is indeed kind of, I hope you agree, similar, it has more similarities  and differences between each of those systems.  We also learned what we can improve in future.  We could create, you know, we can improve TANOS prometeuse encoding, for example, by sharing  the timestamp of the same target, which we learned.  We could play a little bit with this schema-less model.  It's superior for the users.  It's much more flexible.  However, it is kind of a problem to make sure everything works on the storage layer.  That's why there would be kind of nice way, nice optimization if we would know what are  the target labels, what are the metric labels.  Maybe there's a good way to optimize that.  And last but not the least, query pushdown and intermediate indexes would be enormous  gain for query latency and efficiency, and that's our goal for 2021.  Thank you for listening to us and please visit our website and also interactive tutorials  if you are interested.  Thank you.  Thank you.
monarch_open_source_reimplementation_0_131.png monarch_open_source_reimplementation_132_803.png monarch_open_source_reimplementation_804_2063.png monarch_open_source_reimplementation_2064_6743.png monarch_open_source_reimplementation_6744_7379.png monarch_open_source_reimplementation_7380_7463.png monarch_open_source_reimplementation_7464_7511.png monarch_open_source_reimplementation_7512_8123.png monarch_open_source_reimplementation_8124_8243.png monarch_open_source_reimplementation_8244_8495.png monarch_open_source_reimplementation_8496_10631.png monarch_open_source_reimplementation_10632_10703.png monarch_open_source_reimplementation_10704_10775.png monarch_open_source_reimplementation_10776_10931.png monarch_open_source_reimplementation_10932_11003.png monarch_open_source_reimplementation_11004_11123.png monarch_open_source_reimplementation_11124_11183.png monarch_open_source_reimplementation_11184_11231.png monarch_open_source_reimplementation_11232_11279.png monarch_open_source_reimplementation_11280_11327.png monarch_open_source_reimplementation_11328_11699.png monarch_open_source_reimplementation_11700_11759.png monarch_open_source_reimplementation_11760_11819.png monarch_open_source_reimplementation_11820_11879.png monarch_open_source_reimplementation_11880_11975.png monarch_open_source_reimplementation_11976_12323.png monarch_open_source_reimplementation_12324_12839.png monarch_open_source_reimplementation_12840_17099.png monarch_open_source_reimplementation_17100_20867.png monarch_open_source_reimplementation_20868_23927.png monarch_open_source_reimplementation_23928_26855.png monarch_open_source_reimplementation_26856_30863.png monarch_open_source_reimplementation_30864_33311.png monarch_open_source_reimplementation_33312_35219.png monarch_open_source_reimplementation_35220_37043.png monarch_open_source_reimplementation_37044_39419.png monarch_open_source_reimplementation_39420_43487.png monarch_open_source_reimplementation_43488_43595.png monarch_open_source_reimplementation_43596_44711.png monarch_open_source_reimplementation_44712_44891.png monarch_open_source_reimplementation_44892_44939.png monarch_open_source_reimplementation_44940_45055.png