Polyglot Cloud Native Debugger: Going Beyond APM
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Almog, Shai
It's 2022 and we still use logs to debug production issues? All the unit tests in the world, the largest QA team still can’t stop bugs from slithering into production. With a distributed microservice architecture debugging becomes much harder. Especially across language & machine boundaries. APMs/Logs have limits. There’s a new generation of tools in town… Production bugs are the WORST bugs. They got through unit tests, integration tests, QA and staging… They are the spores of software engineering. Yet the only tools most of us use to attack that vermin is quaint little log files and APMs. We cross our fingers and put on the Sherlock Holmes hat hoping that maybe that bug has somehow made it into the log… When it isn’t there our only remedy is guesswork of more logging (which bogs performance for everyone and makes the logs damn near unreadable). But we have no choice other than crossing our fingers and going through CI/CD again. This is 2021. There are better ways. With modern debugging tools we can follow a specific process as it goes through several different microservices and “step into” as if we were using a local debugger without interrupting the server flow. Magic is possible.
 Hello everyone, today we're going to talk about polyglot cloud native debugging and a bit about APMs. We don't have much time so I'll get right to it. But first a few things about me. I was a consultant for over a decade, I worked at San, founded a couple of companies, wrote a couple of books, I wrote a lot of open source code and currently work as a developer advocate for Lightrun. My email and Twitter accounts are listed here so feel free to write to me. I have a blog that talks about debugging and production issues at talktodaduck.dev. It would be great if you check it out and let me know what you think. I love APMs, they are absolutely wonderful. I'm old enough to remember a time where they weren't around and I'm so happy we moved past that. This is absolutely amazing. The dashboard and the details, you get this great dashboard with just everything you need. Amazing. We're truly at a golden age of monitoring. Hell, when I started we used to monitor the server by kicking it and listening to see if the hard drive was spinning properly. Today with Kubernetes, the deployment scale to such a level that we need tools like this to get some insight into production. Without an APM, we're well not blind as a bat, but it's pretty close. A lot of the issues we run into start when we notice an anomaly in the dashboard. We see a spike in failures or something that performs a bit too slow. The APM is amazing in showing those hiccups. But this is where it stops. It can tell us that a web service performed badly or failed. It can't tell us why. It can't point at a line of code. So let's stop for a second and talk about a different line. This line. On the one side we have developers. On the other side we have the ops or DevOps. This is a line we've had for a long time. It's something we drew out of necessity because when developers were given access to production, well, I don't want to be too dramatic. But when developers got access to production, it didn't end well. This was literally the situation not too long ago. Yes, we had sysadmins, but the whole process used to be a mess. That was no good. We need a better solution than this hard separation because the ops guys don't necessarily know how to solve the problems made by the developers. They know how to solve ops problems. So when a container has a problem and the DevOps don't know how to fix it, well, it starts a problematic feedback loop of test, redeploy, rinse, repeat. That isn't ideal. Monitoring tools are like the bat signal. They come up and we the developers, we're Batman, Batwoman, Batperson. All of us heroes. We step up to deal with the bugs and we're the last line of defense against their villainy. Well, we're code about people, really, kind of the same thing without the six pack abs. Code about man needs to know where the crime or bugs are happening in the code. So these dashboards, they point us towards the crime we have to fight in our system. But there's that's where things get hard. We started digging into logs trying to find the problem. The dashboard sent us into a general direction like a performance problem or higher error rates. But now we need to jump into logs and hope that we can find something there that will somehow explain the problems we're seeing. That's like going from jet engine back to Stone Age tools. There are so many problems, log processing platforms that do an amazing job processing these logs and finding the goal within them. But even then, it's a needle in the haystack. That's the good outcome with the logs already there waiting for us. But obviously, we can't have logging all over the place. Our billing will go through the roof and our performance will, it will suffer. We're stuck in this loop of add a new log, go through CI CD, which includes the QA cycle and everything. This can take hours at best. Then we produce the issue in the production server with your fingers crossed and try to analyze what went on. Hopefully you found the issue because if not, it's effectively runes repeat for the whole process. In the meantime, you still have a bug in production and developers are wasting their time. There just has to be a better way. It's 2021 and logs are the way we solve bugs in this day and age. Don't get me wrong. I love logs and today's logs are totally different from what we had even 10 years ago. But you need to know about your problem in advance for a log to work. The problem is I'm not clairvoyant. When I write code, I can tell what bugs or problems the code will have before the code is written. I'm in the same boat as you are. The bug doesn't exist yet. So I'm faced with the dilemma of whether to log something. This is a bit like the dilemma of writing comments. Does it make the code look noisy and stupid or will I find this useful at 2am when everything isn't working and I want to rip out the few strands of hair? I still have left because of this damn production problem. The buggers are amazing. They let us set breakpoints, see call stacks, inspect variables and more. If only we could do the same for production systems. But the buggers weren't designed for this. They're very insecure when debugging remotely. They can block your server while sending the bug commands remotely. A small mistake such as an expensive condition can literally destroy your server. I might be repeating an urban legend here but 20 or so years ago I heard a story about a guy who was debugging a rail system located on a cliff. He stopped at a breakpoint during debugging and the multi-million dollar hardware fell into the sea because it didn't receive the stop command. Again, I don't know if it's a true story but it's plausible. Debuggers weren't really designed for these situations. Worse, the buggers are limited to one server. If you have a cluster with multiple machines, the problem can manifest on one machine, always or might manifest on a random machine. We can't rely on pure luck. If I have multiple servers with multiple languages and platforms crossing from one to another with a debugger, well, it's possible in theory but I can't even imagine it in reality. I also want to revisit this slide because I do love having APMs and looking at their dashboard gives me that type of joy we get from seeing the result of our work plotted out as a graph. I feel there should be a German word to describe that sort of enjoyment. But here's the thing, APMs aren't one thing. The more you instrument, the more you have runtime overhead. The more you have runtime overhead, the more hosts you need to handle the same amount of tasks. The more hosts you have, the more problems you have and they become more complex. I feel Schrodinger should deliver the next line by observing we effectively change some of the outcome. An APM, some people use that as an excuse to avoid APMs, which I feel is like throwing away the baby with the bathwater. We need APMs. We can't manage at scale without them. But we need to tune them and observing everything isn't an option. Thankfully, pretty much every APM vendor knows that and they all let us tune the ratio of observability to performance so we can get a good result. Unfortunately, that means we get less data. Couple that with the reduction of logs that we need to do for the same reason and the bad problems we had having production just got a whole lot worse. Let's take the Batman metaphor all the way. We need a team up. We need some help on the servers, especially in a clustered polyglot environment where the issue appears on one container, moves on to the next, etc. Remember this slide. We need some way to get through that line. Not to remove it. We like that line. We need a way to connect with the server and debug it. Now, I'm a developer, so I try to stay away from management buzzwords. But the word for this is shift left. It essentially means we're letting the developers and QA get back some of the access we used to have into the ops without demolishing the gains we had in security and stability. We love the ops people and we need them. So this is about helping them keep everything running smoothly in production without stepping on their toes or blowing up their deployment. This let's lead us here. What if you could connect your server to a debugger agent that would make sure you don't overload the server and don't make mistakes like sending a breakpoint or something like that? That's what Continuous Observability does. Continuous Observability is complementary to the APM. It works very differently. Before we go on, I'd like to explain what's Continuous Observability. Observability is defined as the ability to understand how your systems work on the inside without shipping new code. The without shipping new code portion is key. But what's Continuous Observability? With Continuous Observability, we don't ship new code either, but we can ask questions about the code. Normal Observability works by instrumenting everything and receiving the information. With Continuous Observability, we flip that and we ask questions, then instrumentation is made based on the questions. So how does that work in practice? Each tool in this field is different. I'll explain the LATRIN architecture since that's what I'm familiar with and I'll try to qualify when it's different from other tools. In LATRIN, we use a native ID plugin to VS code or JetBrains IDE's such as IntelliJ, PyCharm, WebStorm, etc. We can also use a command line tool as well. Other tools sometimes have a web interface or CLI only. This client lets us interact with the LATRIN server. This is an important piece of the architecture that hides the actual production environment. Developers don't get access to production area, which is still the preview of DevOps. We can insert an action, which can be a log or a snapshot or a measurement metric. I'll show all of these soon enough. This talk will go into the code portions soon. Notice that the LATRIN server can be installed in the cloud or as a SAS or on-premise and managed by Ops. The management server sends everything to the agent, which is installed on your production or staging server. This is pretty standard for all continuous observability solutions. I don't know how other solutions work, but I assume they are pretty similar. This means there's clear separation between the developer and production. As you can see, the DevOps still has that guarding line we're talking about. They need to connect the agent to the management server and that's where their job ends. Developers don't have direct access to production, only through the management server. That means no danger to the running production servers from a careless developer like myself. The agent is just a small runtime you can add to your production staging server. It's very low overhead and it implements the debugging logic. Finally, everything is piped through the server back to your IDE directly. So as a developer, you can keep working in the IDE without leaving your comfort zone. Okay, that should raise the vendor alert right here. I heard that bullshit line before, right? APMs have been around forever and have been optimized. How can a new tool claim to have lower overhead than an established and proven solution? As I said before, APMs look at everything. A continuous observability tool is surgical. That means that when an APM raises an interesting issue, we can look at a specific thing like a line of code. When a continuous observability solution isn't running, its overhead is almost nothing. It literally does nothing other than check whether we need it. It doesn't report anything and it's practically idle. When we do need it, we need more data than the APM does. But we get that from one specific area of the code. So there's an overhead, but because it impacts only one area of the code, it's tiny. It's very localized. This is the obvious question. What if I look at code that gets invoked a lot? As I said, continuous observability gets even more data than an APM does. This can bring down a system and well, we could end up here. So this is where continuous observability tools differ. Some tools provide the ability to throttle expensive actions and only show you some of the information. This isn't a big deal unless you have high volume requests. I think these things are best known shown in a demo because I can talk your ears off, but showing a quick demo can explain this faster. I'll use Kotlin for this demo, but this applies to other languages. I'll skip the setup portion since we don't have much time, but notice we have a free tier you can use freely from the website. This is the prime main app in Kotlin. It simply loops over numbers and checks if there are prime numbers. It sleeps for 10 milliseconds, so it won't completely demolish the CPU, but other than that, it's a pretty simple application. It just counts the number of primes it finds along the way and prints the results at the end. We use this code a lot when debugging since it's CPU intensive and yet very simple. In this case, we would like to observe the variable i, which is the variable we're evaluating here, and printout cnt, which represents the number of primes we found so far. The simplest tool we have is the ability to inject a log into the application. We can also inject a snapshot or metric. I'll discuss all of those soon enough. Selecting a log opens the UI to enter a new log. I can write more than just text. In the curly braces, I can include expressions. I want such as the value of the variables that are included in this expression. I can also invoke methods and do all sorts of things, but here's the thing. If I invoke a method that's too computationally intensive, or if I invoke a method that changes the application state, the log won't be added. I'll get an error. After clicking OK, we see the log appearing above the line in the IDE. Notice that this behavior is specific to IntelliJ or JetDrain's IDE's. In Visual Studio Code, it will show a marker on the side. Once the log is hit, we'll see logs appear in batches. Notice I chose to pipe logs into the IDE for convenience, but there's a lot more I can do with them. For now, the thing I want to focus on is the last line. Notice that the log point is paused due to high core rate. This means additional logs won't show for a short time since logging exceeded threshold of CPU usage. This can happen quickly or slowly depending on how you're observing. That's a relatively simple demo. To whet your appetite, let's pull out the big demo guns. I decided to take on Netflix and build my own movie service. Seems easy enough. Turns out it isn't so much, and those people on Netflix sure know how to build applications. This is rough. I'm terrible at front-end, so I hired a developer to build it and react. He also built a Node.js server, which was pretty intuitive for a front-end person. This is actually a pretty sensible architecture. It lets the front-end developer handle all their stuff, including the start of the back-end area. Node.js scales reasonably well and can handle the stuff it's good at, communicating with the UI front-end, etc. Node.js also scales reasonably well. So we've got a pretty good deal overall. Storage transactions and reliability are a decade ahead in the world of Java, so using Spring Boot for the back-end, heavy lifting also makes sense. I know it, and I built a lot of stuff with it. This is a merge of both best of each world has to offer. It also lets us hire the right people for the right job. The problem starts when we need to debug this, especially in production. Finding the source of bugs is a process of digging through logs with different conventions and sometimes even different terms for the same thing. And the problem is we have bugs. So here is the app that will take over the world. As you can see, I have a list of movies and the UI that reminds us a bit of those big guys who, again, are doing a pretty spectacular job. I can add a movie to my list, I can visit my list and see the movies within it, and I can open a specific movie and see details about it. That's pretty much it. But here is where the weird thing. When I click the no time to die movie, there is a problem. I get rid of notice. Instead, it's a weird bug, and I have no idea why it would happen. I think we can debug it using Lightrun. This is the Node.js project that implements the initial backend of our architecture. This is the method that gets invoked when we click the movie and want to see the details. This time, I'll add a snapshot. Some other continuous observability tools call this a capture or non-breaking breakpoint, which to me sounds weird, but the idea is usually the same. Once I press OK, the camera button appears on the left, indicating the location of the snapshot, like you would see with the regular ID breakpoint. Now, let's go back to the browser for a second, and in the browser, we can click the movie and wait a few seconds for the problem to reproduce. Now we need to go back to the IDE, and now we wait a second, and the snapshot is hit. So what is the snapshot? It gives us a stack trace and variables just like a regular breakpoint. We know and love, but it doesn't stop at that point, so your server won't be stuck waiting for a stepover. Now, obviously, you can step over the code, so you need to step by individual snapshots, but this has huge benefits, especially in production scenarios. So what do we see here? We see the idea of the movie, and it seems like the right idea, so this isn't the case of a bad idea passing from the UI. We need to dig deeper into the Spring Boot server. In the Spring Boot server code, we've got the fetch movie details method, which does the actual fetching. I place a snapshot here, and I'll spare you the pain of going to the web every time. I have a curl command that performs the request, so I'm running it in a different window here, and as it's not really interesting. So now we need to wait for the snapshot to get hit. If this seems to take too long, spoiler, it never gets hit, and the reason for that is the cacheable annotation you see above the method. It seems the response from the method is cached. Unfortunately, we can't rely on guesses. We need to verify. So we need to go to the movie manager, which is our REST controller. It handles all the web requests directly, so here we can add another snapshot and again try to trigger the curl script with the hope that this time we'll get it right and the snapshot will get hit. After a bit of waiting, we finally hit the snapshot and in the get method. This means that our request is happening, but the failure we're seeing is because the cache got corrupted with bad data. Why would that happen, and can we continue our investigation? Well, yes we can. The place where the cache might be corrupted is here. In check for file updates, this method runs periodically every 60 seconds and checks the file system for an import file. This is how we add new movies into the system. We place a file within with data and it imports it into the database. It might break existing movies. Let's create a snapshot here and while we wait for a minute for the snapshot to hit, we can use that time productively to delete the no longer used snapshots from before. They will eventually expire, but they're left here in case you need reference. One of the nice things about debugging with snapshots is that normally you don't have to wait. You can just place them over the code and they get hit at some point. Great, the snapshot got hit just in time. So that means this code is working. Now all I need to do is evict the cache with a dummy file and we're done. Or are we? Notice the methods here, the batch update evicts the cache. So it would seem if you're new to Spring, if we'll do that and check for the bug again, we'll see the cache wasn't evicted. The reason for this is that Spring only reads annotations when they're invoked from a different class by default. This is one of those gotchas that often fail developers new to Spring. I can't believe I fell for that, huh? This was relatively simple in terms of observability. Let's up the ante a bit and talk about user-specific problems. So here I have a problem with the trending request. One specific user is complaining that the trending list on his machine doesn't match the trending list for his peers. The problem is that if I put a snapshot, I'll get a lot of noise because there are many users reloading all the time. So the solution is to use conditional snapshots like you can with a regular debugger. Notice that you can define a condition for a log and for metrics as well. This one, this is one of the key features of continuous observability. I add a new snapshot and in it I have an option to define a lot, quite a lot of things. I won't even discuss the advanced version of this dialogue in this session. This is a really trivial condition. We already have a simple security utility class that I can use to query the current user ID. So I just make use of that and compare the response to the idea of the user that's experiencing a problem. Notice I use the fully qualified name of the class. I could have just written a security and it's very possible it would have worked, but it isn't guaranteed. Names can clash and on the agent and the agent side isn't aware of the things that I have in the ID such as often. So it's often a good practice to be more specific with the fully qualified class names. After pressing OK, we see a special version of the snapshot icon with the question mark on it. This indicates that this action has a condition on it. Now it's just a waiting game for the user to hit that snapshot. This is the point where normally you can go to make yourself a cup of coffee or even just go home and check out the next day. That's the beauty of this sort of instrumentation. In this case, I won't make you wait long. The snapshot gets hit by the right user despite other users coming in. This specific request is hit by the right user despite the request coming in. This specific request is from the right user ID. We can now review the stack information and fix the user specific bug. The next thing I want to talk about is metrics. APMs give us large scale performance information, but they don't tell us fine grained details. Here we can count the number of times a line of code was reached using a counter. We can even use a condition to qualify that. So we can go do something like count the number of times a specific user reaches that line of code. We also have a method duration, which tells us how long a method took to execute. We can even measure the time it takes to perform a code block using TikTok. That lets us narrow down the performance impact of a larger method to a specific problematic segment. In this case, I'll just use the method duration. Measurements typically have a name under which we can pipe them or log them. So I'll just give this method duration a clear name. In this case, I'm just printing it out to the console, but all of these measurements can be piped to stats D and Prometheus. I'm pretty awful at DevOps, so I really don't want to demo this that in this case. But it doesn't work. Measurements typically have a name under which we can pipe them or log them. So I'll just give this method duration a clear name. In this case, I'm just printing it out to the console, but all of these measurements can be piped to stats D and Prometheus. I'm pretty awful at DevOps, so I really don't want to demo that in this case. But it does work if you know how to use these tools. As you can see, the duration information is now piped into the logs and provides us some information on the current performance of the method. If I had more time, I'd talk about scaling this further with tools like Airflow and Python or Spark and Scala, etc. These are amazing cases where continuous observability really shines, but we need to wrap things up. So in closing, I'd like to go over what we discussed here and a few things that we didn't have time for. Lutron supports JVM languages like Java, Kotlin, Scala, etc. It supports Node for both JavaScript and TypeScript and Python, even complex stuff like Airflow. We're working hard on adding new platforms and doing it really fast. When we add actions, conditions run within the sandbox, so they don't take up CPU or crash the system. This all happens without networking, so something like a networking hiccup won't crash the server. Security is especially crucial with solutions like this one. One of the core concepts of the server queries information, not the other way around, as you would see with solutions such as JDWP, etc. This means operations are atomic and the server can be hidden behind firewalls, even from the rest of the organization. PII reduction lets us define conditions that would obscure patterns in the logs. So if a user would print out a credit card number by mistake, you can define a rule that would block that. This way, the bad data won't make it into your logs and won't expose you to liability. Block listing lets us block specific classes, methods, or files. This means you can block developers in your organization from debugging specific files. This means a developer won't be able to put a snapshot or a log in a place where a password might be available to steal user credentials or stuff like that. This is hugely important in large organizations. Besides the sandbox, I'd also like to mention that LATREN is very efficient and in our benchmarks has almost no runtime impact when it isn't used. It has very small impact even with multiple actions in place. Finally, LATREN can be used from the cloud or using an on-premise install. It works with any deployment you might have with a cloud-based or container-based on-premise, microservices, serverless, etc. Thanks for bearing with me. I hope you enjoyed the presentation. Please feel free to ask any questions and also feel free to write to me. Also, please check out talktoduduct.dev where I talk about debugging in-depth and check out latron.com, which I think you guys will like a lot. If you have any questions, my email is listed here and I'll be happy to help. Thank you.
 Hello everyone, today we're going to talk about polyglot cloud native debugging and a  bit about APMs. We don't have much time so I'll get right to it. But first a few things  about me. I was a consultant for over a decade, I worked at San, founded a couple of companies,  wrote a couple of books, I wrote a lot of open source code and currently work as a developer  advocate for Lightrun. My email and Twitter accounts are listed here so feel free to write  to me. I have a blog that talks about debugging and production issues at talktodaduck.dev.  It would be great if you check it out and let me know what you think.  I love APMs, they are absolutely wonderful. I'm old enough to remember a time where they  weren't around and I'm so happy we moved past that. This is absolutely amazing. The dashboard and  the details, you get this great dashboard with just everything you need. Amazing. We're truly at a  golden age of monitoring. Hell, when I started we used to monitor the server by kicking it and  listening to see if the hard drive was spinning properly. Today with Kubernetes, the deployment  scale to such a level that we need tools like this to get some insight into production. Without an  APM, we're well not blind as a bat, but it's pretty close. A lot of the issues we run into start  when we notice an anomaly in the dashboard. We see a spike in failures or something that  performs a bit too slow. The APM is amazing in showing those hiccups. But this is where it stops.  It can tell us that a web service performed badly or failed. It can't tell us why. It can't point  at a line of code. So let's stop for a second and talk about a different line. This line.  On the one side we have developers. On the other side we have the ops or DevOps. This is a line  we've had for a long time. It's something we drew out of necessity because when developers were given  access to production, well, I don't want to be too dramatic. But when developers got access to  production, it didn't end well. This was literally the situation not too long ago. Yes, we had  sysadmins, but the whole process used to be a mess. That was no good. We need a better solution  than this hard separation because the ops guys don't necessarily know how to solve the problems  made by the developers. They know how to solve ops problems. So when a container has a problem  and the DevOps don't know how to fix it, well, it starts a problematic feedback loop of test,  redeploy, rinse, repeat. That isn't ideal. Monitoring tools are like the bat signal.  They come up and we the developers, we're Batman, Batwoman, Batperson. All of us heroes.  We step up to deal with the bugs and we're the last line of defense against their villainy.  Well, we're code about people, really, kind of the same thing without the six pack abs.  Code about man needs to know where the crime or bugs are happening in the code. So these dashboards,  they point us towards the crime we have to fight in our system. But there's that's where things  get hard. We started digging into logs trying to find the problem. The dashboard sent us into a  general direction like a performance problem or higher error rates. But now we need to jump into  logs and hope that we can find something there that will somehow explain the problems we're seeing.  That's like going from jet engine back to Stone Age tools. There are so many problems,  log processing platforms that do an amazing job processing these logs and finding the goal within  them. But even then, it's a needle in the haystack. That's the good outcome with the logs already  there waiting for us. But obviously, we can't have logging all over the place. Our billing will go  through the roof and our performance will, it will suffer. We're stuck in this loop of add a new log,  go through CI CD, which includes the QA cycle and everything. This can take hours at best.  Then we produce the issue in the production server with your fingers crossed and try to analyze what  went on. Hopefully you found the issue because if not, it's effectively runes repeat for the whole  process. In the meantime, you still have a bug in production and developers are wasting their time.  There just has to be a better way. It's 2021 and logs are the way we solve bugs in this day and age.  Don't get me wrong. I love logs and today's logs are totally different from what we had even  10 years ago. But you need to know about your problem in advance for a log to work.  The problem is I'm not clairvoyant. When I write code, I can tell what bugs or problems the code  will have before the code is written. I'm in the same boat as you are. The bug doesn't exist yet.  So I'm faced with the dilemma of whether to log something. This is a bit like the dilemma of  writing comments. Does it make the code look noisy and stupid or will I find this useful at 2am  when everything isn't working and I want to rip out the few strands of hair?  I still have left because of this damn production problem. The buggers are amazing. They let us  set breakpoints, see call stacks, inspect variables and more. If only we could do the same for  production systems. But the buggers weren't designed for this. They're very insecure when  debugging remotely. They can block your server while sending the bug commands remotely. A small  mistake such as an expensive condition can literally destroy your server. I might be repeating an  urban legend here but 20 or so years ago I heard a story about a guy who was debugging a rail system  located on a cliff. He stopped at a breakpoint during debugging and the multi-million dollar  hardware fell into the sea because it didn't receive the stop command. Again, I don't know if  it's a true story but it's plausible. Debuggers weren't really designed for these situations.  Worse, the buggers are limited to one server. If you have a cluster with multiple machines,  the problem can manifest on one machine, always or might manifest on a random machine.  We can't rely on pure luck. If I have multiple servers with multiple languages and platforms  crossing from one to another with a debugger, well, it's possible in theory but I can't even imagine  it in reality. I also want to revisit this slide because I do love having APMs and looking at  their dashboard gives me that type of joy we get from seeing the result of our work plotted out as  a graph. I feel there should be a German word to describe that sort of enjoyment. But here's the thing,  APMs aren't one thing. The more you instrument, the more you have runtime overhead. The more you  have runtime overhead, the more hosts you need to handle the same amount of tasks. The more hosts you  have, the more problems you have and they become more complex. I feel Schrodinger should deliver  the next line by observing we effectively change some of the outcome. An APM, some people use that  as an excuse to avoid APMs, which I feel is like throwing away the baby with the bathwater.  We need APMs. We can't manage at scale without them. But we need to tune them and observing  everything isn't an option. Thankfully, pretty much every APM vendor knows that and they all  let us tune the ratio of observability to performance so we can get a good result.  Unfortunately, that means we get less data. Couple that with the reduction of logs that we need to  do for the same reason and the bad problems we had having production just got a whole lot worse.  Let's take the Batman metaphor all the way. We need a team up. We need some help on the servers,  especially in a clustered polyglot environment where the issue appears on one container,  moves on to the next, etc. Remember this slide. We need some way to get through that line.  Not to remove it. We like that line. We need a way to connect with the server and debug it.  Now, I'm a developer, so I try to stay away from management buzzwords. But the word for this is  shift left. It essentially means we're letting the developers and QA get back some of the access we  used to have into the ops without demolishing the gains we had in security and stability.  We love the ops people and we need them. So this is about helping them keep everything running  smoothly in production without stepping on their toes or blowing up their deployment.  This let's lead us here. What if you could connect your server to a debugger agent that would make  sure you don't overload the server and don't make mistakes like sending a breakpoint or something  like that? That's what Continuous Observability does. Continuous Observability is complementary to  the APM. It works very differently. Before we go on, I'd like to explain what's Continuous Observability.  Observability is defined as the ability to understand how your systems work on the inside  without shipping new code. The without shipping new code portion is key. But what's Continuous  Observability? With Continuous Observability, we don't ship new code either, but we can ask  questions about the code. Normal Observability works by instrumenting everything and receiving  the information. With Continuous Observability, we flip that and we ask questions, then instrumentation  is made based on the questions. So how does that work in practice? Each tool in this field is  different. I'll explain the LATRIN architecture since that's what I'm familiar with and I'll try to  qualify when it's different from other tools. In LATRIN, we use a native ID plugin to VS code or  JetBrains IDE's such as IntelliJ, PyCharm, WebStorm, etc. We can also use a command line tool as well.  Other tools sometimes have a web interface or CLI only. This client lets us interact with the  LATRIN server. This is an important piece of the architecture that hides the actual production  environment. Developers don't get access to production area, which is still the preview of  DevOps. We can insert an action, which can be a log or a snapshot or a measurement metric.  I'll show all of these soon enough. This talk will go into the code portions soon.  Notice that the LATRIN server can be installed in the cloud or as a SAS or on-premise and managed by Ops.  The management server sends everything to the agent, which is installed on your production or  staging server. This is pretty standard for all continuous observability solutions. I don't know  how other solutions work, but I assume they are pretty similar. This means there's clear separation  between the developer and production. As you can see, the DevOps still has that guarding line we're  talking about. They need to connect the agent to the management server and that's where their job  ends. Developers don't have direct access to production, only through the management server.  That means no danger to the running production servers from a careless developer like myself.  The agent is just a small runtime you can add to your production staging server.  It's very low overhead and it implements the debugging logic. Finally, everything is piped  through the server back to your IDE directly. So as a developer, you can keep working in the IDE  without leaving your comfort zone. Okay, that should raise the vendor alert right here. I heard  that bullshit line before, right? APMs have been around forever and have been optimized. How can a  new tool claim to have lower overhead than an established and proven solution? As I said before,  APMs look at everything. A continuous observability tool is surgical. That means that when an APM  raises an interesting issue, we can look at a specific thing like a line of code. When a continuous  observability solution isn't running, its overhead is almost nothing. It literally does nothing other  than check whether we need it. It doesn't report anything and it's practically idle. When we do  need it, we need more data than the APM does. But we get that from one specific area of the code.  So there's an overhead, but because it impacts only one area of the code, it's tiny. It's very  localized. This is the obvious question. What if I look at code that gets invoked a lot? As I said,  continuous observability gets even more data than an APM does. This can bring down a system and well,  we could end up here. So this is where continuous observability tools differ. Some tools provide  the ability to throttle expensive actions and only show you some of the information. This isn't a  big deal unless you have high volume requests. I think these things are best known shown in a demo  because I can talk your ears off, but showing a quick demo can explain this faster. I'll use Kotlin  for this demo, but this applies to other languages. I'll skip the setup portion since we don't have  much time, but notice we have a free tier you can use freely from the website. This is the prime main  app in Kotlin. It simply loops over numbers and checks if there are prime numbers. It sleeps for  10 milliseconds, so it won't completely demolish the CPU, but other than that, it's a pretty simple  application. It just counts the number of primes it finds along the way and prints the results at  the end. We use this code a lot when debugging since it's CPU intensive and yet very simple. In this  case, we would like to observe the variable i, which is the variable we're evaluating here,  and printout cnt, which represents the number of primes we found so far. The simplest tool we have  is the ability to inject a log into the application. We can also inject a snapshot or metric. I'll  discuss all of those soon enough. Selecting a log opens the UI to enter a new log. I can write more  than just text. In the curly braces, I can include expressions. I want such as the value of the variables  that are included in this expression. I can also invoke methods and do all sorts of things,  but here's the thing. If I invoke a method that's too computationally intensive, or if I invoke a  method that changes the application state, the log won't be added. I'll get an error.  After clicking OK, we see the log appearing above the line in the IDE. Notice that this behavior is  specific to IntelliJ or JetDrain's IDE's. In Visual Studio Code, it will show a marker on the side.  Once the log is hit, we'll see logs appear in batches. Notice I chose to pipe logs into the  IDE for convenience, but there's a lot more I can do with them. For now, the thing I want to focus on  is the last line. Notice that the log point is paused due to high core rate. This means additional  logs won't show for a short time since logging exceeded threshold of CPU usage. This can happen  quickly or slowly depending on how you're observing. That's a relatively simple demo.  To whet your appetite, let's pull out the big demo guns. I decided to take on Netflix and build my  own movie service. Seems easy enough. Turns out it isn't so much, and those people on Netflix  sure know how to build applications. This is rough. I'm terrible at front-end, so I hired a  developer to build it and react. He also built a Node.js server, which was pretty intuitive for  a front-end person. This is actually a pretty sensible architecture. It lets the front-end  developer handle all their stuff, including the start of the back-end area. Node.js scales reasonably  well and can handle the stuff it's good at, communicating with the UI front-end, etc. Node.js  also scales reasonably well. So we've got a pretty good deal overall. Storage transactions and  reliability are a decade ahead in the world of Java, so using Spring Boot for the back-end,  heavy lifting also makes sense. I know it, and I built a lot of stuff with it. This is a merge of  both best of each world has to offer. It also lets us hire the right people for the right job.  The problem starts when we need to debug this, especially in production. Finding the source  of bugs is a process of digging through logs with different conventions and sometimes even  different terms for the same thing. And the problem is we have bugs. So here is the app  that will take over the world. As you can see, I have a list of movies and the UI that reminds us  a bit of those big guys who, again, are doing a pretty spectacular job. I can add a movie to my  list, I can visit my list and see the movies within it, and I can open a specific movie and see  details about it. That's pretty much it. But here is where the weird thing. When I click the no time  to die movie, there is a problem. I get rid of notice. Instead, it's a weird bug, and I have no  idea why it would happen. I think we can debug it using Lightrun. This is the Node.js project that  implements the initial backend of our architecture. This is the method that gets invoked when we click  the movie and want to see the details. This time, I'll add a snapshot. Some other continuous  observability tools call this a capture or non-breaking breakpoint, which to me sounds weird,  but the idea is usually the same. Once I press OK, the camera button appears on the left, indicating  the location of the snapshot, like you would see with the regular ID breakpoint. Now, let's go back  to the browser for a second, and in the browser, we can click the movie and wait a few seconds for  the problem to reproduce. Now we need to go back to the IDE, and now we wait a second, and the snapshot  is hit. So what is the snapshot? It gives us a stack trace and variables just like a regular  breakpoint. We know and love, but it doesn't stop at that point, so your server won't be stuck  waiting for a stepover. Now, obviously, you can step over the code, so you need to step by individual  snapshots, but this has huge benefits, especially in production scenarios. So what do we see here?  We see the idea of the movie, and it seems like the right idea, so this isn't the case of a bad  idea passing from the UI. We need to dig deeper into the Spring Boot server. In the Spring Boot  server code, we've got the fetch movie details method, which does the actual fetching. I place  a snapshot here, and I'll spare you the pain of going to the web every time. I have a curl command  that performs the request, so I'm running it in a different window here, and as it's not really  interesting. So now we need to wait for the snapshot to get hit. If this seems to take too long,  spoiler, it never gets hit, and the reason for that is the cacheable annotation you see above  the method. It seems the response from the method is cached. Unfortunately, we can't rely on guesses.  We need to verify. So we need to go to the movie manager, which is our REST controller. It handles  all the web requests directly, so here we can add another snapshot and again try to trigger the  curl script with the hope that this time we'll get it right and the snapshot will get hit.  After a bit of waiting, we finally hit the snapshot and in the get method. This means that our request  is happening, but the failure we're seeing is because the cache got corrupted with bad data.  Why would that happen, and can we continue our investigation? Well, yes we can. The place where  the cache might be corrupted is here. In check for file updates, this method runs periodically every  60 seconds and checks the file system for an import file. This is how we add new movies into the  system. We place a file within with data and it imports it into the database. It might break  existing movies. Let's create a snapshot here and while we wait for a minute for the snapshot to hit,  we can use that time productively to delete the no longer used snapshots from before.  They will eventually expire, but they're left here in case you need reference.  One of the nice things about debugging with snapshots is that normally you don't have to wait.  You can just place them over the code and they get hit at some point. Great, the snapshot got hit  just in time. So that means this code is working. Now all I need to do is evict the cache with a  dummy file and we're done. Or are we? Notice the methods here, the batch update evicts the cache.  So it would seem if you're new to Spring, if we'll do that and check for the bug again, we'll see the  cache wasn't evicted. The reason for this is that Spring only reads annotations when they're invoked  from a different class by default. This is one of those gotchas that often fail developers new to  Spring. I can't believe I fell for that, huh? This was relatively simple in terms of observability.  Let's up the ante a bit and talk about user-specific problems. So here I have a problem with the  trending request. One specific user is complaining that the trending list on his machine doesn't  match the trending list for his peers. The problem is that if I put a snapshot, I'll get a lot of  noise because there are many users reloading all the time. So the solution is to use conditional  snapshots like you can with a regular debugger. Notice that you can define a condition for a log  and for metrics as well. This one, this is one of the key features of continuous observability.  I add a new snapshot and in it I have an option to define a lot, quite a lot of things. I won't  even discuss the advanced version of this dialogue in this session. This is a really trivial condition.  We already have a simple security utility class that I can use to query the current user ID.  So I just make use of that and compare the response to the idea of the user that's experiencing a  problem. Notice I use the fully qualified name of the class. I could have just written a security  and it's very possible it would have worked, but it isn't guaranteed. Names can clash and  on the agent and the agent side isn't aware of the things that I have in the ID such as often.  So it's often a good practice to be more specific with the fully qualified class names.  After pressing OK, we see a special version of the snapshot icon with the question mark on it.  This indicates that this action has a condition on it. Now it's just a waiting game for the user  to hit that snapshot. This is the point where normally you can go to make yourself a cup of  coffee or even just go home and check out the next day. That's the beauty of this sort of  instrumentation. In this case, I won't make you wait long. The snapshot gets hit by the right user  despite other users coming in. This specific request is hit by the right user despite the  request coming in. This specific request is from the right user ID. We can now review the stack  information and fix the user specific bug. The next thing I want to talk about is metrics.  APMs give us large scale performance information, but they don't tell us fine grained details.  Here we can count the number of times a line of code was reached using a counter. We can even use  a condition to qualify that. So we can go do something like count the number of times a specific  user reaches that line of code. We also have a method duration, which tells us how long a method  took to execute. We can even measure the time it takes to perform a code block using TikTok.  That lets us narrow down the performance impact of a larger method to a specific problematic  segment. In this case, I'll just use the method duration. Measurements typically have a name  under which we can pipe them or log them. So I'll just give this method duration a clear name. In  this case, I'm just printing it out to the console, but all of these measurements can be piped to  stats D and Prometheus. I'm pretty awful at DevOps, so I really don't want to demo this that in this  case. But it doesn't work. Measurements typically have a name under which we can pipe them or log  them. So I'll just give this method duration a clear name. In this case, I'm just printing it out  to the console, but all of these measurements can be piped to stats D and Prometheus. I'm pretty  awful at DevOps, so I really don't want to demo that in this case. But it does work if you know  how to use these tools. As you can see, the duration information is now piped into the logs and provides  us some information on the current performance of the method. If I had more time, I'd talk about  scaling this further with tools like Airflow and Python or Spark and Scala, etc. These are amazing  cases where continuous observability really shines, but we need to wrap things up. So in closing,  I'd like to go over what we discussed here and a few things that we didn't have time for. Lutron  supports JVM languages like Java, Kotlin, Scala, etc. It supports Node for both JavaScript and TypeScript  and Python, even complex stuff like Airflow. We're working hard on adding new platforms  and doing it really fast. When we add actions, conditions run within the sandbox, so they don't  take up CPU or crash the system. This all happens without networking, so something like a networking  hiccup won't crash the server. Security is especially crucial with solutions like this one.  One of the core concepts of the server queries information, not the other way around,  as you would see with solutions such as JDWP, etc. This means operations are atomic and the server  can be hidden behind firewalls, even from the rest of the organization. PII reduction lets us  define conditions that would obscure patterns in the logs. So if a user would print out a credit  card number by mistake, you can define a rule that would block that. This way, the bad data won't  make it into your logs and won't expose you to liability. Block listing lets us block specific  classes, methods, or files. This means you can block developers in your organization from debugging  specific files. This means a developer won't be able to put a snapshot or a log in a place where  a password might be available to steal user credentials or stuff like that. This is hugely  important in large organizations. Besides the sandbox, I'd also like to mention that LATREN  is very efficient and in our benchmarks has almost no runtime impact when it isn't used. It has very  small impact even with multiple actions in place. Finally, LATREN can be used from the cloud or using  an on-premise install. It works with any deployment you might have with a cloud-based or container-based  on-premise, microservices, serverless, etc. Thanks for bearing with me. I hope you enjoyed the  presentation. Please feel free to ask any questions and also feel free to write to me. Also, please  check out talktoduduct.dev where I talk about debugging in-depth and check out latron.com,  which I think you guys will like a lot. If you have any questions, my email is listed here and  I'll be happy to help. Thank you.
debugger_0_179.png debugger_180_1079.png debugger_1080_1331.png debugger_1332_2879.png debugger_2880_3119.png debugger_3120_3407.png debugger_3408_3923.png debugger_3924_4559.png debugger_4560_5039.png debugger_5040_6863.png debugger_6864_7679.png debugger_7680_12131.png debugger_12132_13715.png debugger_13716_14819.png debugger_14820_15407.png debugger_15408_21515.png debugger_21516_21947.png debugger_21948_25919.png debugger_25920_28127.png debugger_28128_28175.png debugger_28176_28247.png debugger_28248_28583.png debugger_28584_28715.png debugger_28716_28811.png debugger_28812_30035.png debugger_30036_30275.png debugger_30276_30515.png debugger_30516_31415.png debugger_31416_31571.png debugger_31572_32663.png debugger_32664_34559.png debugger_34560_35579.png debugger_35580_35651.png debugger_35652_40367.png debugger_40368_40847.png debugger_40848_40943.png debugger_40944_41687.png debugger_41688_42107.png debugger_42108_46491.png