Profiling Low-End Platforms using HawkTracer Profiler
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kolny, Marcin
HawkTracer is low-overhead instrumentation-based profiler built at Amazon Video for platforms with limited capabilities. It's highly extensible (at compile time) and portable so it can be run on almost any embedded device. In this talk I'll briefly talk about history of the profiler (why did I start it, and how did I manage to publish it to open source world), introduce the architecture of the profiler, present it's advantages and limitations, show how can you instrument the code and demonstrate the profiler in action by running it with an example project.
 Yes, thank you for coming. I know it's late and you're all tired. So my name is Martin. I work for Amazon for a prime video and I want to talk a little about the profiler that we've done at Amazon. So before I start talking about my profiler, just do a quick introduction to profilers in general. Probably some of you never used one, so I just do a quick introduction. Then I explain why we decided to do another profiler even though there is a bunch of profilers already. Then I talk a little about the Hocktracer features, do a demo and hopefully we'll have time for questions at the end. So a profiler basically is a tool that allows you to measure performance of the application. You can basically split existing profilers into two groups. It's a sample-based profiler that runs periodically and checks some information from your application. So the simple profiler, simple sample-based profiler can be like that. You have a while loop and you check, for example, the call stack of a specific process, based on process ID. Later on, you can gather those data and generate some statistics of, for example, which function was called most frequently. This is not very accurate because the accuracy actually depends on the sampling frequency. So, for example, here we have one second frequency. It's a good method to find why your application is in general slow. But if you want to have very detailed data, for example, you have the spike, performance spike, and you want to understand what's going on, it's good to do an instrumentation-based profiling. So you basically modify your source code by putting some trace points and you exactly know how long or how much resources did we use at that point of time. So, for example, here we have very simple profiling, instrumentation-based profiling. We measure the time we spent in the full function by saving the timer before calling the function and after calling the function. And then we print the results. There's apparently another profiling. A methodology is a guessing-based profiler I learned from Alex today, which is basically a developer looks at the code and tries to guess why the code is slow. But, yeah. Guess-based profiling. Well, I'm not going to talk about this. And the Hock Tracer is instrumentation-based profiling. That means that you need to modify your code. You need to put some trace points in the code to find why, to measure some of the metrics. So, why did they, that why did we create the profiler? There's a bunch of profilers already. There's FIR, Ftrace, LTTNG, ETW for Windows and many others. So, we have very specific environment. At Amazon, my team is responsible for delivering Prime Video app on living room devices like streaming sticks, smart TVs, game consoles and stuff like that. And some of the devices are very, have very limited capabilities in terms of development. So, basically all we can do, we can just generate a package which is executable with some assets and upload it to the device. So, there's no way to SSH to that device. There's no way to run another process and stuff like that. So, we kind of, kind of do much. We, well, all we can do is just to run our application that we build. Also, from the language point of view, so we have a native stack and a scripted stack. And we wanted to be able to profile both stacks at the same time. And we couldn't really find a good profiler for this kind of use case. So we decided to build our own. And before we did that, we got the requirements, what we actually want to achieve by building the new profiler. And so, those are the requirements. First of all, we only targeting user space profiling. Obviously, for those limited platforms, we couldn't even load anything to kernel. So, user space only. We needed to build it as a library because, as I said, we can't run another process. So, it needs to be embedded to the application itself. Since those devices are sometimes very low-end devices like, I don't know, a single core CPU, 600, 900 megahertz. Well, for some people, it's not really low-end, but for us, it is. So, we tried to make the profiler so the overhead is not that significant. For some devices, we also don't have access to persistent storage. So, we can't save results on the disk. Sometimes we can, but we cannot access those data after that because this persistent storage is only available from the application point of view. So, you can't log into the device and gather the data. So, we decided, okay, let's assume that we don't have the persistent storage at all. And since we're running on different platforms, different manufacturers, we wanted to have the profiler as portable as possible so we can build it once. Our developers learned the tool once and they can use it for all the possible platforms that we support. And it should be easy to use, of course, so everybody can instrument the code easily and gather data quite easily. So we come up with a very, very basic design. As you can see, there's a user application layer. It's basically the application that is running on the device. And this application links to the Hocktracer library. Hocktracer library might have a bunch of timelines. The timelines are basically like buffers where a user sends events. So whenever you want to trace something, I don't know, for example, we want to know how much memory we use at this point of time. We generate an event that contains the information about the memory usage. It goes to the timeline. And the timeline accumulates those events. And once the timeline is full, it calls the flush method. And the flush method sends all the events that we gather to a listener. And the listener can be either, you can use either the one that is already existing in the library or you can define your own listener. And what listener can do? Well, basically, listeners should save the data somehow. So there is a file listener that saves it to the file or there's a TCP listener that streams the data over the network. And then the data is basically the binary stream. And you can't really, you can't really analyze the binary stream. You need to convert this binary stream to some human readable format. And in order to do that, we created a library. It's called Hocktracer parcel library that allows us to convert this byte stream to some structures that can be converted to another format. And so there are two options. You can either use Hocktracer converter, which is an application that converts this byte stream to one of the well-known formats like we currently support trace event format that is supported by Google trace viewer and we also generate flame graphs. If that's not enough for you, the parcel library can be used, you can use that for writing your own client, your own converter. And it's available either for C++, Python or Rust. I'll show later how to do it in Python. It's quite easy. So I mentioned those three components already. So the event is, as a thing that basically carries the information that we want to kind of have in the results, like for example, I don't know, time spent in the function or memory usage, CPU usage at that point of time. Supports inheritance, so you can have an event and then you can inherit from that event to add more fields, for example. The timeline is basically a buffer where you push events. The timeline also timestamps the event. So every time you push the event to the timeline, it gets time stumped. So you know exactly when the event happened. It can be either thread safe or not thread safe, lock free. That means that if you know that the timeline is used only in the single thread, there's no point of introducing mutexes and stuff like that. But if you want to use the same timeline across different threads and push events from different threads, then you need to enable the thread safety feature. But obviously that introduces some extra overhead because you need to lock the mutex every time you push an event. And the timeline listener is basically a C function that the user defines that gets all the events and does something with them. And so I show you how to define your custom class. So let's say you want to trace memory usage and the CPU usage. So how you do that? You basically use the HD declare event class macro. And the first argument of that macro is the name of your event. The second one is the base event class because as I said, it supports inheritance. And all the events must at least inherit from HD event is a base class. And then you define fields. And each field is defined by three properties. It's a type. It's either integer, string, struct, flow, double, and pointer. And then the second argument is a C type. So it's either int, uint, 6040, here, or car, or other. And the last one is the name of that field. And that macro generates basically loads of code. The most important one is that it actually generates the C structure. So you can see here that it generated the resource usage event C structure with those fields and with the base field of type HD event. And the HD event type has basically an ID. Each event has unique identifier so we can distinguish them. It has a timestamp. As I mentioned, that's the pointer to a class that describes that data structure. And it also generates few helper functions. The first one is for serializing the event. So it serializes that event to the byte stream. The other one is for getting event class instance. So if you at runtime need to know what's the structure of that event, you can use that function. And the last one is the most important, I think, because it's used for registering this event class in the Hoke Tracer system. So before you use the event class, you need to call that function. Otherwise, Hoke Tracer doesn't know what's the class and probably will crash if you forget about this. And basically, so the byte stream that is sent to a client, the client doesn't know about events, event classes that you defined in your application. You can define your applications and then have the client that receives the stream but he doesn't know what events you define. So first, before we send any events, we send the metadata stream. And it basically contains information about all the event classes that you registered. So it contains information about the class name, about the class identifier, and all the fields that are defined in the class. So then, and the next, when we actually send this stream, so it's below, first we send the class ID and then we send all the other fields. So then the parser knows, okay, I got this class ID. I know how to parse this because I already got the metadata stream. So I know that the next field is, for example, a timestamp field and it has 8 bytes. The next field is a CPU usage. It also has 8 bytes in the integer. So I know how to parse it. So there's no need to recompile a client if you add a new class to your library. Timeline. Basically, most of the time, we are using the global timeline. And the global timeline is not actually the single timeline. It's a timeline per thread. So we don't have to lock everything. Every time we push something to a timeline. But it shares listeners. So if you register a listener to one of the timelines in one of the threads, you will kind of automatically register it for all the other threads for global timeline. If you really need to create your own timeline, you can do that. But as I said, it's very, very uncommon use case. So I just recommend you to use global timeline and to access the global timeline, just call htglobaltimeline.get. It will return a pointer to that timeline. So our most common use case is to measure the time spent in a function or time spent in the scope. So we've also introduced some of the helper macros that allow you to measure the time we spend in the scope. So there's HTTP function that takes timeline pointer as an argument. And it basically measures the time spent in the full function. And the output, if you look at the end of the slide, is basically it generates a new event with the duration, which is how much we spent in that scope. It also adds information about the thread identifier. So you know that this function was called in this particular thread. And it also adds the label. So in that case, the HTTP function macros sets the function name as a label. If you want to trace custom scope, you can do that. It's just another macro. And plus, you also need to add the custom label for that. And those macros are only available in EC++ and in the GNU-C compiler because C basically does not have such thing like a destructor. So we don't have a way to call a call back at the end of the scope. If you want to measure arbitrary code, not necessarily a scope. There's also a set of functions start, call stack start string, and the stop. And that measures the time between calling those two functions. And again, it generates the same event with the label that you specified here. So the scope is basically like a scope of the C++ variable. So in this example, the scope of this trace point HTTP function is the whole function. But here, it only is inside this curly braces. So, yeah, that's the scope. And so that was more or less about the Hock Tracer internals. And now, how can you integrate it with your project? If you just download the source code, do make, make, install, it will install the PKG config file. So you can just use the PKG config and compile it with your project. This Hock Tracer itself uses a CMake as a build system. So you can either use it as an external project very easily. There is an example how to actually do that. So I recommend you to copy just the Hock Tracer.CMake file to your repository and include it in your project. Or if you installed it as a system library, you can just use a fine package Hock Tracer with a specific version and then link it to your project. And the third option is the one that I actually recommend is that we basically is like SQLite. It has many files, but at the end, all those files get merged to a single file. So the same Hock Tracer. We have three files eventually. We have a HT config where you can modify a configuration. We have a header file and the CPP file where the whole implementation is. And the CPP file, it can actually be compiled using a C compiler. You might just not have all the features. But yeah, it's possible to compile it just using the C compiler. What is the license? MIT. Sorry. I should mention that before. Yeah, the license is MIT, so you should be able to use it as you want. Okay, so demo. The first demo. So I've implemented a sorting algorithm. It sorts 400 numbers. And turns out that the algorithm is very slow. So if you see, I press enter now. And yeah, it took a while to sort those numbers. It was only 400 numbers, so it should be like very quickly. So let's look at the source code. We call the quick sort here. So now we want to know why is it slow. And in order to know why is it slow, we need to instrument our code. So I'll show you how to set up the code to basically work with Hock Tracer. At the beginning, you need to call HT init function. This is very important. It initializes some internal buffers and registers some base classes. So don't forget about this. Then we need to create the listener. As I said, the timeline needs to have a listener, otherwise no one will be able to access those events that we generate. So for this purpose, we decided to use a file listener. A file listener basically saves all the events to a file. So we decided to save it to a sort.htdump file. So it will generate the binder file. There is some check if the listener was created correctly. Then we registered the listener to the timeline because we only created an instance here, but now we need to actually register it to the timeline. And we use the global timeline as this is the most convenient thing to do. This is the callback of the function, and this is like a user data. So the callback knows the context. And then we initialize our input. So we generate 400 random numbers, and then we quick sort this array. So if you see, if you look at the functions, we basically I added to all the functions, I've added this trace point. So whenever function gets called, we generate the event, how long did it take to execute that particular function. And we have a few other functions. We have a partition function, we have a quick sort function, and we also have a swap function. So now I run this. It was already instrumented. So if you look here, it already generated this sort.htdump file. And now this is a binary file. So if you try to open this, you will see this is basically the binary file. So what we can do, we can convert that binary stream to something that we can actually read. And I mentioned about the HowTracerConverter program. So if you look at this, you see that this program takes basically three mandatory parameters. The first is format, which is an output format, which we convert our data to. And it can be either a flame graph or a Chrome trace, or it can be a debug, which basically prints everything as it is. So we decide to use Chrome trace. Another parameter is output file. Let's say sort slow JSON and the source. Source is very important, as the program needs to know where to get the data from. And the source, this Chrome tracing, HowTracerConverter tool only supports two sources. It's either a file name or a TCP IP address. In case we, for example, stream the data over the network, it can listen to a specific port and receive the data. So the source is sort.htdump. And we run it. It's processing the data. It's completed successfully. Let's see if we have this. Yes. So we have sort slow. And now we can use the Chrome tracing viewer to actually see why our application is slow. So I load the JSON file that we generated, and we see basically this shows the cost tax of our application. The x axis is the time, and we see that basically since the quick sort is a recursive algorithm, we see that quick sort cost quick sort and so on and so on. But if we look closer, we basically see that, apart from calling quick sort, it also calls something different. And if we zoom in more, we see that actually most of the time we spend in the swap function. I don't know if you can see, but this is basically it's quick sort. The blue one is a quick sort function. Then the green one is partition, and the green, the light green is a swap method. So we spent quite a lot of time in the swap method. So that's probably a function that we should look into. And if we see how the swap method is implemented, it takes A and B as an argument, and we want to swap values. So how did we do that? Well, yeah. If someone implemented it as the saving is to a temporary file, and then it's actually serialized to a string, and then we read it, that's probably why it's slow. So if we just change the implementation to something like this. Can you do it without temporary viable? Yeah, there is a trick with XOR, yeah. I would have to think about it, but yeah, I'm definitely, that's possible. Yeah, so if we fix that, then probably our trace is going to look completely differently, believe me. Saving temporary data to a file and read it after that is ways of time. So I'm not going to recompile it and rerun it. I guess you all believe me that this is going to fix our performance problem and it's going to be much faster. You need to save the, right? Sorry? Oh. Absolutely. Thanks. Yeah, live demo never works. So that was the first demo and the second demo. So as I mentioned, so imagine that previously I showed you how to create your own class, which was for memory usage and CPU usage tracking. And actually using the Chrome tracing viewer, we couldn't visualize this. So how can we see like a custom data? So for that, as I mentioned, we can build our own client to process events. So there is a module Hoke Tracer for Python. It's very simple to use. You create a client. You start the connection. So I said, okay, 1-2-7-0-0-1 on port 8765. And then whenever we receive an event here, we check if the event's name is resource usage event. If it is, then we take the CPU usage, we take a memory usage, we print CPU usage, and we draw memory usage on the graph. I'm not a Python developer, so maybe that's not the best way to graph something, but it works. And our code is very, very similar to the previous one. Instead of doing the file listener, we created the TCP listener that basically creates a server that streams the data. We also had this class that I showed before. So this is the definition of our class. And then every second here in the while loop, we allocate some amount of memory, half of the kilobyte. Then we make our CPU busy and we sleep for one second. And we report resource usage. So let's see how do we report the resource usage. There is a macro, ht-timeline push event. It takes a timeline as a first argument. Then the events type, then values of that event that we want to attach to this event. And then we push. And that pushes it. We also flush the timeline manually instead of waiting for the buffer to get full, because we want to have the data immediately in the client, so it's kind of smooth drawing. So we can run the demo. I run the resource usage. And I run my Python client. And you can see that it draws the memory usage here. So basically what we've done, we made our custom kind of event converter. And we decided how we visualized the data in Python just in, I don't know, 20 lines of code. And also the CPU usage is printed here. So it's around, yeah, 1, 2, 3, 4, 5, 6, 7, 8. Yeah, so that was the second demo. There's a bunch of things that we want to do for the future. There are some missing features. Like, we don't support floating point numbers at the moment. Optional fields is also something we want to have. So for example, you have a class with some event, with some fields, but sometimes some of the fields should not be included. So that would be nice to have. We also want to have more converters in the whole tracer converter. Like, a CTF is a common trace format supported by TraceCompass or LTTNG scope viewer. So it would be nice to have. We use that profiler also for profiling C++ and Lua or JavaScript stack at the same time. But Lua bindings and JavaScript bindings are not open sourced. So we also want to do that. And lots of documentation improvements. Even though the documentation is quite okay, I see we can improve that more. So yeah, if you want to help, just go to hocktracer.org slash community. And you will find how to contact us. And we can work together. Now there's a bunch of links. There was also Rust bindings. Alex had a talk today afternoon. So if you search for FOSDEM 2019 profiling Rust, you should find that presentation. It was pretty cool. Okay, thank you. I think we have. Yes. So that depends. Creating new event, having ability to create a new event class in the other language might be quite tricky. But assuming that you already have all the event classes defined in your C or C++ code base and you run your other language on top of that, it should be fairly easy. Because you just need to expose. So for measuring time, you need to expose to function start and stop. Plus maybe start hocktracer or start register listener. So it should be fairly easy. For Rust, I don't know how much work that was for Alex. Yeah. And the interfacing. Yeah. Yeah. That depends what you want to do. But if you just want to measure time, it should be fairly easy. Yeah. Thank you. Thank you.
 Yes, thank you for coming. I know it's late and you're all tired. So my name is Martin.  I work for Amazon for a prime video and I want to talk a little about the profiler that  we've done at Amazon. So before I start talking about my profiler, just do a quick introduction  to profilers in general. Probably some of you never used one, so I just do a quick introduction.  Then I explain why we decided to do another profiler even though there is a bunch of profilers  already. Then I talk a little about the Hocktracer features, do a demo and hopefully we'll have  time for questions at the end. So a profiler basically is a tool that allows you to measure  performance of the application. You can basically split existing profilers into two groups.  It's a sample-based profiler that runs periodically and checks some information from your application.  So the simple profiler, simple sample-based profiler can be like that. You have a while  loop and you check, for example, the call stack of a specific process, based on process ID.  Later on, you can gather those data and generate some statistics of, for example, which function  was called most frequently. This is not very accurate because the accuracy actually depends  on the sampling frequency. So, for example, here we have one second frequency. It's a  good method to find why your application is in general slow. But if you want to have very  detailed data, for example, you have the spike, performance spike, and you want to understand  what's going on, it's good to do an instrumentation-based profiling. So you basically modify your source  code by putting some trace points and you exactly know how long or how much resources  did we use at that point of time. So, for example, here we have very simple profiling,  instrumentation-based profiling. We measure the time we spent in the full function by saving  the timer before calling the function and after calling the function. And then we print  the results. There's apparently another profiling. A methodology is a guessing-based profiler  I learned from Alex today, which is basically a developer looks at the code and tries to  guess why the code is slow. But, yeah.  Guess-based profiling. Well, I'm not going to talk about this. And the Hock Tracer is  instrumentation-based profiling. That means that you need to modify your code. You need  to put some trace points in the code to find why, to measure some of the metrics. So, why  did they, that why did we create the profiler? There's a bunch of profilers already. There's  FIR, Ftrace, LTTNG, ETW for Windows and many others. So, we have very specific environment.  At Amazon, my team is responsible for delivering Prime Video app on living room devices like  streaming sticks, smart TVs, game consoles and stuff like that. And some of the devices  are very, have very limited capabilities in terms of development. So, basically all we  can do, we can just generate a package which is executable with some assets and upload it  to the device. So, there's no way to SSH to that device. There's no way to run another  process and stuff like that. So, we kind of, kind of do much. We, well, all we can do is  just to run our application that we build. Also, from the language point of view, so we  have a native stack and a scripted stack. And we wanted to be able to profile both stacks  at the same time. And we couldn't really find a good profiler for this kind of use case.  So we decided to build our own. And before we did that, we got the requirements, what  we actually want to achieve by building the new profiler. And so, those are the requirements.  First of all, we only targeting user space profiling. Obviously, for those limited platforms,  we couldn't even load anything to kernel. So, user space only. We needed to build it  as a library because, as I said, we can't run another process. So, it needs to be embedded  to the application itself. Since those devices are sometimes very low-end devices like, I  don't know, a single core CPU, 600, 900 megahertz. Well, for some people, it's not really low-end,  but for us, it is. So, we tried to make the profiler so the overhead is not that significant.  For some devices, we also don't have access to persistent storage. So, we can't save  results on the disk. Sometimes we can, but we cannot access those data after that because  this persistent storage is only available from the application point of view. So, you  can't log into the device and gather the data. So, we decided, okay, let's assume that we  don't have the persistent storage at all. And since we're running on different platforms,  different manufacturers, we wanted to have the profiler as portable as possible so we  can build it once. Our developers learned the tool once and they can use it for all  the possible platforms that we support. And it should be easy to use, of course, so everybody  can instrument the code easily and gather data quite easily.  So we come up with a very, very basic design. As you can see, there's a user application  layer. It's basically the application that is running on the device. And this application  links to the Hocktracer library. Hocktracer library might have a bunch of timelines. The  timelines are basically like buffers where a user sends events. So whenever you want  to trace something, I don't know, for example, we want to know how much memory we use at  this point of time. We generate an event that contains the information about the memory  usage. It goes to the timeline. And the timeline accumulates those events. And once the timeline  is full, it calls the flush method. And the flush method sends all the events that we  gather to a listener. And the listener can be either, you can use either the one that  is already existing in the library or you can define your own listener. And what listener  can do? Well, basically, listeners should save the data somehow. So there is a file  listener that saves it to the file or there's a TCP listener that streams the data over  the network. And then the data is basically the binary stream. And you can't really, you  can't really analyze the binary stream. You need to convert this binary stream to some  human readable format. And in order to do that, we created a library. It's called Hocktracer  parcel library that allows us to convert this byte stream to some structures that can be  converted to another format. And so there are two options. You can either use Hocktracer  converter, which is an application that converts this byte stream to one of the well-known  formats like we currently support trace event format that is supported by Google trace viewer  and we also generate flame graphs. If that's not enough for you, the parcel library can  be used, you can use that for writing your own client, your own converter. And it's available  either for C++, Python or Rust. I'll show later how to do it in Python. It's quite easy.  So I mentioned those three components already. So the event is, as a thing that basically  carries the information that we want to kind of have in the results, like for example,  I don't know, time spent in the function or memory usage, CPU usage at that point of time.  Supports inheritance, so you can have an event and then you can inherit from that event to  add more fields, for example. The timeline is basically a buffer where you push events.  The timeline also timestamps the event. So every time you push the event to the timeline,  it gets time stumped. So you know exactly when the event happened. It can be either  thread safe or not thread safe, lock free. That means that if you know that the timeline  is used only in the single thread, there's no point of introducing mutexes and stuff  like that. But if you want to use the same timeline across different threads and push  events from different threads, then you need to enable the thread safety feature. But obviously  that introduces some extra overhead because you need to lock the mutex every time you  push an event. And the timeline listener is basically a C function that the user defines  that gets all the events and does something with them.  And so I show you how to define your custom class. So let's say you want to trace memory  usage and the CPU usage. So how you do that? You basically use the HD declare event class  macro. And the first argument of that macro is the name of your event. The second one  is the base event class because as I said, it supports inheritance. And all the events  must at least inherit from HD event is a base class. And then you define fields. And each  field is defined by three properties. It's a type. It's either integer, string, struct,  flow, double, and pointer. And then the second argument is a C type. So it's either int,  uint, 6040, here, or car, or other. And the last one is the name of that field. And that  macro generates basically loads of code. The most important one is that it actually generates  the C structure. So you can see here that it generated the resource usage event C structure  with those fields and with the base field of type HD event. And the HD event type has  basically an ID. Each event has unique identifier so we can distinguish them. It has a timestamp.  As I mentioned, that's the pointer to a class that describes that data structure. And it  also generates few helper functions. The first one is for serializing the event. So it serializes  that event to the byte stream. The other one is for getting event class instance. So if  you at runtime need to know what's the structure of that event, you can use that function.  And the last one is the most important, I think, because it's used for registering this  event class in the Hoke Tracer system. So before you use the event class, you need to call  that function. Otherwise, Hoke Tracer doesn't know what's the class and probably will crash  if you forget about this. And basically, so the byte stream that is sent to a client,  the client doesn't know about events, event classes that you defined in your application.  You can define your applications and then have the client that receives the stream but  he doesn't know what events you define. So first, before we send any events, we send  the metadata stream. And it basically contains information about all the event classes that  you registered. So it contains information about the class name, about the class identifier,  and all the fields that are defined in the class. So then, and the next, when we actually  send this stream, so it's below, first we send the class ID and then we send all the  other fields. So then the parser knows, okay, I got this class ID. I know how to parse this  because I already got the metadata stream. So I know that the next field is, for example,  a timestamp field and it has 8 bytes. The next field is a CPU usage. It also has 8 bytes  in the integer. So I know how to parse it. So there's no need to recompile a client if  you add a new class to your library. Timeline. Basically, most of the time, we are using  the global timeline. And the global timeline is not actually the single timeline. It's  a timeline per thread. So we don't have to lock everything. Every time we push something  to a timeline. But it shares listeners. So if you register a listener to one of the timelines  in one of the threads, you will kind of automatically register it for all the other threads for global  timeline. If you really need to create your own timeline, you can do that. But as I said,  it's very, very uncommon use case. So I just recommend you to use global timeline and to  access the global timeline, just call htglobaltimeline.get. It will return a pointer to that timeline.  So our most common use case is to measure the time spent in a function or time spent  in the scope. So we've also introduced some of the helper macros that allow you to measure  the time we spend in the scope. So there's HTTP function that takes timeline pointer  as an argument. And it basically measures the time spent in the full function. And the  output, if you look at the end of the slide, is basically it generates a new event with  the duration, which is how much we spent in that scope. It also adds information about  the thread identifier. So you know that this function was called in this particular thread.  And it also adds the label. So in that case, the HTTP function macros sets the function  name as a label. If you want to trace custom scope, you can do that. It's just another  macro. And plus, you also need to add the custom label for that. And those macros are  only available in EC++ and in the GNU-C compiler because C basically does not have such thing  like a destructor. So we don't have a way to call a call back at the end of the scope.  If you want to measure arbitrary code, not necessarily a scope. There's also a set of  functions start, call stack start string, and the stop. And that measures the time between  calling those two functions. And again, it generates the same event with the label that  you specified here.  So the scope is basically like a scope of the C++ variable. So in this example, the  scope of this trace point HTTP function is the whole function. But here, it only is inside  this curly braces. So, yeah, that's the scope. And so that was more or less about the Hock  Tracer internals. And now, how can you integrate it with your project? If you just download  the source code, do make, make, install, it will install the PKG config file. So you can  just use the PKG config and compile it with your project. This Hock Tracer itself uses  a CMake as a build system. So you can either use it as an external project very easily.  There is an example how to actually do that. So I recommend you to copy just the Hock Tracer.CMake  file to your repository and include it in your project. Or if you installed it as a system  library, you can just use a fine package Hock Tracer with a specific version and then link  it to your project.  And the third option is the one that I actually recommend is that we basically is like SQLite.  It has many files, but at the end, all those files get merged to a single file. So the  same Hock Tracer. We have three files eventually. We have a HT config where you can modify a  configuration. We have a header file and the CPP file where the whole implementation is.  And the CPP file, it can actually be compiled using a C compiler. You might just not have  all the features. But yeah, it's possible to compile it just using the C compiler.  What is the license?  MIT. Sorry. I should mention that before. Yeah, the license is MIT, so you should be  able to use it as you want.  Okay, so demo. The first demo. So I've implemented a sorting algorithm. It sorts 400 numbers.  And turns out that the algorithm is very slow. So if you see, I press enter now. And yeah,  it took a while to sort those numbers. It was only 400 numbers, so it should be like  very quickly. So let's look at the source code. We call the quick sort here. So now  we want to know why is it slow. And in order to know why is it slow, we need to instrument  our code. So I'll show you how to set up the code to basically work with Hock Tracer.  At the beginning, you need to call HT init function. This is very important. It initializes  some internal buffers and registers some base classes. So don't forget about this.  Then we need to create the listener. As I said, the timeline needs to have a listener,  otherwise no one will be able to access those events that we generate. So for this purpose,  we decided to use a file listener. A file listener basically saves all the events to  a file. So we decided to save it to a sort.htdump file. So it will generate the binder file.  There is some check if the listener was created correctly. Then we registered the listener  to the timeline because we only created an instance here, but now we need to actually  register it to the timeline. And we use the global timeline as this is the most convenient  thing to do. This is the callback of the function, and this is like a user data. So the callback  knows the context. And then we initialize our input. So we generate 400 random numbers,  and then we quick sort this array. So if you see, if you look at the functions, we basically  I added to all the functions, I've added this trace point. So whenever function gets called,  we generate the event, how long did it take to execute that particular function. And we  have a few other functions. We have a partition function, we have a quick sort function, and  we also have a swap function. So now I run this. It was already instrumented. So if you  look here, it already generated this sort.htdump file. And now this is a binary file. So if  you try to open this, you will see this is basically the binary file. So what we can  do, we can convert that binary stream to something that we can actually read. And I mentioned  about the HowTracerConverter program. So if you look at this, you see that this program  takes basically three mandatory parameters. The first is format, which is an output format,  which we convert our data to. And it can be either a flame graph or a Chrome trace, or  it can be a debug, which basically prints everything as it is. So we decide to use Chrome  trace. Another parameter is output file. Let's say sort slow JSON and the source. Source  is very important, as the program needs to know where to get the data from. And the source,  this Chrome tracing, HowTracerConverter tool only supports two sources. It's either a file  name or a TCP IP address. In case we, for example, stream the data over the network,  it can listen to a specific port and receive the data. So the source is sort.htdump. And  we run it. It's processing the data. It's completed successfully. Let's see if we have  this. Yes. So we have sort slow. And now we can use the Chrome tracing viewer to actually  see why our application is slow. So I load the JSON file that we generated, and we see  basically this shows the cost tax of our application. The x axis is the time, and we see that basically  since the quick sort is a recursive algorithm, we see that quick sort cost quick sort and  so on and so on. But if we look closer, we basically see that, apart from calling quick  sort, it also calls something different. And if we zoom in more, we see that actually  most of the time we spend in the swap function. I don't know if you can see, but this is  basically it's quick sort. The blue one is a quick sort function. Then the green one  is partition, and the green, the light green is a swap method. So we spent quite a lot  of time in the swap method. So that's probably a function that we should look into. And if  we see how the swap method is implemented, it takes A and B as an argument, and we want  to swap values. So how did we do that? Well, yeah. If someone implemented it as the saving  is to a temporary file, and then it's actually serialized to a string, and then we read it,  that's probably why it's slow. So if we just change the implementation to something like  this. Can you do it without temporary viable? Yeah, there is a trick with XOR, yeah. I would  have to think about it, but yeah, I'm definitely, that's possible. Yeah, so if we fix that,  then probably our trace is going to look completely differently, believe me. Saving temporary  data to a file and read it after that is ways of time. So I'm not going to recompile it  and rerun it. I guess you all believe me that this is going to fix our performance problem  and it's going to be much faster. You need to save the, right? Sorry? Oh. Absolutely.  Thanks. Yeah, live demo never works. So that was the first demo and the second demo. So  as I mentioned, so imagine that previously I showed you how to create your own class,  which was for memory usage and CPU usage tracking. And actually using the Chrome tracing viewer,  we couldn't visualize this. So how can we see like a custom data? So for that, as I  mentioned, we can build our own client to process events. So there is a module Hoke  Tracer for Python. It's very simple to use. You create a client. You start the connection.  So I said, okay, 1-2-7-0-0-1 on port 8765. And then whenever we receive an event here,  we check if the event's name is resource usage event. If it is, then we take the CPU usage,  we take a memory usage, we print CPU usage, and we draw memory usage on the graph. I'm  not a Python developer, so maybe that's not the best way to graph something, but it works.  And our code is very, very similar to the previous one. Instead of doing the file listener,  we created the TCP listener that basically creates a server that streams the data. We  also had this class that I showed before. So this is the definition of our class. And  then every second here in the while loop, we allocate some amount of memory, half of  the kilobyte. Then we make our CPU busy and we sleep for one second. And we report resource  usage. So let's see how do we report the resource usage. There is a macro, ht-timeline push  event. It takes a timeline as a first argument. Then the events type, then values of that  event that we want to attach to this event. And then we push. And that pushes it. We also  flush the timeline manually instead of waiting for the buffer to get full, because we want  to have the data immediately in the client, so it's kind of smooth drawing. So we can  run the demo. I run the resource usage. And I run my Python client. And you can see that  it draws the memory usage here. So basically what we've done, we made our custom kind of  event converter. And we decided how we visualized the data in Python just in, I don't know,  20 lines of code. And also the CPU usage is printed here. So it's around, yeah, 1, 2,  3, 4, 5, 6, 7, 8. Yeah, so that was the second demo. There's a bunch of things that we want  to do for the future. There are some missing features. Like, we don't support floating  point numbers at the moment. Optional fields is also something we want to have. So for  example, you have a class with some event, with some fields, but sometimes some of the  fields should not be included. So that would be nice to have. We also want to have more  converters in the whole tracer converter. Like, a CTF is a common trace format supported  by TraceCompass or LTTNG scope viewer. So it would be nice to have. We use that profiler  also for profiling C++ and Lua or JavaScript stack at the same time. But Lua bindings and  JavaScript bindings are not open sourced. So we also want to do that. And lots of documentation  improvements. Even though the documentation is quite okay, I see we can improve that more.  So yeah, if you want to help, just go to hocktracer.org slash community. And you will find how to  contact us. And we can work together. Now there's a bunch of links. There was also Rust  bindings. Alex had a talk today afternoon. So if you search for FOSDEM 2019 profiling  Rust, you should find that presentation. It was pretty cool. Okay, thank you.  I think we have. Yes.  So that depends. Creating new event, having ability to create a new event class in the  other language might be quite tricky. But assuming that you already have all the event  classes defined in your C or C++ code base and you run your other language on top of  that, it should be fairly easy. Because you just need to expose. So for measuring time,  you need to expose to function start and stop. Plus maybe start hocktracer or start register  listener. So it should be fairly easy. For Rust, I don't know how much work that was  for Alex. Yeah.  And the interfacing. Yeah.  Yeah. That depends what you want to do. But if you just want to measure time, it should  be fairly easy. Yeah.  Thank you. Thank you.
Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_0_179.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_180_551.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_552_1259.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_1260_8819.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_8820_12119.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_12120_14207.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_14208_17579.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_17580_19475.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_19476_20591.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_20592_23195.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_23196_23243.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_23244_23807.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_23808_25979.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_25980_26231.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_26232_26339.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_26340_26387.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_26388_26435.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_26436_26507.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_26508_27035.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_27036_29903.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_29904_32699.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_32700_32759.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_32760_32819.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_32820_33431.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_33432_33479.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_33480_33539.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_33540_34223.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_34224_35915.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_35916_35963.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_35964_36155.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_36156_37043.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_37044_38255.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_38256_39083.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_39084_39923.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_39924_39995.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_39996_40655.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_40656_42251.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_42252_44915.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_44916_44963.png Profiling_Low-End_Platforms_using_HawkTracer_Profiler_SczRceAa4OE_44964_45077.png