XRay: A Function Call Tracing System
Debugging high throughput, low latency C/C+ systems in production is hard. At Google we developed XRay, a function call tracing system that allows Google engineers to get accurate function call traces with negligible overhead when off and moderate overhead when on, suitable for services deployed in production. XRay enables efficient function call entry/exit logging with high accuracy timestamps, and can be dynamically enabled and disabled. This white paper describes the XRay tracing system and its implementation. It also describes future plans with open sourcing XRay and engaging open source communities. Function Call Tracing in Production At Google we run distributed systems handling massive amounts of concurrent requests that do everything from serving a webpage to storing bytes on disks. Understanding the performance of these complex systems is difficult given the abstractions involved, asynchronous programming with callbacks, and even interaction between threads synchronising amongst each other. It can be especially difficult to track down what affects performance in rare events, which can be rare because they occur at the latency tail or 1 because they are outlier requests to the service. Seeing what happens "at the tail" allows software engineers to diagnose the performance of systems in these rare events that have an impact on the overall performance of the system. The question then becomes how do you find these "needle in a haystack" events while running in production? Time based traces let developers track down these issues when standard approaches such as sampling, logging, and aggregate statistics fail. What kind of time based trace data would be useful? Ideally, we'd like to get non sampled function call traces with high precision timestamps from production servers that tell us which threads are running what code and when. However, instrumenting all functions is usually too expensive for production use. To deploy in production, we require that: The cost is acceptable when tracing and barely measurable when not tracing. Instrumentation is automatic and directed towards functions that are important for understanding the binary's execution time. Tracing is efficient in both space and time only recording what is required and what matters. In addition to the above, we also require: Tracing is configurable with thresholds for storage (how much memory to use) and accuracy (whether to log everything or only function calls taking at least some amount of time) Tracing does not require changes to the operating system nor super user privileges. Tracing can be turned on and off dynamically without having to restart the server. 1 For more on why optimising tail latency matters, refer to " The Tail at Scale " by Jeff Dean and Luiz Barroso, published to the Communications of the ACM, vol. 56 (2013) pp. 74 80. 1