<story><title>Google XRay: A Function Call Tracing System [pdf]</title><url>https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45287.pdf</url></story><parent_chain><item><author>brendangregg</author><text>NOT ANOTHER TRACER!!&lt;p&gt;I&amp;#x27;m sure it&amp;#x27;s impressive engineering work, but why oh why...&lt;p&gt;How does it compare to Linux uprobes, which are built into Linux mainline? Bear in mind there are different front ends for uprobes (ftrace, perf_events, bcc, ...), and these are also still in development, so if one lacked certain features they needed, such features could be added. There&amp;#x27;s been a LOT of work in this area in the past 6 months, as well (see lkml).&lt;p&gt;If the goal was lowest performance, then why compile with no-op sleds (&amp;quot;negligible overhead&amp;quot;) instead of using dynamic tracing (literally &amp;quot;zero overhead&amp;quot;)? Or, if the existing kernel-based dynamic tracers benchmarked poorly, then why not something like LTTng?&lt;p&gt;How does it compare to DTrace, as well? (Doesn&amp;#x27;t Google have some FreeBSD?).&lt;p&gt;All the tracers I mentioned can not only do dynamic tracing, but also instrument all user and kernel code, without special recompilation.</text></item></parent_chain><comment><author>davidtgoldblatt</author><text>Most of XRay was written before uprobes was merged into the Linux kernel (and well before such kernels were widely available).&lt;p&gt;I don&amp;#x27;t think any of the alternatives you mentioned are Pareto superior to XRay when considering all of &amp;quot;speed while tracing&amp;quot;, &amp;quot;speed while not tracing&amp;quot;, and &amp;quot;flexibility&amp;quot;.&lt;p&gt;E.g.:&lt;p&gt;- In &amp;quot;speed while tracing&amp;quot;, anything that takes a context switch per traced function will probably be dramatically slower. Even if there&amp;#x27;s some fast dispatch mechanism you have in mind that I&amp;#x27;m not familiar with when you say dynamic tracing, if it doesn&amp;#x27;t insert the moral equivalent of a nop-sled, it will have to either choose between logging the whole PC (spending data, which means spending RAM and disk time) or figuring out how to map it to a function-specific unique int (spending cycles).&lt;p&gt;- In &amp;quot;speed while not tracing&amp;quot;, anything much more expensive than nop-sleds will be too slow to run in production.&lt;p&gt;- Anything that doesn&amp;#x27;t have a compile time component probably won&amp;#x27;t be able to completely hook functions that get inlined, or whose source you aren&amp;#x27;t able to change, won&amp;#x27;t be able to pick out information the runtime wants to summarize from function arguments, etc.&lt;p&gt;To me, the neat thing about XRay isn&amp;#x27;t so much the &amp;quot;function patching&amp;quot; aspect, except insofar as it serves as a mechanism to execute arbitrary code at function entry or exit in a way that&amp;#x27;s runtime-customizable and very low overhead when you want it to be.</text></comment>
<story><title>Google XRay: A Function Call Tracing System [pdf]</title><url>https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45287.pdf</url></story><parent_chain><item><author>brendangregg</author><text>NOT ANOTHER TRACER!!&lt;p&gt;I&amp;#x27;m sure it&amp;#x27;s impressive engineering work, but why oh why...&lt;p&gt;How does it compare to Linux uprobes, which are built into Linux mainline? Bear in mind there are different front ends for uprobes (ftrace, perf_events, bcc, ...), and these are also still in development, so if one lacked certain features they needed, such features could be added. There&amp;#x27;s been a LOT of work in this area in the past 6 months, as well (see lkml).&lt;p&gt;If the goal was lowest performance, then why compile with no-op sleds (&amp;quot;negligible overhead&amp;quot;) instead of using dynamic tracing (literally &amp;quot;zero overhead&amp;quot;)? Or, if the existing kernel-based dynamic tracers benchmarked poorly, then why not something like LTTng?&lt;p&gt;How does it compare to DTrace, as well? (Doesn&amp;#x27;t Google have some FreeBSD?).&lt;p&gt;All the tracers I mentioned can not only do dynamic tracing, but also instrument all user and kernel code, without special recompilation.</text></item></parent_chain><comment><author>deadmutex</author><text>Unfortunately, I think the atleast one of developers was not aware of uprobes.  I saw a post from the developer that said he was looking into uprobes after this point, and he was not aware of it.&lt;p&gt;He seems like a nice guy, and just seems like an oversight.</text></comment>