Q:

How to measure mutex contention?

I have some threaded code using PThreads on Linux that, I suspect, is suffering from excessive lock contention. What tools are available for me to measure this?
Solaris has DTrace and plockstat. Is there something similar on Linux? (I know about a recent DTrace port for Linux but it doesn't seem to be ready for prime time yet.)

A:

mutrace is the tool:
http://0pointer.de/blog/projects/mutrace.html
Its easy to build, install and use.

A:

After not having much luck with SystemTap, I decided to try and use the DTrace Linux port with some success, despite the lack of a plockstat provider. The following DTrace script is not quite a plockstat replacement but it managed to show me some of the information I was after.
#!/usr/sbin/dtrace -s 

/* Usage: ./futex.d '"execname"' */

long total;

END
{
    printf("total time spent on futex(): %ldms\n", total);
}

/* arg1 == 0 means FUTEX_WAIT */
syscall::futex:entry
/execname == $1 && arg1 == 0/
{
    self->start = timestamp;
}

syscall::futex:return
/self->start/
{
    this->elapsed = (timestamp - self->start) / 1000000;
    @[execname] = quantize(this->elapsed);
    total += this->elapsed;
    self->start = 0;
}

Here's an example using the above DTrace script to measure time spent in FUTEX_WAIT for a simple test program from this DTrace article.
$ ./futex.d '"mutex-test"'
dtrace: script './futex.d' matched 3 probes
^C
CPU     ID                    FUNCTION:NAME
  1      2                             :END total time spent on futex(): 11200ms

  mutex-test                                        
           value  ------------- Distribution ------------- count    
             128 |                                         0        
             256 |@@@@@@@@@@@@@@@@@@@@                     1        
             512 |                                         0        
            1024 |                                         0        
            2048 |                                         0        
            4096 |                                         0        
            8192 |@@@@@@@@@@@@@@@@@@@@                     1        
           16384 |                                         0        

Definitely not great, but at least it's a starting point.

A:

valgrind latest versions has a lock contention and lock validation tools:
http://valgrind.org/docs/manual/drd-manual.html
Which is great if you can produce the issue under Valgrind (it effects code run time speed) and have enough memory to run Valgrind.
For other uses, the more hard core Linux Trace Toolkit NG is recommended:
http://ltt.polymtl.ca/
Cheers,
Gilad

