<commit_before>---
layout: post
title: Collecting Hardware Performance Counters with BenchmarkDotNet
excerpt_separator: <!--more-->
---

## Introduction

BenchmarkDotNet is a powerful .NET library for benchmarking ([more about it](https://benchmarkdotnet.org/)). This post describes how you can collect [Hardware Performance Counters](https://en.wikipedia.org/wiki/Hardware_performance_counter) with BenchmarkDotNet. If you want to learn about the ETW internals behind it then you might find [this post](https://adamsitnik.com/Hardware-Counters-ETW/) useful.

## The Story

Some time ago I have noticed that software engineers who are working on the `Span<T>` are struggling with some benchmarking issues. With problems like unstable results, which BenchmarkDotNet solves. So I [asked](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-284959960) why don't they use BenchmarkDotNet?

And then <NAME> [replied](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-285082017):

> Yeah, we have. In general I love BenchmarkDotNet and use it in many cases. It does have better front end reporting capabilities and is more mature from reliability and statistical analysis perspective. But it seems like **BenchmarkDotNet does not support measuring instructions retired, cache misses, and branch mispredictions that we often find very useful**. Would be great to combine the efforts/features :-)
<!--more-->


And <NAME> [mentioned](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-285083415) that `xunit-performance` can do it:

> @KrzysztofCwalina, for Instructions Retired, you need [assembly: MeasureInstructionsRetired]. 
> 
> I also know cache misses and branch mispredictions are also supported, as they are displayed on the internal benchmark comparison site (I'm not sure how to enable those ones though).

I could finally add a new feature to BenchmarkDotNet that was related to performance, not the MSBuild/project.json stuff ;) So I just downloaded the source codes of [xunit-performance](https://github.com/Microsoft/xunit-performance) and [PerfView](https://github.com/microsoft/perfview) and dig into the details. `Microsoft.Diagnostics.Tracing.TraceEvent` is not OSS yet, so I used my favorite .NET decompiler [ILSpy](https://ilspy.net/) to get the full picture. 

## Limitations

What we have today comes with following limitations:

* Accuracy is limited by sampling frequency and additional code performed in the benchmarked process by our Engine. It's good but not perfect. For more accurate results you should use some profilers like Intel VTune Amplifier.
* Windows 8+ only
* No Hyper-V (Virtualization) support
* Requires to run as Admin (ETW Kernel Session)
* No `InProcessToolchain` support ([#394](https://github.com/dotnet/BenchmarkDotNet/issues/394))
* The initial list of available Counters:
	* Timer,
	* TotalIssues,
	* BranchInstructions,
	* CacheMisses,
	* BranchMispredictions,
	* TotalCycles,
	* UnhaltedCoreCycles,
	* InstructionRetired,
	* UnhaltedReferenceCycles,
	* LlcReference,
	* LlcMisses,
	* BranchInstructionRetired,
	* BranchMispredictsRetired

## How to use it?

You need to install latest `BenchmarkDotNet.Diagnostics.Windows` package.

It can be enabled in three ways:

* Use the new attribute (apply it on a class that contains Benchmarks):

```cs
[HardwareCounters(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions)]
public class TheClassThatContainsBenchmarks { /* benchmarks go here */ }
```

* Tell your custom config to use it:

```cs
private class CustomConfig : ManualConfig
{
    public CustomConfig()
    {
        Add(Job.Default);
        Add(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions);
    }
}
```

* Passing `--counters` command line argument to `BenchmarkSwitcher`. Example: `--counters BranchInstructions+BranchMispredictions`


## Sample


Let's use our new diagnoser to answer the famous *"[Why is it faster to process a sorted array than an unsorted array?
](https://stackoverflow.com/questions/11227809/why-is-it-faster-to-process-a-sorted-array-than-an-unsorted-array)*".

```cs
class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Cpu_BranchPerdictor>();
}

[HardwareCounters(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions)]
public class Cpu_BranchPerdictor
{
    private const int N = 32767;
    private readonly int[] sorted, unsorted;

    public Cpu_BranchPerdictor()
    {
        var random = new Random(0);<commit_msg>fix a typo, thanks to <NAME> for pointing this out<commit_after>---
layout: post
title: Collecting Hardware Performance Counters with BenchmarkDotNet
excerpt_separator: <!--more-->
---

## Introduction

BenchmarkDotNet is a powerful .NET library for benchmarking ([more about it](https://benchmarkdotnet.org/)). This post describes how you can collect [Hardware Performance Counters](https://en.wikipedia.org/wiki/Hardware_performance_counter) with BenchmarkDotNet. If you want to learn about the ETW internals behind it then you might find [this post](https://adamsitnik.com/Hardware-Counters-ETW/) useful.

## The Story

Some time ago I have noticed that software engineers who are working on the `Span<T>` are struggling with some benchmarking issues. With problems like unstable results, which BenchmarkDotNet solves. So I [asked](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-284959960) why don't they use BenchmarkDotNet?

And then <NAME> [replied](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-285082017):

> Yeah, we have. In general I love BenchmarkDotNet and use it in many cases. It does have better front end reporting capabilities and is more mature from reliability and statistical analysis perspective. But it seems like **BenchmarkDotNet does not support measuring instructions retired, cache misses, and branch mispredictions that we often find very useful**. Would be great to combine the efforts/features :-)
<!--more-->


And <NAME> [mentioned](https://github.com/dotnet/corefxlab/pull/1278#issuecomment-285083415) that `xunit-performance` can do it:

> @KrzysztofCwalina, for Instructions Retired, you need [assembly: MeasureInstructionsRetired]. 
> 
> I also know cache misses and branch mispredictions are also supported, as they are displayed on the internal benchmark comparison site (I'm not sure how to enable those ones though).

I could finally add a new feature to BenchmarkDotNet that was related to performance, not the MSBuild/project.json stuff ;) So I just downloaded the source codes of [xunit-performance](https://github.com/Microsoft/xunit-performance) and [PerfView](https://github.com/microsoft/perfview) and dig into the details. `Microsoft.Diagnostics.Tracing.TraceEvent` is not OSS yet, so I used my favorite .NET decompiler [ILSpy](https://ilspy.net/) to get the full picture. 

## Limitations

What we have today comes with following limitations:

* Accuracy is limited by sampling frequency and additional code performed in the benchmarked process by our Engine. It's good but not perfect. For more accurate results you should use some profilers like Intel VTune Amplifier.
* Windows 8+ only
* No Hyper-V (Virtualization) support
* Requires to run as Admin (ETW Kernel Session)
* No `InProcessToolchain` support ([#394](https://github.com/dotnet/BenchmarkDotNet/issues/394))
* The initial list of available Counters:
	* Timer,
	* TotalIssues,
	* BranchInstructions,
	* CacheMisses,
	* BranchMispredictions,
	* TotalCycles,
	* UnhaltedCoreCycles,
	* InstructionRetired,
	* UnhaltedReferenceCycles,
	* LlcReference,
	* LlcMisses,
	* BranchInstructionRetired,
	* BranchMispredictsRetired

## How to use it?

You need to install latest `BenchmarkDotNet.Diagnostics.Windows` package.

It can be enabled in three ways:

* Use the new attribute (apply it on a class that contains Benchmarks):

```cs
[HardwareCounters(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions)]
public class TheClassThatContainsBenchmarks { /* benchmarks go here */ }
```

* Tell your custom config to use it:

```cs
private class CustomConfig : ManualConfig
{
    public CustomConfig()
    {
        Add(Job.Default);
        Add(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions);
    }
}
```

* Passing `--counters` command line argument to `BenchmarkSwitcher`. Example: `--counters BranchInstructions+BranchMispredictions`


## Sample


Let's use our new diagnoser to answer the famous *"[Why is it faster to process a sorted array than an unsorted array?
](https://stackoverflow.com/questions/11227809/why-is-it-faster-to-process-a-sorted-array-than-an-unsorted-array)*".

```cs
class Program
{
    static void Main(string[] args) => BenchmarkRunner.Run<Cpu_BranchPredictor>();
}

[HardwareCounters(HardwareCounter.BranchMispredictions, HardwareCounter.BranchInstructions)]
public class Cpu_BranchPredictor
{
    private const int N = 32767;
    private readonly int[] sorted, unsorted;

    public Cpu_BranchPredictor()
    {
        var random = new Random(0);
{'fraction_non_alphanumeric': 0.09008710993494322, 'fraction_numerical': 0.013231888852133642, 'mean_word_length': 4.998677248677248, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 26, 'https://': 22, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6432106', 'n_tokens_mistral': 2724, 'n_tokens_neox': 2558, 'n_words': 924}