<commit_before>CacheSplain - A tool for helping with Memcached
=============
*** TBA: Travis build status ***

This readme provides a basic set of information about CacheSplain. CacheSplain is MIT licensed. Please see [LICENSE.txt](https://github.com/kog/cachesplain/blob/master/LICENSE.txt) for details.

It helps if you have .NET already installed (Mono, MS CLR), libpcap installed (via WireShark or otherwise) and have some familiarity with the [Memcached binary protocol](https://code.google.com/p/memcached/wiki/BinaryProtocolRevamped).


What is this?
--

CacheSplain is a utility, written in C#, that helps you make sense of what your Memcache clients/servers are doing at a network level. CacheSplain takes either a live network interface or a prior capture via [tcpdump](http://www.tcpdump.org/), [WireShark](https://www.wireshark.org/), or other such utility and analyzes it for Memcached binary protocol traffic. Each TCP packet is inspected and broken down into its individual operations, then reconstituted into lightweight data structures that make the traffic easier to understand. Once the traffic is broken down, it can be stored in something like [MongoDB](http://www.mongodb.org/) ***(TBD)*** or piped into something like [LogStash](http://logstash.net/) ***(TBD)***.

While it's true there are a number of Memcached-related utilities, they all seemed to fall short of what I was looking for:

 * [WireShark](https://www.wireshark.org/) - Has a Memcached protocol analyzer, but is somewhat cumbersome, has trouble with pipelined operations (multiple operations in a single TCP packet). Can be rather slow with larger captures, can be somewhat painful on a Mac (though getting better).
 * [Etsy's McTop](https://github.com/etsy/mctop) - Doesn't support the binary protocol, extremely limited, tends to drop packets under load. Incredibly limited in terms of what it tracks.
 * [Memcache Top](https://code.google.com/p/memcache-top/) - does what it says on the tin... it's Top, which doesn't really provide details as to ***why*** you're doing what you're doing.
 * [MemKeys](https://github.com/bmatheny/memkeys) - didn't quite do what I needed, and since it's written in C++ I thought I'd avoid hacking on it...
 * [The official links](https://code.google.com/p/memcached/wiki/NewLinks) - A lot of these were different types of monitoring or incomplete, or had significant caveats - many are just plain abandoned.

McTop was actually the most useful project (for my needs) I'd found before I started writing CacheSplain, and I'd originally hacked it to support the binary protocol using a combination of the pull requests on the project. This still dropped a lot of packets, so it seemed that it was time to do what all good novices do and write something from scratch...


Why would you write this in C#?
--

So why choose C#? The short answer is that I wasn't particularly happy with the PCAP library support in Java: both of the major contenders I tried had different issues. Ruby was out due to performance (not... that I'm great with it anyway), and I didn't really want to deal with either C or C++.

That's when I remembered that [ShapPcap](http://sourceforge.net/projects/sharppcap/) was supposed to be a pretty decent, fairly well performing library. It's cross platform and works quite well on Mono. SharpPcap supports filtering, it supports PCAP files for both reading and writing, and it's got some great documentation on [CodeProject](http://www.codeproject.com/Articles/12458/SharpPcap-A-Packet-Capture-Framework-for-NET).

I have yet to try this on incredibly large dumps, so it's possible that it has issues scaling to gigs and terabytes of data. So far it's handled my ~500mb dumps just fine. I would highly recommend using tcpdump in your production environment and replaying the files in CacheSplain.


Build Instructions
--

**(TODO: Try and remove WireShark install dependency)**

Before you try and build or run CacheSplain you should install [WireShark](https://www.wireshark.org/) for your platform. This should ensure that you have the appropriate libraries (namely libpcap). This may be addressed at a later time, hopefully working out of the box.

Once you have WireShark installed, you should be able to build the project using an IDE such as Visual Studio or MonoDevelop, or by using something like XBuild on Mono. The build should work out of the box **(TBA: Travis link)**, and you should be able to run the resulting executable. Please note that the App.config is only used for configuring NLog, and all arguments are passed on the command line.

If you are planning on running this on anything but Windows, make sure that you have [Mono](http://www.mono-project.com/) installed. If you can't find a package for your OS/distro, you can always compile your own from source easily enough from their [GitHub repo](https://github.com/mono/mono).


###### Building via XBuild (Mono):
```
nuget restore cachesplain.sln
xbuild cachesplain.sln /target:Rebuild /p:Configuration=release /p:PlatformTarget=x86
```

Don't have NuGet for some reason? Grab the Command-Line Utility from [NuGet.org](http://docs.nuget.org/docs/start-here/installing-nuget). You'll have to prefix it with "mono" (which should hopefully be on your path...).<commit_msg>Replace Travis link placeholders with real URLs<commit_after>CacheSplain - A tool for helping with Memcached
=============
![Build Status](https://travis-ci.org/kog/cachesplain.svg?branch=master)

This readme provides a basic set of information about CacheSplain. CacheSplain is MIT licensed. Please see [LICENSE.txt](https://github.com/kog/cachesplain/blob/master/LICENSE.txt) for details.

It helps if you have .NET already installed (Mono, MS CLR), libpcap installed (via WireShark or otherwise) and have some familiarity with the [Memcached binary protocol](https://code.google.com/p/memcached/wiki/BinaryProtocolRevamped).


What is this?
--

CacheSplain is a utility, written in C#, that helps you make sense of what your Memcache clients/servers are doing at a network level. CacheSplain takes either a live network interface or a prior capture via [tcpdump](http://www.tcpdump.org/), [WireShark](https://www.wireshark.org/), or other such utility and analyzes it for Memcached binary protocol traffic. Each TCP packet is inspected and broken down into its individual operations, then reconstituted into lightweight data structures that make the traffic easier to understand. Once the traffic is broken down, it can be stored in something like [MongoDB](http://www.mongodb.org/) ***(TBD)*** or piped into something like [LogStash](http://logstash.net/) ***(TBD)***.

While it's true there are a number of Memcached-related utilities, they all seemed to fall short of what I was looking for:

 * [WireShark](https://www.wireshark.org/) - Has a Memcached protocol analyzer, but is somewhat cumbersome, has trouble with pipelined operations (multiple operations in a single TCP packet). Can be rather slow with larger captures, can be somewhat painful on a Mac (though getting better).
 * [Etsy's McTop](https://github.com/etsy/mctop) - Doesn't support the binary protocol, extremely limited, tends to drop packets under load. Incredibly limited in terms of what it tracks.
 * [Memcache Top](https://code.google.com/p/memcache-top/) - does what it says on the tin... it's Top, which doesn't really provide details as to ***why*** you're doing what you're doing.
 * [MemKeys](https://github.com/bmatheny/memkeys) - didn't quite do what I needed, and since it's written in C++ I thought I'd avoid hacking on it...
 * [The official links](https://code.google.com/p/memcached/wiki/NewLinks) - A lot of these were different types of monitoring or incomplete, or had significant caveats - many are just plain abandoned.

McTop was actually the most useful project (for my needs) I'd found before I started writing CacheSplain, and I'd originally hacked it to support the binary protocol using a combination of the pull requests on the project. This still dropped a lot of packets, so it seemed that it was time to do what all good novices do and write something from scratch...


Why would you write this in C#?
--

So why choose C#? The short answer is that I wasn't particularly happy with the PCAP library support in Java: both of the major contenders I tried had different issues. Ruby was out due to performance (not... that I'm great with it anyway), and I didn't really want to deal with either C or C++.

That's when I remembered that [ShapPcap](http://sourceforge.net/projects/sharppcap/) was supposed to be a pretty decent, fairly well performing library. It's cross platform and works quite well on Mono. SharpPcap supports filtering, it supports PCAP files for both reading and writing, and it's got some great documentation on [CodeProject](http://www.codeproject.com/Articles/12458/SharpPcap-A-Packet-Capture-Framework-for-NET).

I have yet to try this on incredibly large dumps, so it's possible that it has issues scaling to gigs and terabytes of data. So far it's handled my ~500mb dumps just fine. I would highly recommend using tcpdump in your production environment and replaying the files in CacheSplain.


Build Instructions
--

**(TODO: Try and remove WireShark install dependency)**

Before you try and build or run CacheSplain you should install [WireShark](https://www.wireshark.org/) for your platform. This should ensure that you have the appropriate libraries (namely libpcap). This may be addressed at a later time, hopefully working out of the box.

Once you have WireShark installed, you should be able to build the project using an IDE such as Visual Studio or MonoDevelop, or by using something like XBuild on Mono. The build should work out of the box (check the build status top of the page), and you should be able to run the resulting executable. Please note that the App.config is only used for configuring NLog, and all arguments are passed on the command line.

If you are planning on running this on anything but Windows, make sure that you have [Mono](http://www.mono-project.com/) installed. If you can't find a package for your OS/distro, you can always compile your own from source easily enough from their [GitHub repo](https://github.com/mono/mono).


###### Building via XBuild (Mono):
```
nuget restore cachesplain.sln
xbuild cachesplain.sln /target:Rebuild /p:Configuration=release /p:PlatformTarget=x86
```

Don't have NuGet for some reason? Grab the Command-Line Utility from [NuGet.org](http://docs.nuget.org/docs/start-here/installing-nuget). You'll have to prefix it with "mono" (which should hopefully be on your path...).
{'fraction_non_alphanumeric': 0.08281748264865879, 'fraction_numerical': 0.0018758206715438004, 'mean_word_length': 4.527734577501296, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 21, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4450499', 'n_tokens_mistral': 2872, 'n_tokens_neox': 2740, 'n_words': 1458}