<commit_before>
#NOT PRODUCTION READY
There are missing features and known problems in this code.  Wait a week or two and I'll have a v0.1 that should be production-ready.

# gopassivedns
Network-based DNS logging in Go

##Summary

a network-capture-based DNS logger, inspired by https://github.com/gamelinux/passivedns.  It uses gopacket to deal with libpcap and packet processing.  It outputs JSON logs.  It is intended to deal with high volume query capture in environments with anywhewre from one to hundreds of DNS resolvers.  Future development work will add a native kafka output channel (in addition to file logging).     

###Why not just use resolver query logging?
Resolver support for query logging, including both the question and answer is spotty at best.  One of the most-deployed DNS servers, BIND, doesn't support it at all.  Others, like Windows DNS, have really horrible log formats.  Additionally, network-based logging will catch queries sent directly to remote servers (e.g. Google DNS) from your clients.

##Usage

   * -i dev
   * -b bpf filter
   * -p pcap file
   * -l log file

You must supply either -i or -p.  

There are known issues with goroutines and the standard daemonize process (https://github.com/golang/go/issues/227), so I strongly recommend you use one of the methods detaild here: http://stackoverflow.com/questions/10067295/how-to-start-a-go-program-as-a-daemon-in-ubuntu to run this process as a daemon using system tools.

##Deployment Guide

###Where do I deploy this?
You have 3 choices: deploy it on your resolver(s) or deploy it on your gateway(s) or both.  Deploying on your resolvers is good because you will get the IP address of the client that sent the origianl request.  You can also see the upstream leg of the request (from the resolver to the next resolver in the chain), unless you tune your BPF filter to ignore that leg.  Deploying on your gateways means you don't see the client -> internal resolver leg, so it can be hard to tie a request back to a specific client.  On the other hand, you'll see requests that bypass your internal resolvers.  You will also, of course, see the queries coming from the resolver to whatever upstream resolver it uses.  In an ideal world, I would deploy this tool on each of my internal resolvers and on a tap on my gateways.  The internal resolvers would have a BPF filter such that it ignores the upstream leg of the query, and the gateway would not ignore anything.

###What should I do with the results?
Right now, I'd recommend using logstash to ship the logs to an elasticsearch cluster.  All the logs are JSON, so this should be pretty easy.  In the future, I will add native kafka support that should make that easier.  I would also suggest using something like HDFS for long term storage and bulk analysis.  DNS queries are an amazing source of internal data!


##Build and install
   * clone this repo
   * 'go get'
   * 'go build -o gopassivedns'  (the -o is really just being careful, assuming you cloned the repo you shouldn't need it)
   * 'cp gopassivedns /some/path/to/gopassivedns'

<commit_msg>doc update<commit_after>
#BETA-ISH QUALITY
There are missing features in this code.  I don't think it is going to crash and burn at this point, but no promises on performance or long-term stability in the face of actual dns lookups.

# gopassivedns
Network-based DNS logging in Go

##Summary

a network-capture-based DNS logger, inspired by https://github.com/gamelinux/passivedns.  It uses gopacket to deal with libpcap and packet processing.  It outputs JSON logs.  It is intended to deal with high volume query capture in environments with anywhewre from one to hundreds of DNS resolvers.  Future development work will add a native kafka output channel (in addition to file logging).     

###Why not just use resolver query logging?
Resolver support for query logging, including both the question and answer is spotty at best.  One of the most-deployed DNS servers, BIND, doesn't support it at all.  Others, like Windows DNS, have really horrible log formats.  Additionally, network-based logging will catch queries sent directly to remote servers (e.g. Google DNS) from your clients.

##Usage

   * -i dev
   * -b bpf filter
   * -p pcap file
   * -l log file

You must supply either -i or -p.  

There are known issues with goroutines and the standard daemonize process (https://github.com/golang/go/issues/227), so I strongly recommend you use one of the methods detaild here: http://stackoverflow.com/questions/10067295/how-to-start-a-go-program-as-a-daemon-in-ubuntu to run this process as a daemon using system tools.

##Deployment Guide

###Where do I deploy this?
You have 3 choices: deploy it on your resolver(s) or deploy it on your gateway(s) or both.  Deploying on your resolvers is good because you will get the IP address of the client that sent the origianl request.  You can also see the upstream leg of the request (from the resolver to the next resolver in the chain), unless you tune your BPF filter to ignore that leg.  Deploying on your gateways means you don't see the client -> internal resolver leg, so it can be hard to tie a request back to a specific client.  On the other hand, you'll see requests that bypass your internal resolvers.  You will also, of course, see the queries coming from the resolver to whatever upstream resolver it uses.  In an ideal world, I would deploy this tool on each of my internal resolvers and on a tap on my gateways.  The internal resolvers would have a BPF filter such that it ignores the upstream leg of the query, and the gateway would not ignore anything.

###What should I do with the results?
Right now, I'd recommend using logstash to ship the logs to an elasticsearch cluster.  All the logs are JSON, so this should be pretty easy.  In the future, I will add native kafka support that should make that easier.  I would also suggest using something like HDFS for long term storage and bulk analysis.  DNS queries are an amazing source of internal data!


##Build and install
   * clone this repo
   * 'go get'
   * 'go build -o gopassivedns'  (the -o is really just being careful, assuming you cloned the repo you shouldn't need it)
   * 'cp gopassivedns /some/path/to/gopassivedns'


{'fraction_non_alphanumeric': 0.051269765213224724, 'fraction_numerical': 0.004152691263376458, 'mean_word_length': 3.8769470404984423, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '549995', 'n_tokens_mistral': 1725, 'n_tokens_neox': 1636, 'n_words': 983}