<commit_before># pcapdaemon

## Description
This is a daemon that will subscribe to a redis pub/sub channel for requests to capture.  It will capture and then optionally upload to Cloudshark or save to the local filesystem.  It could really be adapted to upload anywhere but the key was that I wanted to be able to trigger captures based on any number of events (traps, log events etc) via a lightweight mechanism.  A design goal was to have it capture into a buffer in memory and post the buffer without adding any kind of filesystem/io dependency.  

## Options
    -cshost string          cloudshark host (default "localhost")
    -csscheme string        cloudshark scheme http|https (default "https")
    -cstoken string         cloudshark api token (default "<PASSWORD>")
    -csport int             cloudshark port
    -redischannel string    Redis channel to subscribe to.  Default capture (default "capture")
    -redisnode string       Hostname|IP of redis server.  Default localhost (default "127.0.0.1")
    -redisport int          Port of redis server. Default 6379 (default 6379)
    -upload                 Upload pcap
    -writelocal             Write pcap files locally.  Requires setting destdir
    -destdir                Directory to store locally written pcap files in
    -maxpackets             Maximum number of packets per capture.  Default 50000.
    -config string          Path to configuration file.  No default.
    
## Message format
    {
        "node": "node name",
        "nodere": "node regex",
        "interface": ["bond1","bond2"],
        "alias": ["local","public"],
        "tags": "blah,tagme,stuff",
        "bpf": "dst ip 10.0.0.1",
        "customer": "importantcustomer",
        "snap": 1500,
        "packets": 50,
        "alertid": 655443,
        "timeout": 15
		"folder": "myfolder",
		"bucket": "mybucket",
		"acl": "public-read",
		"region": "us-east-1",
		"endpoint": "s3.amazonaws.com",
		"encryption": false
    }
    
 * node - node name to capture on (exact match) Use either node or nodere but not both and one is required. Keyword: "any" matches any host
 * nodere - node regex to capture on.  Use either node or nodere but not both and one is required.
 * interface - an array of interfaces to dump on. (Must supply either interface or alias but not both)
 * alias - an array of interface aliases to dump on.  (Must supply either interface or alias but not both)
 * tags - additional metadata for the capture file.  Comma separated list.
 * bpf - A filter string to capture on
 * customer - Additional metadata field ( not required )
 * snap - Snaplength or amount to capture into the packet.  Integer.
 * packets - Number of packets to capture.  Integer.
 * alertid - An integer ID for the event or alert or whatever you're tracking (not required)
 * timeout - Number of seconds to let capture last should the number of packets not get hit.  Integer.
 * folder - S3 folder inside your bucket // S3 ONLY
 * bucket - S3 bucket // S3 ONLY
 * folder - S3 ACL // S3 ONLY
 * folder - S3 Endpoint // S3 ONLY
 * encryption - S3 Server side encryption AES256 // S3 ONLY

## Configuration File format (toml)
 * Defining interfaces is optional.  You only need to do it if you'd like to use an alias.  The basic use case is to group catpure interfaces across several nodes that may have different physical names for a variety of reasons.
``` 
## Config file
[general]
maxpackets  = 50000
writelocal  = false
localdir    = "/tmp"
snaplength  = 500

[cloudshark]
host        = "www.cloudshark.org"
scheme      = "https"
port        = 443
token       = "fffffffffffffffffff"
upload      = true

[redis]
host        = "node.running.redis.net"
port        = 6379
channel     = "capture"

# consult /usr/include/sys/syslog.h for Priority which is a combination 
# of facility and severity
[syslog]
priority    = 85
tag         = "pcapdaemon"
<commit_msg>add server side encryption for s3<commit_after># pcapdaemon

## Description
This is a daemon that will subscribe to a redis pub/sub channel for requests to capture.  It will capture and then optionally upload to Cloudshark or save to the local filesystem.  It could really be adapted to upload anywhere but the key was that I wanted to be able to trigger captures based on any number of events (traps, log events etc) via a lightweight mechanism.  A design goal was to have it capture into a buffer in memory and post the buffer without adding any kind of filesystem/io dependency.  

## Options
    -cshost string          cloudshark host (default "localhost")
    -csscheme string        cloudshark scheme http|https (default "https")
    -cstoken string         cloudshark api token (default "<PASSWORD>")
    -csport int             cloudshark port
    -redischannel string    Redis channel to subscribe to.  Default capture (default "capture")
    -redisnode string       Hostname|IP of redis server.  Default localhost (default "127.0.0.1")
    -redisport int          Port of redis server. Default 6379 (default 6379)
    -upload                 Upload pcap
    -writelocal             Write pcap files locally.  Requires setting destdir
    -destdir                Directory to store locally written pcap files in
    -maxpackets             Maximum number of packets per capture.  Default 50000.
    -config string          Path to configuration file.  No default.
    
## Message format
    {
        "node": "node name",
        "nodere": "node regex",
        "interface": ["bond1","bond2"],
        "alias": ["local","public"],
        "tags": "blah,tagme,stuff",
        "bpf": "dst ip 10.0.0.1",
        "customer": "importantcustomer",
        "snap": 1500,
        "packets": 50,
        "alertid": 655443,
        "timeout": 15
		"folder": "myfolder",
		"bucket": "mybucket",
		"acl": "public-read",
		"region": "us-east-1",
		"endpoint": "s3.amazonaws.com",
		"encryption": false
    }
    
 * node - node name to capture on (exact match) Use either node or nodere but not both and one is required. Keyword: "any" matches any host
 * nodere - node regex to capture on.  Use either node or nodere but not both and one is required.
 * interface - an array of interfaces to dump on. (Must supply either interface or alias but not both)
 * alias - an array of interface aliases to dump on.  (Must supply either interface or alias but not both)
 * tags - additional metadata for the capture file.  Comma separated list.
 * bpf - A filter string to capture on
 * customer - Additional metadata field ( not required )
 * snap - Snaplength or amount to capture into the packet.  Integer.
 * packets - Number of packets to capture.  Integer.
 * alertid - An integer ID for the event or alert or whatever you're tracking (not required)
 * timeout - Number of seconds to let capture last should the number of packets not get hit.  Integer.
 * folder - S3 folder inside your bucket // S3 ONLY
 * bucket - S3 bucket // S3 ONLY
<<<<<<< HEAD
 * folder - S3 ACL // S3 ONLY
 * folder - S3 Endpoint // S3 ONLY
 * encryption - S3 Server side encryption AES256 // S3 ONLY
=======
 * acl - S3 ACL // S3 ONLY
 * endpoint - S3 Endpoint // S3 ONLY
>>>>>>> 4b878c3b314b96b75ca1456be8e0b653f31c4821

## Configuration File format (toml)
 * Defining interfaces is optional.  You only need to do it if you'd like to use an alias.  The basic use case is to group catpure interfaces across several nodes that may have different physical names for a variety of reasons.
``` 
## Config file
[general]
maxpackets  = 50000
writelocal  = false
localdir    = "/tmp"
snaplength  = 500

[cloudshark]
host        = "www.cloudshark.org"
scheme      = "https"
port        = 443
token       = "<PASSWORD>"
upload      = true

[redis]
host        = "node.running.redis.net"
port        = 6379
channel     = "capture"

# consult /usr/include/sys/syslog.h for Priority which is a combination 
# of facility and severity
[syslog]
priority    = 85
tag         = "pcapdaemon"

{'fraction_non_alphanumeric': 0.0824210393859318, 'fraction_numerical': 0.02214672203347175, 'mean_word_length': 2.747289014615747, 'pattern_counts': {'":': 34, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1808035', 'n_tokens_mistral': 2359, 'n_tokens_neox': 2181, 'n_words': 1011}