<commit_before># Metrics Collector

Lightweight standalone process which runs on mesos-agent nodes.

Inputs metrics [sent over TCP](../schema/) from local sources:
- The [mesos-agent module](../module/), for metrics relating to containers
- Other DCOS processes on the system

Outputs metrics to the following destinations:
- Kafka metrics service
- Local partner processes, if any

A basic HTTP server provides:
- Returning input/output TCP endpoints for local processes to use
- On-the-fly configuration changes

## Sample Producer

The sample producer is a Collector implementation which just grabs some data from the mesos agent and forwards it to a Kafka cluster. It's meant to allow Kafka Consumers to be implemented while the real Collector is still being built.

### Build

Prerequisites:
- git
- [Go 1.5+](https://golang.org/dl/)

First, get the code and set up the environment (edit `/YOUR/CODEPATH` and `/YOUR/GOPATH` as needed):

```bash
export CODEPATH=/YOUR/CODEPATH # eg ~/code
export GOPATH=/YOUR/GOPATH # eg ~/go
export GO15VENDOREXPERIMENT=1 # only required if using Go 1.5. for Go 1.6+ this step can be skipped

mkdir -p $CODEPATH
cd $CODEPATH

git clone git@github.com:mesosphere/dcos-stats
mkdir -p $GOPATH/src/github.com/mesosphere
ln -s $CODEPATH/dcos-stats $GOPATH/src/github.com/mesosphere/dcos-stats
```

Then preprocess the Avro schema and build the code:

```bash
cd $GOPATH/src/github.com/mesosphere/dcos-stats/collector
go generate # creates "metrics_schema_generated.go" in collector dir

cd cmd/sample-producer
go build # creates "sample-producer" executable
./sample-producer -h
```

If you see errors about `undefined: collector.[*]Namespace` and `undefined: collector.[*]Schema`, you forgot to run `go generate` in the `dcos-stats/collector/` directory to create `dcos-stats/collector/metrics_schema_generated.go`.

### Deploy

1. Install/configure the Kafka framework on your DCOS cluster. By default it will be named `kafka`.
2. Upload the `sample-producer` executable you built to an agent node. Ideally the agent node should have containers currently running on it. If no containers are running, `sample-producer` will send nothing to Kafka until containers have appeared.
3. Run the producer as `./sample-producer -framework <kafka-fmwk-name>` (add an `&` to launch as a background task). Assuming the targeted Kafka framework is up and running, `sample-producer` should automatically detect the brokers and start forwarding stats from `http://<agent-ip>:5051/monitor/statistics.json` to those brokers.<commit_msg>Start on marathon instructions, update generate step<commit_after># Metrics Collector

Lightweight standalone process which runs on mesos-agent nodes.

Inputs metrics [sent over TCP](../schema/) from local sources:
- The [mesos-agent module](../module/), for metrics relating to containers
- Other DCOS processes on the system

Outputs metrics to the following destinations:
- Kafka metrics service
- Local partner processes, if any

A basic HTTP server provides:
- Returning input/output TCP endpoints for local processes to use
- On-the-fly configuration changes

## Sample Producer

The sample producer is a Collector implementation which just grabs some data from the mesos agent and forwards it to a Kafka cluster. It's meant to allow Kafka Consumers to be implemented while the real Collector is still being built.

### Build

Prerequisites:
- git
- [Go 1.5+](https://golang.org/dl/)

First, get the code and set up the environment (edit `/YOUR/CODEPATH` and `/YOUR/GOPATH` as needed):

```bash
export CODEPATH=/YOUR/CODEPATH # eg ~/code
export GOPATH=/YOUR/GOPATH # eg ~/go
export GO15VENDOREXPERIMENT=1 # only required if using Go 1.5. for Go 1.6+ this step can be skipped

mkdir -p $CODEPATH
cd $CODEPATH

git clone <EMAIL>:mesosphere/dcos-stats
mkdir -p $GOPATH/src/github.com/mesosphere
ln -s $CODEPATH/dcos-stats $GOPATH/src/github.com/mesosphere/dcos-stats
```

Then preprocess the Avro schema and build the code:

```bash
cd $GOPATH/src/github.com/mesosphere/dcos-stats/collector
go generate # creates 'metrics-schema' package

cd cmd/sample-producer/
go build
./sample-producer -h
```

If you see errors about `cannot find package "github.com/.../metrics-schema"`, you forgot to perform `go generate` in the `dcos-stats/collector/` directory.

### Deploy

1. Configure and deploy a Kafka instance on your DC/OS cluster. By default it will be named `kafka`.
2. Upload the `sample-producer` executable you built to somewhere that's visible to your cluster (eg S3).
3. Run `sample-producer` as a task in Marathon, by editing the following JSON config. Edit `-framework kafka` to match the name of your deployed Kafka cluster, if needed:

```json
TODO marathon task config: require port 8124 (even if sample-producer isn't using it)
```

As `sample-producer` is deployed on every node, each instance should automatically start forwarding stats from `http://<local-agent-ip>:5051/monitor/statistics.json` to the brokers it got from querying the Kafka Scheduler. If the agent isn't running any containers, or if the Kafka Scheduler isn't reachable, it should automatically retry until either situation changes.
{'fraction_non_alphanumeric': 0.08950318963850763, 'fraction_numerical': 0.007152522714092403, 'mean_word_length': 4.4809322033898304, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '69219', 'n_tokens_mistral': 1576, 'n_tokens_neox': 1470, 'n_words': 650}