<commit_before># onyx-metrics

Onyx Lifecycle plugin for instrumenting workflows. Track throughput and metrics of your tasks and log the output to Timbre and/or dashboard.

#### Installation

In your project file:

```clojure
[com.mdrogalis/onyx-metrics "0.6.0-SNAPSHOT"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.lifecycle.metrics.throughput]
          [onyx.lifecycle.metrics.latency]
          [onyx.lifecycle.metrics.timbre])
```

#### Lifecycle entries

##### Throughput

```clojure
{:lifecycle/task :my-task-name
 :lifecycle/pre :onyx.lifecycle.metrics.throughput/pre
 :lifecycle/post-batch :onyx.lifecycle.metrics.throughput/post-batch
 :lifecycle/post :onyx.lifecycle.metrics.throughput/post
 :throughput/retention-ms 60000
 :lifecycle/doc "Instruments a task's throughput metrics"}
```

##### Timbre Logging

```clojure
{:lifecycle/task :my-task-name<commit_msg>Docs.<commit_after># onyx-metrics

Onyx Lifecycle plugin for instrumenting workflows. Track throughput and metrics of your tasks and log the output to Timbre and/or dashboard.

#### Installation

In your project file:

```clojure
[com.mdrogalis/onyx-metrics "0.6.0-SNAPSHOT"]
```

In your peer boot-up namespace:

```clojure
(:require [onyx.lifecycle.metrics.throughput]
          [onyx.lifecycle.metrics.latency]
          [onyx.lifecycle.metrics.timbre])
```

#### Lifecycle entries

##### Throughput

Computes throughput in terms of segments per second for 10s, 30s, and 60s windows.

```clojure
{:lifecycle/task :my-task-name
 :lifecycle/pre :onyx.lifecycle.metrics.throughput/pre
 :lifecycle/post-batch :onyx.lifecycle.metrics.throughput/post-batch
 :lifecycle/post :onyx.lifecycle.metrics.throughput/post
 :throughput/retention-ms 60000
 :lifecycle/doc "Instruments a task's throughput metrics"}
```

##### Batch Latency

Computes the 50th, 90th, and 99th percentile latency in milliseconds per batch of segments for 10s, 30s, and 60s windows.

```clojure
{:lifecycle/task :inc
 :lifecycle/pre :onyx.lifecycle.metrics.latency/pre
 :lifecycle/pre-batch :onyx.lifecycle.metrics.latency/pre-batch
 :lifecycle/post-batch :onyx.lifecycle.metrics.latency/post-batch
 :lifecycle/post :onyx.lifecycle.metrics.latency/post
 :latency/retention-ms 60000
 :lifecycle/doc "Instruments a task's latency metrics per batch"}
```

##### Timbre Logging

Logs all statistics collected to Timbre.

```clojure
{:lifecycle/task :my-task-name
{'fraction_non_alphanumeric': 0.1281521289789169, 'fraction_numerical': 0.016122364613476645, 'mean_word_length': 4.803357314148681, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '275296', 'n_tokens_mistral': 876, 'n_tokens_neox': 789, 'n_words': 194}