<commit_before># clj-dstream

## What Is It
Density-based data stream clustering for arbitrary dimension data written in Clojure.

Reference [paper](refpaper).


Sample clustering for time-series data with moving hotspot:
![Three Clusters GIF][clustergif]


[refpaper]: https://github.com/ogeagla/clj-dstream/raw/master/papers/sigproc-sp.pdf
[clustergif]: https://github.com/ogeagla/clj-dstream/raw/master/doc/resources/3-clusters-1-at-a-time/animated-loop.gif "Three Clusters One At A Time"

## Running Tests

```
lein test
```

Note: If you install `imagemagick` for your platform, which supplies the command `convert`, animated gifs (like in this README) will be generated by the tests in addition to heatmaps of clusters and grid density data. 

In addition to the unit tests, the additional 'real-world data' tests will generate directories `crater-out` and `normal-sample-out`.   These directories will each contain heatmaps in a time-series for the clusters at a given time and the grid densities at a given time.  This enables us to reason about the clusters wrt to the grid densities at a given time.

## Crater Dataset

When faced with data shapes like a crater, which can look like so:
![Crater Grids SVG][cratergrids]

Many clustering algorithms would combine these clusters into one, but DStream results look better:<commit_msg>[doc] readme<commit_after># clj-dstream

## What Is It
Density-based data stream clustering for arbitrary dimension data written in Clojure.

Reference [paper][refpaper].


Sample clustering for time-series data with moving hotspot:
![Three Clusters GIF][clustergif]


[refpaper]: https://github.com/ogeagla/clj-dstream/raw/master/papers/sigproc-sp.pdf
[clustergif]: https://github.com/ogeagla/clj-dstream/raw/master/doc/resources/3-clusters-1-at-a-time/animated-loop.gif "Three Clusters One At A Time"

## Running Tests

```
lein test
```

Note: If you install `imagemagick` for your platform, which supplies the command `convert`, animated gifs (like in this README) will be generated by the tests in addition to heatmaps of clusters and grid density data. 

In addition to the unit tests, the additional 'real-world data' tests will generate directories `crater-out` and `normal-sample-out`.   These directories will each contain heatmaps in a time-series for the clusters at a given time and the grid densities at a given time.  This enables us to reason about the clusters wrt to the grid densities at a given time.

## Crater Dataset

When faced with data shapes like a crater, which can look like so:
![Crater Grids SVG][cratergrids]

Many clustering algorithms would combine these clusters into one, but DStream results look better:
{'fraction_non_alphanumeric': 0.07982096232748974, 'fraction_numerical': 0.0014919806042521448, 'mean_word_length': 4.706382978723404, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4681191', 'n_tokens_mistral': 781, 'n_tokens_neox': 747, 'n_words': 336}