<commit_before># clj-similar

Experimental library for (fast) similar set lookup.
Under the hood it uses a [MinHash](https://en.wikipedia.org/wiki/MinHash) to compute a locality sensitive hashes of a collection of sets, and loads them into a [k-d tree](https://en.wikipedia.org/wiki/K-d_tree).
The constructed `similar` data structure can be used to retrieve [nearest neighbors](https://en.wikipedia.org/wiki/Nearest_neighbor_search) of a given target set.

While the construction of the data structure is expensive, lookups should be fast.

Note that it will always return some set that is considered nearest.
Thresholds for a value that is "too dissimilar" is currently at your own discretion, the distance can be accessed with the `meta` data on the returned result(s).


## Caveats
Collections of sets that have more than the maximum integer value of distinct values are currently unsupported.
The construction of the tree is expensive, due to various overheads.
Currently this data structure is read-only, support for modifications is not currently planned.
This is a *very* experimental library, it was literally written in 30 minutes, so don't expect much.

## Usage

```clojure
(require '[clj-similar.core :refer [similar nearest]])
(def coll [#{"a" "b" "c"} #{"d" "e" "c"} #{"f" "e" "a" "b"}])
;; Creates the data structure, optionally an error rate can be defined (default 0.05)
(def s (similar coll))
(def s (similar coll 0.01)) ;; different error rate

;; A single nearest neighbor
(nearest s #{"f" "e" "a" "b"})
;=> #{"f" "e" "a" "b"}

(nearest s #{"f" "e" "a"})
;=> #{"f" "e" "a" "b"}

;; Two nearest neighbors
(nearest s #{"a" "b"} 2)
;=> (#{"a" "b" "c"} #{"f" "e" "a" "b"})<commit_msg>Update README <commit_after># clj-similar

Experimental library for (fast) similar set lookup.
Under the hood it uses a [MinHash](https://en.wikipedia.org/wiki/MinHash) to compute a locality sensitive hashes of a collection of sets, and loads them into a [k-d tree](https://en.wikipedia.org/wiki/K-d_tree).
The constructed `similar` data structure can be used to retrieve [nearest neighbors](https://en.wikipedia.org/wiki/Nearest_neighbor_search) of a given target set.

While the construction of the data structure is expensive, lookups should be fast.

Note that it will always return some set that is considered nearest.
Thresholds for a value that is "too dissimilar" is currently at your own discretion, the distance can be accessed with the `meta` data on the returned result(s).


## Caveats
Collections of sets that have more than the maximum integer value of distinct values are currently unsupported.
The construction of the tree is expensive, due to various overheads.
Currently this data structure is read-only, support for modifications is not currently planned.

## Usage

```clojure
(require '[clj-similar.core :refer [similar nearest]])
(def coll [#{"a" "b" "c"} #{"d" "e" "c"} #{"f" "e" "a" "b"}])
;; Creates the data structure, optionally an error rate can be defined (default 0.05)
(def s (similar coll))
(def s (similar coll 0.01)) ;; different error rate

;; A single nearest neighbor
(nearest s #{"f" "e" "a" "b"})
;=> #{"f" "e" "a" "b"}

(nearest s #{"f" "e" "a"})
;=> #{"f" "e" "a" "b"}

;; Two nearest neighbors
(nearest s #{"a" "b"} 2)
;=> (#{"a" "b" "c"} #{"f" "e" "a" "b"})
{'fraction_non_alphanumeric': 0.1362810417928528, 'fraction_numerical': 0.004845548152634767, 'mean_word_length': 4.685025817555938, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3329583', 'n_tokens_mistral': 1058, 'n_tokens_neox': 998, 'n_words': 472}