<commit_before>[![Build Status](https://travis-ci.org/F483/dejavu.svg)](https://travis-ci.org/F483/dejavu)
[![Go Report Card](https://goreportcard.com/badge/github.com/f483/dejavu)](https://goreportcard.com/report/github.com/f483/dejavu)
[![Issues](https://img.shields.io/github/issues/f483/dejavu.svg)](https://github.com/f483/dejavu/issues)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/f483/dejavu/master/LICENSE)
[![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/f483/dejavu)


# Déjà vu

Quickly detect already witnessed data.

Memory of witnessed data entries is limited and oldest are forgotten.
It may give false negatives, but not false positives. Library is thread
safe.


# Example

```
package main

import (
	"fmt"
	"github.com/f483/dejavu"
)

func main() {

	// remembers last three entries
	d := dejavu.NewDejaVu(3)

	// add entries
	fmt.Println(d.Witness([]byte("foo")))
	fmt.Println(d.Witness([]byte("bar")))

	// remembers entry
	fmt.Println(d.Witness([]byte("bar")))

	// remembers oldest entry before overwriting
	fmt.Println(d.Witness([]byte("foo")))

	// add entries
	fmt.Println(d.Witness([]byte("bam")))
	fmt.Println(d.Witness([]byte("baz")))

	// forgot oldest
	fmt.Println(d.Witness([]byte("bar")))
}
```


# Performance

### Constant witness time: O(1)

![Benchmark Time](https://github.com/f483/dejavu/raw/master/benchmark-time.png)

### Linear memory usage: O(n)

![Benchmark Memory](https://github.com/f483/dejavu/raw/master/benchmark-memory.png)



# Support

TODO donation text and bitcoin/paypal links<commit_msg>added donation badges<commit_after>[![Build Status](https://travis-ci.org/F483/dejavu.svg)](https://travis-ci.org/F483/dejavu)
[![Go Report Card](https://goreportcard.com/badge/github.com/f483/dejavu)](https://goreportcard.com/report/github.com/f483/dejavu)
[![Issues](https://img.shields.io/github/issues/f483/dejavu.svg)](https://github.com/f483/dejavu/issues)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/f483/dejavu/master/LICENSE)
[![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/f483/dejavu)
[![Donate PayPal](https://img.shields.io/badge/Donate-PayPal-orange.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=fabian%2ebarkhau%40gmail%2ecom&lc=DE&item_name=https%3a%2f%2fgithub%2ecom%2fF483%2fdejavu&no_note=0&currency_code=EUR&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHostedGuest)
[![Donate Bitcoin](https://img.shields.io/badge/Donate-Bitcoin-orange.svg)](https://blockchain.info/address/13nAHLVo5GRdwVeLxEjbgEvyusrjdQogdD)


# Déjà vu

Quickly detect already witnessed data.

Memory of witnessed data entries is limited and oldest are forgotten.
It may give false negatives, but not false positives. Library is thread
safe.


# Example

```
package main

import (
	"fmt"
	"github.com/f483/dejavu"
)

func main() {

	// remembers last three entries
	d := dejavu.NewDejaVu(3)

	// add entries
	fmt.Println(d.Witness([]byte("foo")))
	fmt.Println(d.Witness([]byte("bar")))

	// remembers entry
	fmt.Println(d.Witness([]byte("bar")))

	// remembers oldest entry before overwriting
	fmt.Println(d.Witness([]byte("foo")))

	// add entries
	fmt.Println(d.Witness([]byte("bam")))
	fmt.Println(d.Witness([]byte("baz")))

	// forgot oldest
	fmt.Println(d.Witness([]byte("bar")))
}
```


# Performance

### Constant witness time: O(1)

![Benchmark Time](https://github.com/f483/dejavu/raw/master/benchmark-time.png)

### Linear memory usage: O(n)

![Benchmark Memory](https://github.com/f483/dejavu/raw/master/benchmark-memory.png)
{'fraction_non_alphanumeric': 0.18804112554112554, 'fraction_numerical': 0.02462121212121212, 'mean_word_length': 5.2449324324324325, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 28, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2665426', 'n_tokens_mistral': 1517, 'n_tokens_neox': 1407, 'n_words': 184}