<commit_before>see "matpalm.com/resemblance":http://matpalm.com/resemblance for a proper walkthrough
instructions here are more for replicating the results on the above project page

h1. measuring exact resemblance (jaccard coeff against shingles)

h2. ruby

run ruby version and output all with resemblance > 0.5
<pre>
> cat test.data | ./ruby/shingle.rb coeff 0.5 > result
</pre>

h2. cpp

run cpp version outputting resemblances above 0 (ie all of them)

outputs to N files generated from N cores so need to collate

<pre>
> cat test.data | cpp/bin/Release/resemblance 0
> cat resemblance.*.out > result
</pre>

h1. munging 

h2. examine a result file

ie see phrases used for comparison rather than just raw result (eg 1 3 0.88)

<pre>
> cat test.data | ./examine_result.rb result
</pre>

h2. generate a histogram of resemblance values

<pre>
> cat result | ./histo.rb | sort -n > histo.dat
> ./plot_histo.sh histo.dat histo.100.png
</pre>

h1. measuring resemblance (distance)

h2. ruby

<pre>
> cat test.data | ./ruby/shingle.rb distance 0 > distances.original
</pre>

h2. cpp

change the code, cause i havent yet (ie wip!)
<commit_msg>very simple isntructions on erlang version<commit_after>see "matpalm.com/resemblance":http://matpalm.com/resemblance for a proper walkthrough
instructions here are more for replicating the results on the above project page

h1. measuring exact resemblance (jaccard coeff against shingles)

h2. ruby

run ruby version and output all with resemblance > 0.5
<pre>
> cat test.data | ./ruby/shingle.rb coeff 0.5 > result
</pre>

h2. cpp

run cpp version outputting resemblances above 0 (ie all of them)

outputs to N files generated from N cores so need to collate

<pre>
> cat test.data | cpp/bin/Release/resemblance 0
> cat resemblance.*.out > result
</pre>

h2. erlang

run early days work-in-progress erlang version as 

<pre>
> cd erl && rake && cd ..
> erl -pa erl/ebin -s main main
</pre>

h1. munging 

h2. examine a result file

ie see phrases used for comparison rather than just raw result (eg 1 3 0.88)

<pre>
> cat test.data | ./examine_result.rb result
</pre>

h2. generate a histogram of resemblance values

<pre>
> cat result | ./histo.rb | sort -n > histo.dat
> ./plot_histo.sh histo.dat histo.100.png
</pre>

h1. measuring resemblance (distance)

h2. ruby

<pre>
> cat test.data | ./ruby/shingle.rb distance 0 > distances.original
</pre>

h2. cpp

change the code, cause i havent yet (ie wip!)

{'fraction_non_alphanumeric': 0.1000408329930584, 'fraction_numerical': 0.02000816659861168, 'mean_word_length': 3.6139359698681734, 'pattern_counts': {'":': 2, '<': 25, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2302957', 'n_tokens_mistral': 893, 'n_tokens_neox': 834, 'n_words': 328}