<commit_before>see "matpalm.com/resemblance":http://matpalm.com/resemblance for more info

h1. measuring resemblance (coeff)

h2. ruby

run ruby version and output all with resemblance > 0.5
<pre>
> cat test.data | ruby shingle.rb 0.5 > result
</pre>

h2. cpp

run cpp version outputting resemblances above 0 (ie all of them)

outputs to N files generated from N cores so need to collate

<pre>
> cat test.data | cpp/bin/Release/resemblance 0
> cat resemblance.*.out > result
</pre>

h1. munging 

h2. generate a histogram of resemblance values

<pre>
> cat result | ./histo.rb | sort -n > histo.dat
> plot_histo.sh histo.dat histo.100.png
</pre>

h1. measuring resemblance (distance)

h2. ruby

edit shingle.rb and swapped comment out line (ie wip!)

<pre>
> cat test.data | ruby shingle.rb 0.5 > distance.dat
</pre>

h2. cpp

change the code, cause i havent yet (ie wip!)

h2. converting from distances to points

project distances into 2 dimensional space

then check mean square error for projected distances versus original distances

<pre>
> cat distances.original | ./fastmap.rb 2 > points.2d
> cat points.2d | ./convert_points_to_distances.rb > distances.projected.2d
> ./mse.rb distances.original distances.projected.2d
</pre>

plot 2d and 3d data with gnuplot

<pre>
gnuplot> plot 'points.2d' with dots, 'points.2d' with labels
gnuplot> splot 'points.3d' with dots, 'points.3d' with labels
gnuplot> splot 'points.11d' with dots, 'points.11d' with labels # good luck with this one! ;)<commit_msg>gardening<commit_after>see "matpalm.com/resemblance":http://matpalm.com/resemblance for more info

h1. measuring resemblance (coeff)

h2. ruby

run ruby version and output all with resemblance > 0.5
<pre>
> cat test.data | ruby shingle.rb 0.5 > result
</pre>

h2. cpp

run cpp version outputting resemblances above 0 (ie all of them)

outputs to N files generated from N cores so need to collate

<pre>
> cat test.data | cpp/bin/Release/resemblance 0
> cat resemblance.*.out > result
</pre>

h1. munging 

h2. generate a histogram of resemblance values

<pre>
> cat result | ./histo.rb | sort -n > histo.dat
> plot_histo.sh histo.dat histo.100.png
</pre>

h1. measuring resemblance (distance)

h2. ruby

edit shingle.rb and swapped comment out line (ie wip!)

<pre>
> cat test.data | ruby shingle.rb 0.5 > distances.origina
</pre>

h2. cpp

change the code, cause i havent yet (ie wip!)

h2. converting from distances to points

project distances into 2 dimensional space

then check mean square error for projected distances versus original distances

(similiarly change 2 to whatever dimensionality)

<pre>
> cat distances.original | ./fastmap.rb 2 > points.2d
> cat points.2d | ./convert_points_to_distances.rb > distances.projected.2d
> ./mse.rb distances.original distances.projected.2d
</pre>

plot 2d and 3d data with gnuplot

<pre>
gnuplot> plot 'points.2d' with dots, 'points.2d' with labels
gnuplot> splot 'points.3d' with dots, 'points.3d' with labels
gnuplot> splot 'points.11d' with dots, 'points.11d' with labels # good luck with this one! ;)
{'fraction_non_alphanumeric': 0.10192747468147664, 'fraction_numerical': 0.023848415550473703, 'mean_word_length': 3.970779220779221, 'pattern_counts': {'":': 2, '<': 25, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '452816', 'n_tokens_mistral': 1112, 'n_tokens_neox': 1037, 'n_words': 386}