<commit_before># WCSA

We include here source code for the WCSA used in our paper:

"Universal Indexes for Highly Repetitive Document Collections"
submitted to Information Systems Journal in April 2016 
by <NAME>; <NAME>.; <NAME>.; and <NAME>.

It is an adaptation of the source code used in our paper:
<NAME>.; <NAME>.; <NAME>.; <NAME>.: 
"Dynamic Lightweight Text Compression", en ACM Transactions on Information Systems (TOIS), 
28(3), ACM, New York, NY (Estados Unidos), 2010, pp. 1-32.

http://vios.dc.fi.udc.es/wsi

The main modification consisted in allowing WCSA to work in a doc-oriented fashion.
Therefore we provide to the indexer not only a dataset, but a dataset (doc-collection) and the
doc-beginnings (absolute positions were each doc starts with the doc-collection).

Then, we adapted locate to return not actual absolute
positions within a collection, but pairs of (doc,rel-pos) values, that is, 
it returns in which docs a pattern occurs and also the positions within those documents.


-----
We include the source code in directory "sourceCode" and scripts to run our experiments 
in the folder "scripts.swcsa3.is"
  
The resulting results will be saved in 
  (locate)
	N.Wa_swcsa.dat  --> low freq words
	N.Wb_swcsa.dat  --> high freq words
	N.P2_swcsa.dat  --> 2-words phrases
	N.P5_swcsa.dat  --> 5-words phrases

	e.Words20_swcsa.dat    --> extracting 20 words sequences
	e.Words3000_swcsa.dat  --> extracting 3000 words sequences


-----
Steps to run experiments.

cd data
sh go2gb.sh           --> sets links to data/patterns/intervals from root folder (dataset for INF. systems paper)
cd ..
cd sourceCode  --> makes source code.
make
cd ..
cd scripts.swcsa3.is
sh goAll.sh    --> runs experiments.
 



If you have any doubt or need more information, please feel free to contact us:

- <NAME>: \<<EMAIL>\>
- <NAME>: \<<EMAIL>\>
- <NAME>: \<<EMAIL>\>
- <NAME>:  \<<EMAIL>\>



<commit_msg>uiHRDC Reproducibility Framework v1.0<commit_after># WCSA

We include here source code for the WCSA used in our paper:

"Universal Indexes for Highly Repetitive Document Collections"
submitted to Information Systems Journal in April 2016 
by <NAME>; <NAME>.; <NAME>.; and <NAME>.

It is an adaptation of the source code used in our paper:
<NAME>.; <NAME>.; <NAME>.; <NAME>.: 
"Dynamic Lightweight Text Compression", en ACM Transactions on Information Systems (TOIS), 
28(3), ACM, New York, NY (Estados Unidos), 2010, pp. 1-32.

http://vios.dc.fi.udc.es/wsi

The main modification consisted in allowing WCSA to work in a doc-oriented fashion.
Therefore we provide to the indexer not only a dataset, but a dataset (doc-collection) and the
doc-beginnings (absolute positions were each doc starts with the doc-collection).

Then, we adapted locate to return not actual absolute
positions within a collection, but pairs of (doc,rel-pos) values, that is, 
it returns in which docs a pattern occurs and also the positions within those documents.


-----
We include the source code in directory "sourceCode" and scripts to run our experiments 
in the folder "scripts.swcsa3.is"
  
The resulting results will be saved in 
  (locate)
	N.Wa_swcsa.dat  --> low freq words
	N.Wb_swcsa.dat  --> high freq words
	N.P2_swcsa.dat  --> 2-words phrases
	N.P5_swcsa.dat  --> 5-words phrases

	e.Words20_swcsa.dat    --> extracting 20 words sequences
	e.Words3000_swcsa.dat  --> extracting 3000 words sequences


-----
Steps to run experiments.

cd data

sh go2gb.sh           --> sets links to data/patterns/intervals from root folder (dataset for INF. systems paper)

cd ..

cd sourceCode  --> makes source code.

make

cd ..

cd scripts.swcsa3.is

sh goAll.sh    --> runs experiments.
 



If you have any doubt or need more information, please feel free to contact us:

- <NAME>: \<<EMAIL>\>
- <NAME>: \<<EMAIL>\>
- <NAME>: \<<EMAIL>\>
- <NAME>:  \<<EMAIL>\>




{'fraction_non_alphanumeric': 0.10443119979269241, 'fraction_numerical': 0.01762114537444934, 'mean_word_length': 3.4572748267898383, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2745217', 'n_tokens_mistral': 1282, 'n_tokens_neox': 1196, 'n_words': 511}