<commit_before>red-glint
=========

some scripts designed to handle big matrix handling for 2nd order vector matrix<commit_msg>- updated a bit on readme<commit_after>red-glint
=========

It contains some scripts designed to handle big matrix. The goal here is generating 2nd-order (smoothed) matrix. 

Terminology
- e-f-s file: file that holds element-feature (vector) matrix. Here, it refers to a specific file format that holds many line where each line has the format of"element \t feature \t score \n". 

* List of the test scripts: 
- test1.rb: reads e-f-s file and converts it to f-e-s. Not very effective (uses lots of ram.) 
- test2.rb: reads e-f-s file and outputs integer ID based feature-vectors. The format is "feature_id \t\t element_id \t value_for_this \t element_id \t value_of_that \t ... \n". Each line shows one feature vector (one column vector, of full matrix). 

* The main scripts: 
(will package some of the test scripts into this, when finished. ... when?) 
{'fraction_non_alphanumeric': 0.09572301425661914, 'fraction_numerical': 0.004073319755600814, 'mean_word_length': 3.748792270531401, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4054687', 'n_tokens_mistral': 292, 'n_tokens_neox': 281, 'n_words': 141}