<commit_before># bigData
big data course project


1. Please download the database "reviews_Electronics_5.json.gz" in our google drive, link below

https://drive.google.com/drive/folders/0ByehYI2Lxf1nQWZTQ05LbzV2SGM

  1.) database/reviews.txt
  2.) database/train_data.json
  3.) doc2vec/Archive.zip

2. Put 1.) and 2.) in database folder and unzip 3.) into doc2vec folder where you downloaded from github repository.

4. Run tf_train.py (You can cahange the "learning_rate", "loops", "mode". Also you can change comparison sets by tweeking "CLASSES = [[1], [5]]" in myconstants.py)

5. You don't need to run preprocess.py any more.




Note: Please Keep the folder structure the same as below, otherwise you might get unexpected Errors(Read this structure in your text editor)<commit_msg>change in readme<commit_after># bigData
big data course project


1. Please download the database and models from our google drive, link below

https://drive.google.com/drive/folders/0ByehYI2Lxf1nQWZTQ05LbzV2SGM

  1.) database/reviews.txt
  2.) database/train_data.json
  3.) doc2vec/Archive.zip

2. Put 1.) and 2.) in database folder and unzip 3.) into doc2vec folder where you downloaded from github repository.

4. Run tf_train.py (You can cahange the "learning_rate", "loops", "mode". Also you can change comparison sets by tweeking "CLASSES = [[1], [5]]" in myconstants.py)

5. You don't need to run preprocess.py any more.




Note: Please Keep the folder structure the same as below, otherwise you might get unexpected Errors(Read this structure in your text editor)
{'fraction_non_alphanumeric': 0.08695652173913043, 'fraction_numerical': 0.026214833759590793, 'mean_word_length': 4.287162162162162, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1932371', 'n_tokens_mistral': 517, 'n_tokens_neox': 484, 'n_words': 200}