<commit_before># Description:
This project will run a webcrawler onto twitter looking for a specific searchable term and 
scrape all the tweets with the given phrase. Then it will write each tweet to a file where
it is piped into Stanford's sentiment analysis server coreNLP to rate each sentence.

# How to Run Project:
## How to Run Web Crawler:
In order to deploy the scrapy crawler... first cd into scrapy project
folder and then type:
    
    scrapy crawl [name of crawler]


## How to Run NLP Server:
In order to run the stanford NLP server... first cd into the stanford
project fold and then type:
    
    cd stanford-corenlp-full-2016-10-31
    java -mx5g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -timeout 10000<commit_msg>Added misc descriptions<commit_after># Description:
This project will run a webcrawler onto twitter looking for a specific searchable term and 
scrape all the tweets with the given phrase. Then it will write each tweet to a file where
it is piped into Stanford's sentiment analysis server coreNLP to rate each sentence.

# How to Run Project:
## How to Run Web Crawler:
In order to deploy the scrapy crawler... first cd into scrapy project
folder and then type:
    
    scrapy crawl [name of crawler]


## How to Run NLP Server:
In order to run the stanford NLP server... first cd into the stanford
project fold and then type:
    
    cd stanford-corenlp-full-2016-10-31
    java -mx5g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -timeout 10000

#Misc
## What is geckodriver?
Gecko Driver is the link between your tests in Selenium and the Firefox browser. GeckoDriver is a proxy for using W3C WebDriver-compatible clients to interact with Gecko-based browsers i.e. Mozilla Firefox in this case. As Selenium 3 will not have any native implementation of FF, we have to direct all the driver commands through Gecko Driver. Gecko Driver is an executable file that you need to have in one of the system path before starting your tests.
{'fraction_non_alphanumeric': 0.047666833918715505, 'fraction_numerical': 0.015052684395383844, 'mean_word_length': 3.8634146341463413, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2418503', 'n_tokens_mistral': 589, 'n_tokens_neox': 545, 'n_words': 300}