<commit_before># twitter-sentiment - WORK IN PROGRESS

## Intro

This is my Machine Learning project that I'm currently working on,
Twitter sentiment analysis (3-point scale) using character-input LSTM network with attention mechanism.
Implemented in Python 2.7 with Lasagne (0.2.dev1) and Theano (0.8.2.dev-RELEASE).
In my implementations of layers I reuse some Lasagne's code.
The main goal of this project is to experiment with different architectures and techniques.

Data from SemEval-2017 Task 4A:<br>
http://alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools<br>
and from Sentiment140:<br>
http://help.sentiment140.com/for-students

The problem: given a tweet predict its overall sentiment - negative, neutral or positive.

## Structure

### get_data.py

The script uses data from 5 separate locations:
* http://alt.qcri.org/semeval2017/task4/data/uploads/download.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/codalab/4a-english.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/semeval2016_task4_submissions_and_scores.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/semeval2017-task4-test-input-phase1-v3.0.zip
* http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip

You need to download and extract the files to `data\extracted`. Since Twitter's TOS specifies you cannot
share downloaded tweets, some datasets include only tweet
ids and labels. The script downloads all of those that still exist, but it needs your Twitter credentials.
You should rename `get_data_template.cfg` to `get_data.cfg` and provide them there.
The program loads all the data into one format and pickles it into separate files.

Then it unescapes HTML entities into proper characters. The script gets all the
characters and symbols used in all the datasets except Stanford into one charset. Then it loops through all the tweets
in the Stanford dataset to filter out those that contain characters not included in this charset.

Some of the SemEval tweets in training and validation datasets intersect. Also some tweets are repeated. The script
handles both issues.<commit_msg>note about datsets<commit_after># twitter-sentiment - WORK IN PROGRESS

## Intro

This is my Machine Learning project that I'm currently working on,
Twitter sentiment analysis (3-point scale) using character-input LSTM network with attention mechanism.
Implemented in Python 2.7 with Lasagne (0.2.dev1) and Theano (0.8.2.dev-RELEASE).
In my implementations of layers I reuse some Lasagne's code.
The main goal of this project is to experiment with different architectures and techniques.

Data from SemEval-2017 Task 4A:<br>
http://alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools<br>
and from Sentiment140:<br>
http://help.sentiment140.com/for-students

The problem: given a tweet predict its overall sentiment - negative, neutral or positive.

Note: So far I only use data from the first source (SemEval competition).
But I'm planning to experiment with the Sentiment140 dataset in the not too distant future.
I would like to see if it will help to achieve a better performance on the competition data,
especially given its properties:
* biased - it is a different dataset, collected by another body;
* noisy - it was collected automatically, based on emoticons;
* labeled with only two classes (positive and negative);
* 2 orders of magnitude bigger (1.6M and 23K).

## Structure

### get_data.py

The script uses data from 5 separate locations:
* http://alt.qcri.org/semeval2017/task4/data/uploads/download.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/codalab/4a-english.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/semeval2016_task4_submissions_and_scores.zip
* http://alt.qcri.org/semeval2017/task4/data/uploads/semeval2017-task4-test-input-phase1-v3.0.zip
* http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip

You need to download and extract the files to `data\extracted`. Since Twitter's TOS specifies you cannot
share downloaded tweets, some datasets include only tweet
ids and labels. The script downloads all of those that still exist, but it needs your Twitter credentials.
You should rename `get_data_template.cfg` to `get_data.cfg` and provide them there.
The program loads all the data into one format and pickles it into separate files.

Then it unescapes HTML entities into proper characters. The script gets all the
characters and symbols used in all the datasets except Stanford into one charset. Then it loops through all the tweets
in the Stanford dataset to filter out those that contain characters not included in this charset.

Some of the SemEval tweets in training and validation datasets intersect. Also some tweets are repeated. The script
handles both issues.
{'fraction_non_alphanumeric': 0.0723781388478582, 'fraction_numerical': 0.027009917704156994, 'mean_word_length': 4.752427184466019, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2029640', 'n_tokens_mistral': 1433, 'n_tokens_neox': 1297, 'n_words': 563}