<commit_before>## Problem 4.3. Twitter: Regular Expressions.

A template for this problem is provided:
[twitter_regex.ipynb](twitter_regex.ipynb)

In this problem, you will use regular expressions to process real Twitter data.

(If you are curious where this data came from, 1,000 tweets were fetched using
the search query `#informatics` on Tuesday, February 3rd, 2015, around 10pm.
See [Mining the Social Web 2nd
Edition](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition) by
<NAME>.
[twitter_to_ascii.py](https://github.com/INFO490/spring2015/tree/master/week04/twitter_to_ascii.py)
on the course Github repository is a Python 2 script that collects 1,000 tweets
and generates an ASCII text file of status texts.)

If you used `git clone`/`git pull` from `/data`, the path to the data file will
be `/data/spring2015/week04/informatics.txt`. It's an ASCII text file, and when
you first get a new data like this, it's a good idea to see how the data looks
like. You have learned various ways to do this: you could write a short Python
script that reads the file line by line and print them out; you could mount the
data container onto a Docker container in terminal mode and use `head`; but
here I'll show you how to use shell commands in IPython notebook. Open up a new
IPython notebook and run

```console
%%bash
head /data/spring2015/week04/informatics.txt
```

Then the output will be
```console
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc<commit_msg>Fix link to twitter_to_ascii.py.<commit_after>## Problem 4.3. Twitter: Regular Expressions.

A template for this problem is provided:
[twitter_regex.ipynb](twitter_regex.ipynb)

In this problem, you will use regular expressions to process real Twitter data.

(If you are curious where this data came from, 1,000 tweets were fetched using
the search query `#informatics` on Tuesday, February 3rd, 2015, around 10pm.
See [Mining the Social Web 2nd
Edition](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition) by
<NAME>.
[twitter_to_ascii.py](twitter_to_ascii)
on the course Github repository is a Python 2 script that collects 1,000 tweets
and generates an ASCII text file of status texts.)

If you used `git clone`/`git pull` from `/data`, the path to the data file will
be `/data/spring2015/week04/informatics.txt`. It's an ASCII text file, and when
you first get a new data like this, it's a good idea to see how the data looks
like. You have learned various ways to do this: you could write a short Python
script that reads the file line by line and print them out; you could mount the
data container onto a Docker container in terminal mode and use `head`; but
here I'll show you how to use shell commands in IPython notebook. Open up a new
IPython notebook and run

```console
%%bash
head /data/spring2015/week04/informatics.txt
```

Then the output will be
```console
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
{'fraction_non_alphanumeric': 0.0876645103725184, 'fraction_numerical': 0.018514387686816864, 'mean_word_length': 5.010723860589812, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1818269', 'n_tokens_mistral': 1559, 'n_tokens_neox': 1398, 'n_words': 572}