<commit_before>## Problem 4.3. Twitter: Regular Expressions.

A template for this problem is provided:
[twitter_regex.ipynb](twitter_regex.ipynb)

In this problem, you will use regular expressions to process real Twitter data.

(If you are curious where this data came from, 1,000 tweets were fetched using the search query `#informatics` on Tuesday, February 3rd, 2015, around 10pm.  See [Mining the Social Web 2nd Edition](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition) by <NAME>.  [twitter_to_ascii.py](twitter_to_ascii.py) on the course Github repository is a Python 2 script that collects 1,000 tweets and generates an ASCII text file of status texts.)

If you used `git clone`/`git pull` from `/data`, the path to the data file will be `/data/spring2015/week04/informatics.txt`. It's an ASCII text file, and when you first get a new data like this, it's a good idea to see how the data looks like. You have learned various ways to do this: you could write a short Python script that reads the file line by line and print them out; you could mount the data container onto a Docker container in terminal mode and use `head` or `less`.

But here I'll show you a very useful Docker command that lets you enter 
  a running container. Here, we want to use `head` on the `informatics.txt`
  file in `/data` directory. Your notebook server should have `/data` mounted,
  so we will access this container in an interactive, terminal mode.
  In Boot2Docker (or in your host's command prompt), run `docker ps -a`
  to find the name of the container you want to have access to.
  Say the name is `insane_newton`. Run
  
  ```console
  $ docker exec -it insane_newton /bin/bash
  ```
  
  You should have the command prompt of the container that is running the
  IPython notebook server. Now you can do (in the container)

  ```console
  root@container:~# head /data/spring2015/week04/informatics.txt
  ```

The output should be
```console
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
```
<commit_msg>Update problem 4.3 directions.<commit_after>## Problem 4.3. Twitter: Regular Expressions.

A template for this problem is provided:
[twitter_regex.ipynb](twitter_regex.ipynb)

In this problem, you will use regular expressions to process real Twitter data.

(If you are curious where this data came from, 1,000 tweets were fetched using the search query `#informatics` on Tuesday, February 3rd, 2015, around 10pm.  See [Mining the Social Web 2nd Edition](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition) by <NAME>.  [twitter_to_ascii.py](twitter_to_ascii.py) on the course Github repository is a Python 2 script that collects 1,000 tweets and generates an ASCII text file of status texts.)

If you used `git clone`/`git pull` from `/data`, the path to the data file will be `/data/spring2015/week04/informatics.txt`. It's an ASCII text file, and when you first get a new data like this, it's a good idea to see how the data looks like. You have learned various ways to do this: you could write a short Python script that reads the file line by line and print them out; you could mount the data container onto a Docker container in terminal mode and use `head` or `less`.

But here I'll show you a very useful Docker command that lets you enter 
  a running container. Here, we want to use `head` on the `informatics.txt`
  file in `/data` directory. Your notebook server should have `/data` mounted,
  so we will access this container in an interactive, terminal mode.
  In Boot2Docker (or in your host's command prompt), run `docker ps -a`
  to find the name of the container you want to have access to.
  Say the name is `insane_newton`. Run
  
  ```console
  $ docker exec -it insane_newton /bin/bash
  ```
  
  You should have the command prompt of the container that is running the
  IPython notebook server. Now you can do (in the container)

  ```console
  root@container:/# head /data/spring2015/week04/informatics.txt
  ```

The output should be
```console
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @vikpant: #SocialMedia #BigData #Mining May Improve Population #Health -
HIA. http://t.co/YlVwRJb9rH #analytics #informatics #datascienc
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
RT @KirkDBorne: Feature Selection based on #MachineLearning in MRIs:
http://t.co/fwdZRLaSMd Medical imaging #informatics #DataScience
```

{'fraction_non_alphanumeric': 0.0858548218810553, 'fraction_numerical': 0.011924280816813236, 'mean_word_length': 4.7106382978723405, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2428398', 'n_tokens_mistral': 2250, 'n_tokens_neox': 2032, 'n_words': 861}