<commit_before>---
layout: single
title: Which MTA station should you assign people to?
modified:
excerpt:
tags: [pandas]
image: 'assets/images/500.jpg'
    feature:
date: 2017-01-12T10:35:45-00:00
--- 

Recently Metis received a new project from `WomenTechWomenYes (WTWY)` with blow request:

>WomenTechWomenYes (WTWY) has an annual gala at the beginning of the summer each year. As we are new and inclusive organization, we try to do double duty with the gala both to fill our event space with individuals passionate about increasing the participation of women in technology, and to concurrently build awareness and reach.  
>To this end we place street teams at entrances to subway stations. The street teams collect email addresses and those who sign up are sent free tickets to our gala.  
>Where we’d like to solicit your engagement is to use MTA subway data, which as I’m sure you know is available freely from the city, to help us optimize the placement of our street teams, such that we can gather the most signatures, ideally from those who will attend the gala and contribute to our cause.

The request specifically mentioned about dataset from NYC MTA subway data since the major task here is to find out the best location for street team placement, possibly with the busiest traffic. Therefore, even though there are many other available data sources in the scenario, due to time constrain, our first project was mostly based upon of the MTA data.   

### Data Exploration  

Cleaning the MTA data is much harder than expected and most of time we found new issues with the dataset when we started data visualizations.   

The few things we noticed in the first places included:

- The entry/exit count is an **accumulative number** which needs to be normalize by each count.  

- Each station has at least one but most stations have few turnstiles, each turnstile has its Control Area code and SCP ID.  

- Some counts were **reset to be 0** when previous count is above 1 million.

*Later on during the exploration process, we discovered few other issues:*

- Most counts were measured every 4 hours however there are some very peculiar count intervals, few were less than 20 minutes.  

- Some exact same stations were listed with different names probably capitalization error or typo, etc.
> For example, '14th ST' is equal to '14TH ST'.  <commit_msg>Update 2017-01-12-Blog.md<commit_after>---
layout: single
title: Which MTA station should you assign people to?
modified:
excerpt:
tags: [pandas]
image: assets/images/500.jpg
    feature:
date: 2017-01-12T10:35:45-00:00
--- 

Recently Metis received a new project from `WomenTechWomenYes (WTWY)` with blow request:

>WomenTechWomenYes (WTWY) has an annual gala at the beginning of the summer each year. As we are new and inclusive organization, we try to do double duty with the gala both to fill our event space with individuals passionate about increasing the participation of women in technology, and to concurrently build awareness and reach.  
>To this end we place street teams at entrances to subway stations. The street teams collect email addresses and those who sign up are sent free tickets to our gala.  
>Where we’d like to solicit your engagement is to use MTA subway data, which as I’m sure you know is available freely from the city, to help us optimize the placement of our street teams, such that we can gather the most signatures, ideally from those who will attend the gala and contribute to our cause.

The request specifically mentioned about dataset from NYC MTA subway data since the major task here is to find out the best location for street team placement, possibly with the busiest traffic. Therefore, even though there are many other available data sources in the scenario, due to time constrain, our first project was mostly based upon of the MTA data.   

### Data Exploration  

Cleaning the MTA data is much harder than expected and most of time we found new issues with the dataset when we started data visualizations.   

The few things we noticed in the first places included:

- The entry/exit count is an **accumulative number** which needs to be normalize by each count.  

- Each station has at least one but most stations have few turnstiles, each turnstile has its Control Area code and SCP ID.  

- Some counts were **reset to be 0** when previous count is above 1 million.

*Later on during the exploration process, we discovered few other issues:*

- Most counts were measured every 4 hours however there are some very peculiar count intervals, few were less than 20 minutes.  

- Some exact same stations were listed with different names probably capitalization error or typo, etc.
> For example, '14th ST' is equal to '14TH ST'.  
{'fraction_non_alphanumeric': 0.04173693086003373, 'fraction_numerical': 0.01433389544688027, 'mean_word_length': 4.202850877192983, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4978318', 'n_tokens_mistral': 1231, 'n_tokens_neox': 1137, 'n_words': 755}