<jupyter_start><jupyter_text>Analyzing the NYC Subway Dataset Section 1: Statistical TestStatistical test that is choosen to analyze this dataset is the **Mann-Whitney U Test** because as will be seen below, the distribution of the sample is not following a Gaussian or Normal distribution. Therefore, it is appropriate to use Mann-Whitney U test, because unlike Welch's T test, it does not assume that the data come from any probability distribution.Furthermore, a testing could be conducted to strengthen the decision of using Mann-Whitney U test by performing **Shapiro-Wilk Test**, which will test whether the distribution of the sample is normally distributed.Moreover, since Mann-Whitney U test only tests whether the two samples come from the same population and not necessarily tells the direction of the test, the test will be a **two-tail test**. P-Critical Value$$P_{\text{Critical}} = .05$$ Hypotheses**Null Hypothesis**: The samples of entries on raining days and entries on clear days came from the same general population.$$H_0: P_{\text{(Rain = Clear)}} \geq 0.5$$**Alternative Hypothesis**: The samples of entries on on raining days and entries on clear days came from different populations.$$H_1: P_{\text{(Rain = Clear)}} < 0.5$$<jupyter_code>import pandas as pd
import numpy as np
import scipy.stats

# The path of the dataset CSV file
path = r'improved-dataset/turnstile_weather_v2.csv'

dataFrame = pd.read_csv(path)

# Storing data of entries when rain and when clear
withRain = dataFrame['ENTRIESn_hourly'][dataFrame['rain'] == 1]
withoutRain = dataFrame['ENTRIESn_hourly'][dataFrame['rain'] == 0]

# Calculating means of samples
mean_withRain = np.mean(withRain)
mean_withoutRain = np.mean(withoutRain)

# Calculating Mann-Whitney U Test
U, p = scipy.stats.mannwhitneyu(withRain, withoutRain)

print 'Mean of sample entries when raining: ', mean_withRain
print 'Mean of sample entries when clear: ', mean_withoutRain
print 'Test Statistics (U value): ', U

# P-value is multiplied by 2 because the result from Scipy is from a one-tail test
print 'P-Value: ', p * 2<jupyter_output>Mean of sample entries when raining:  2028.19603547
Mean of sample entries when clear:  1845.53943866
Test Statistics (U value):  153635120.5
P-Value:  5.48213914249e-06<jupyter_text>Results- Null Hypothesis is rejected because:$$P_{\text{Value}} < P_{\text{Critical}}$$- It is concluded that there is a probability of less than five percent for obtaining the U value and the samples come from the same population (Less than 5% probability of having a Type-I-Error).- It is reasonable to conclude that the samples come from different populations and there is a significance difference in number of NYC subway entries when it is raining and when it is not raining.  Section 2: Linear Regression ComputationComputation of the linear regression is conducted using **Statsmodels** and **Ordinary Least Squares** method.**Input variables** used in this linear regression are these:  - Hour  - Day of the Week  - Rain  - Average Temperature  This linear regression also uses the **unit variable** as a dummy variable to better predict ouput variable.The input variables are chosen due to experimentations and that they produce high value of R-squared when included.<jupyter_code>import statsmodels.api as sm

# Function to compute Linear Regression
def linear_regression(features, values):
    '''
    This function takes input and output variables to return
    intercept and parameters as a result of ordinary least
    squares liner regression.
    '''
    # Adding variable to calculate intercept
    features = sm.add_constant(features)
    
    # Calculating linear regression using Ordinary Least Square
    model = sm.OLS(values, features)
    results = model.fit()
    
    # Storing results
    intercept = results.params[0]
    params = results.params[1:]
    
    return intercept, params

# Function to compute predictions
def predictions(features, intercept, params):
    '''
    This function takes input variables and their parameters
    and intercept to compute to return prediction values.
    '''
    return intercept + np.dot(features, params)

# Function to compute R^2
def compute_r_squared(data, predictions):
    '''
    This function takes output variables and predictions values
    to return the computed R Squared value.
    '''
    # Sum of squared difference between data and predictions
    dp_diff = np.square(data - predictions).sum()
    
    # Sum of squared difference between data and its mean
    dmean_diff = np.square(data.map(lambda x: x - np.mean(data))).sum()
    
    r_squared = 1 - (dp_diff / dmean_diff)
    
    return r_squared

# Storing input and output variables that would be used in the model
inputVars = ['hour', 'day_week', 'rain', 'meantempi']
weights = dataFrame[inputVars]
outputs = dataFrame['ENTRIESn_hourly']

# Adding dummy variable to the weights
dummy_units = pd.get_dummies(dataFrame['UNIT'], prefix='unit')
weights = weights.join(dummy_units)

# Compute Linear Regression
intercept, params = linear_regression(weights, outputs)

# Compute prediction values
predictionValues = predictions(weights, intercept, params)

# Compute R Squared
r_squared = compute_r_squared(outputs, predictionValues)

print 'Intercept: ', intercept, '\n'
print 'Parameters:\n', params[:len(inputVars)], '\n'
print 'R Squared: ', r_squared<jupyter_output>Intercept:  1946.69132875 

Parameters:
hour         123.329727
day_week    -144.708298
rain          36.169638
meantempi    -14.899818
dtype: float64 

R Squared:  0.470411121826<jupyter_text>ResultsThe R-squared obtained above is a good measure to tell whether the predictions produce more errors. Therefore, the linear regression obtained above show that the around 47% of the response variable can be explained by the model.However, the measure of goodness-of-fit might be best calculated with the **standard-error of the regression**, which is not calculated in this analysis, since even a high R-squared value might have a large bias.  Section 3: Visualization<jupyter_code>%matplotlib inline
from ggplot import *

entries_and_rain = dataFrame[['rain', 'ENTRIESn_hourly']]

print "Sample size of entries with rain: ", len(entries_and_rain[entries_and_rain['rain'] == 1])
print "Sample size of entries without rain: ", len(entries_and_rain[entries_and_rain['rain'] == 0])

ggplot(entries_and_rain, aes(x='ENTRIESn_hourly', fill='rain', color='rain')) + geom_histogram(binwidth=300) +\
ylim(0,8000) + xlim(0,15000) + xlab('Hourly Ridership') + ylab('Frequency') +\
ggtitle('Histogram of Ridership with and without Rain')<jupyter_output>Sample size of entries with rain:  9585
Sample size of entries without rain:  33064<jupyter_text>- Sample size of entries with rain: 9,585- Sample size of entries without rain: 33,064- The y-axis is truncated since the frequency of ridership when raining spans to more than 10,000 occurrence.- The x-axis is truncated since the ridership spans to more than 35,000 riderships.The visualization above related to the point made in section 1. Since the histogram above shows that the distributions of the samples are positively skewed, the appropriate test statistics is the Mann-Whitney U test.Furthermore, the histogram above shows a large difference in sample size between two sample categories that is shown by the number of frequencies between two categories.<jupyter_code>entries_and_day = dataFrame[['day_week', 'ENTRIESn_hourly']].groupby('day_week',\
                      as_index=False).sum()
ggplot(entries_and_day, aes('day_week', 'ENTRIESn_hourly')) + geom_point() + geom_line() +\
xlab('Day of Week (0 = Monday and 6 = Sunday)') + ylab('Number of Entries (x 10^7)') +\
ggtitle('NYC Subway Entries by Day of Week')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08200718685831622, 'fraction_numerical': 0.02335728952772074, 'mean_word_length': 4.811334824757644, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '351099', 'license': 'MIT', 'n_tokens_mistral': 2377, 'n_tokens_neox': 2119, 'n_words': 906, 'path': 'dand-nyc-subway.ipynb', 'repo_name': 'WisnuMulya/dataanalyst-nanodegree-nyc-subway', 'size': '7792'}