<jupyter_start><jupyter_text>Check the lenght of the data<jupyter_code>length = len(df)
length<jupyter_output><empty_output><jupyter_text>Univariate Analysis to determine the time range of data setWe sort the data frame by `starttime`what sorting did was to change the position of records within the data frame, and hence the change in positions disturbed the arrangement of the indexes which were earlier in an ascending order. Hence, considering this, we decided to reset the indexes so that the ordered data frame now has indexes in an ascending order. Finally, we printed the date range that started from the first value of starttime and ended with the last value of stoptime<jupyter_code>df2 = df.sort('starttime')
df2
df2.reset_index(inplace=True)
df2
print("Data Range is from: ",'[',df2.loc[1,'starttime'],']',' to ','[',df2.loc[len(df2)-1,'stoptime'],']')<jupyter_output>Data Range is from:  [ 1/1/2015 0:37 ]  to  [ 4/9/2015 10:09 ]<jupyter_text>There are two insights. One is that the data ranges from October 2014 up till September 2016 (i.e., three years of data). Moreover, it seems like the cycle sharing service is usually operational beyond the standard 9 to 5 business hours. new users would be short-term pass holders however once they try out the service and become satisfied would ultimately avail the membership to receive the perks and benefits offeredplot a bar graph of trip frequencies by user type groupby groups the data by a given field, that is, usertype, in the current situationsize counts trips falling within each of the grouped usertypes<jupyter_code>fre_usr_stat = df2.groupby('usertype').size()
fre_usr_stat
fre_usr_stat.plot.barh(figsize=(5,5),title='FREQUENCY-USER-GRAPH')<jupyter_output><empty_output><jupyter_text>You may want know more about her target customers to whom to company’s marketing message will be targetted toFind out the gender and age groups that are most likely to ride a cycle or the ones that are more prone to avail the service.Fing out the distribution of gender and birth years<jupyter_code>gender_stat = df2.groupby('gender').size()
gender_stat
gender_stat.plot.barh(figsize=(10,5),title='GENDER GRAPH')
birth_stat = df2.groupby('birthyear').size()
birth_stat
birth_stat.plot.bar(figsize=(15,5),title='BIRTH-YEAR-GRAPH')<jupyter_output><empty_output><jupyter_text>millenials are supposed to make a conscious decision, a decision they will remain loyal to for a long period. Hence we mat assume that the most millennials would be members rather than short-term pass holders. Based on data below, it appeared to be valid, and we can make sure that the brand engaged millennials as part of the marketing plan.<jupyter_code>mill_stat = df2.groupby('usertype')['birthyear'].size()
mill_stat
mill_stat.plot.barh(figsize=(10,5))<jupyter_output><empty_output><jupyter_text>Multivariate Analysis Multivariate analysis refers to incorporation of multiple exploratory variables to understand the behavior of a response variable. This seems to be the most feasible and realistic approach considering the fact that entities within this world are usually interconnected. Thus the variability in response variable might be affected by the variability in the interconnected exploratory variables Who completed more trips male or female. Draw a stacked bar graph (i.e., a bar graph for birth year, but each bar having two colors, one for each gender) We at first transformed the data frame by **unstacking**, that is, splitting, the gender column into three columns, that is, Male, Female, and Other. This meant that for each of the birth years we had the trip count for all three gender types. Finally, a stacked bar graph was created by using this transformed data frame.<jupyter_code>birth_gen_stat = df2.groupby(['birthyear', 'gender'])['birthyear'].count()
birth_gen_stat = birth_gen_stat.unstack('gender')
birth_gen_stat = birth_gen_stat.fillna(0)
birth_gen_stat
birth_gen_stat[['Male','Female','Other']].plot.bar(figsize=(15,5),stacked=True,title='BIRTH-YEAR-GENDER-GRAPH')<jupyter_output><empty_output><jupyter_text>It seemed as if males were dominating the distribution. It made sense as well. No? Well, it did; as seen earlier, that majority of the trips were availed by males, hence this skewed the distribution in favor of males. However, subscribers born in 1947 were all females. Moreover, those born in 1964 and 1994 were dominated by females as wel what the distribution of user type was for the other age generations. Is it that the majority of people in the other age generations were short-term pass holders?<jupyter_code>birth_usr_stat = df2.groupby(['birthyear', 'usertype'])['birthyear'].count()
birth_usr_stat = birth_usr_stat.unstack('usertype')
birth_usr_stat = birth_usr_stat.fillna(0)
birth_usr_stat
birth_usr_stat[['Member']].plot.bar(figsize=(15,5),title='BIRTH-YEAR-USER-GRAPH')<jupyter_output><empty_output><jupyter_text>The graph shows the distribution of only one user type and not two (i.e., membership and short-term pass holders)? Does this mean that birth year information was only present for only one user type?Validation If We Don’t Have Birth Year Available for Short-Term Pass Holders<jupyter_code>df2[df2['usertype'] == ('Short-Term Pass Holder')]['birthyear'].isnull().values.all()<jupyter_output><empty_output><jupyter_text>We first sliced the data frame to consider only short-term pass holders. Then we went forward to find out if all the values in birth year are missing (i.e., null) for this slice. Members have to provide details like birth year when applying for the membership, something which is not a prerequisite for short-term pass holders. WE decided to test this deduction by checking if gender is available for short-term pass holders or not<jupyter_code>df2[df2['usertype'] == ('Short-Term Pass Holder')]['gender'].isnull().values.all()<jupyter_output><empty_output><jupyter_text>how the frequency of trips vary across date and time (i.e., a time series analysis). Trip start time is given with the data, but to make a time series plot, we had to transform the date from string to date time format. Split the datetime into date components (i.e., year, month, day, and hour ) At first we converted start time column of the dataframe into a list. Next we converted the string dates into python datetime objects. We then converted the list into a series object and converted the dates from datetime object to pandas date object. The time components of year, month, day and hour were derived from the list with the datetime objects.<jupyter_code>List_ = list(df2['starttime'])

List_ = [datetime.datetime.strptime(x, "%m/%d/%Y %H:%M") for x in List_]
df2['starttime_mod'] = pd.Series(List_,index=df2.index)
df2['starttime_date'] = pd.Series([x.date() for x in List_],index=df2.index)
df2['starttime_year'] = pd.Series([x.year for x in List_],index=df2.index)
df2['starttime_month'] = pd.Series([x.month for x in List_],index=df2.index)
df2['starttime_day'] = pd.Series([x.day for x in List_],index=df2.index)
df2['starttime_hour'] = pd.Series([x.hour for x in List_],index=df2.index)
df2.groupby('starttime_date')['tripduration'].mean().plot.bar(title = 'Distribution of Trip duration by date', figsize = (20,5))<jupyter_output><empty_output><jupyter_text>Determining the measures of centers using the statistics package will require us to transform the input data structure to a list type.<jupyter_code>trip_duration = list(df2['tripduration'])
station_from = list(df2['from_station_name'])
print ('Mean of trip duration: %f'%statistics.mean(trip_duration))
print ('Median of trip duration: %f'%statistics.median(trip_duration))
print ('Mode of station originating from: %s'%statistics.mode(station_from))<jupyter_output>Mean of trip duration: nan
Median of trip duration: 596.196500
Mode of station originating from: REI / Yale Ave N & John St<jupyter_text>The output revealed that most trips originated from Pier 69/Alaskan Way & Clay St station. Hence this was the ideal location for running promotional campaigns targeted to existing customers. Moreover, the output showed the mean to be greater than that of the mean.why the average (i.e., mean) is greater than the central value (i.e., median).this might be either due to some extreme values after the median or due to the majority of values lying after the median. plot a distribution of the trip durations in order to determine which premise holds true .<jupyter_code>df2['tripduration'].plot.hist(bins=100,title='Frequency distribution of Trip duration',figsize=(5,5))<jupyter_output><empty_output><jupyter_text>The distribution has only one peak (i.e., mode). The distribution is not symmetric and has majority of values toward the right-hand side of the mode. These extreme values toward the right are negligible in quantity, but their extreme nature tends to pull the mean toward themselves. Thus the reason why the mean is greater than the medianThe distribution is referred to as a **normal distribution.** Find out whether outliers exist within our dataset—more precisely in the tripduration feature. For that first create a box plot by writing code to see the outliers visually and then checked the same by applying the interval calculation method<jupyter_code>box = df2.boxplot(column=['tripduration'])<jupyter_output><empty_output><jupyter_text>See a huge number of outliers in trip duration from the box plot. She asked Eric if he could determine the proportion of trip duration values which are outliers. She wanted to know if outliers are a tiny or majority portion of the dataset<jupyter_code>q75, q25 = np.percentile(trip_duration, [75 ,25])
iqr = q75 - q25
print ('Proportion of values as outlier: %f percent'%(
(len(df2) - len([x for x in trip_duration if q75+(1.5*iqr) >=x>= q25-(1.5*iqr)]))*100/float(len(df2))))<jupyter_output>Proportion of values as outlier: 100.000000 percent<jupyter_text>Q3 refers to the 75th percentile and Q1 refers to the 25th percentile. Hence we use the numpy.percentile() method to determine the values for Q1 and Q3. Next we compute the IQR by subtracting both of them. Then we determine the subset of values by applying the interval. We then used the formula to get the number of outliers. In our code, len(data) determines Length of all values and Length of all non outliers values is determined by `len([x for x in trip_duration if q75+(1.5*iqr) >=x>= q25-(1.5*iqr)]))` Considering the time series nature of the dataset, removing these outliers wouldn’t be an option. The only option we could rely on was to apply transformation to these outliers to negate their extreme nature. However, we are interested in observing the mean of the non-outlier values of trip duration. This we can compare with the mean of all values calculated earlier<jupyter_code>mean_trip = np.mean([x for x in trip_duration if q75+(1.5*iqr) >=x>= q25-(1.5*iqr)])
upper_whisker = q75+(1.5*iqr)
print('Mean of trip duration: %f'%mean_trip)<jupyter_output>Mean of trip duration: nan<jupyter_text>The mean of non-outlier trip duration values in (i.e., approximately 712) is considerably lower than that calculated in the presence of outliers in Listing 1-15 (i.e., approximately 1,203). This best describes the notion that mean is highly affected by the presence of outliers in the dataset. upper_whisker is the maximum value of the right (i.e., positive) whisker i.e. boundary uptill which all values are valid and any value greater than that is considered as an outlier WE are interested to see the outcome statistics once the outliers were transformed into valid value sets. Hence we decided to start with a simple outlier transformation to the mean of valid values<jupyter_code>def transform_tripduration(x):

    if x > upper_whisker:
        return mean_trip_duration
    return x

df2['tripduration_mean'] = df2['tripduration'].apply(lambda x: transform_tripduration(x))

print(df2['tripduration_mean'])

df2['tripduration_mean'].plot.hist(bins=100, title='Frequency distribution of mean transformed Trip duration')
plt.show()<jupyter_output>0         1403.479
1          459.469
2          255.004
3         1145.254
4          470.801
5          136.101
6         1294.421
7         1461.638
8          666.286
9          524.885
10       16390.548
11       16321.122
12         902.576
13         699.075
14         514.110
15         472.876
16        2502.464
17        2449.811
18         684.994
19         397.101
20         740.454
21        1463.476
22         682.798
23        1219.227
24        1132.444
25        1101.863
26        1057.551
27        1166.184
28         801.236
29        1111.246
           ...    
57032      465.089
57033      530.809
57034      410.825
57035      515.103
57036      566.387
57037      835.321
57038      385.658
57039      411.887
57040      667.543
57041      396.963
57042      394.039
57043      338.995
57044      430.216
57045     4741.051
57046      994.094
57047      587.449
57048      390.669
57049      175.865
57050      631.674
57051       98.209
57052      162.906
57053      7[...]<jupyter_text>We initialized a function by the name of transform_tripduration . The function will check if a trip duration value is greater than the upper whisker boundary value, and if that is the case it will replace it with the mean. Next we add tripduration_mean as a new column to the data frame. We did so by custom modifying the already existing tripduration column by applying the transform_tripduration function.  the skewness has now decreased to a great extent after the transformation. Moreover, the majority of the observations have a tripduration of 712 primarily because all values greater than the upper whisker boundary are now converted into the mean of the non-outlier values calculated. We are now interested in understanding how the center of measures appear for this transformed distribution. Hence we came up with the code<jupyter_code>print('Mean of trip duration: %f'%df2['tripduration_mean'].mean())
print('Standard deviation of trip duration: %f'%df2['tripduration_mean'].std())
print('Median of trip duration: %f'%df2['tripduration_mean'].median())<jupyter_output>Mean of trip duration: 1101.187491
Standard deviation of trip duration: 1953.504561
Median of trip duration: 601.272000<jupyter_text>We were expecting the mean to appear the same as previous because of the mean transformation of the outlier values. WE knew that the hike at 711.7 is the mode, which meant that after the transformation the mean is the same as that of the mode. The thing that surprised the most was that the median is approaching the mean, which means that the positive skewness we saw in is not that strong. Pearson R correlation into practice and decided to make a scatter plot between the two quantities for them to see the relationship visually.<jupyter_code>pd.set_option('display.width', 100)
pd.set_option('precision', 3)

df2['age'] = df2['starttime_year'] - df2['birthyear']

correlations = df2[['tripduration','age']].corr(method='pearson')
print(correlations)
df2 = df2.dropna()
sns.pairplot(df2, vars=['age', 'tripduration'], kind='reg')
plt.show()
for cat in ['gender','usertype']:
    print ('Category: %s\n'%cat)
    groupby_category = df2.groupby(['starttime_date', cat])['starttime_date'].count().unstack(cat)
    groupby_category = groupby_category.dropna()
    category_names = list(groupby_category.columns)

    for comb in [(category_names[i],category_names[j]) for i in range(len(category_names)) for j in range(i+1, len(category_names))]:

        print ('%s %s'%(comb[0], comb[1]))
        t_statistics = stats.ttest_ind(list(groupby_category[comb[0]]), list(groupby_category[comb[1]]))
        print ('Statistic: %f, P value: %f'%(t_statistics.statistic, t_statistics.pvalue))

        print ('\n')
"""SUMMERY"""
"""
At first we added some liberaries. Next we added the data file. Check its length and sort it by a column starttime and after it we print the range of 
data. 

After it we print the number of members and short-term pass holders and plot their graph.

We find the gender of all customer which is male, female or others. And then we plot a graph showing th gender of patients.

Next we get the birthyear of the costomer and plot the graph for that.

Aftr it we did some analysis and form graphs according to user type, birthyear and gender , birth year.

Next we converted the string dates into python datetime objects. We then converted the list into a series object and converted the dates from datetime 
object to pandas date object. The time components of year, month, day and hour were derived from the list with the datetime objects.

Next we did some statistics on the trip duration  column from the main data. And made some graphs to easily compare the data.

Next we determine the proportion of trip duration values which are outliers.

Next we see the outcome statistics once the outliers were transformed into valid value sets. Hence we decided to start with a simple outlier 
transformation to the mean of valid values

We are now interested in understanding how the center of measures appear for this transformed distribution. Hence we came up with the code to do so.

Next we make Pearson R correlation into practice and decided to make a scatter plot between the two quantities for them to see the relationship visually.

"""<jupyter_output><empty_output><jupyter_text>"""So now after analyzing the whole Project we can say that this project should be launched near ""REI / <NAME> & <NAME>"""""<jupyter_code>"""ありがとう"""<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07772522650863298, 'fraction_numerical': 0.04188272836059035, 'mean_word_length': 4.759763702001969, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '42968', 'license': 'MIT', 'n_tokens_mistral': 5438, 'n_tokens_neox': 4818, 'n_words': 2219, 'path': 'cycle-sharing-scheme-stats-Project1.ipynb', 'repo_name': 'GagandeepSingh011/Cycle-Sharing-Scheme-Stats-Project', 'size': '17554'}