<jupyter_start><jupyter_text>Ride-sharing Analysis You're working as an analyst for Zuber, a new ride-sharing company that's launching in Chicago. Your task is to find patterns in the available information. You want to understand passenger preferences and the impact of external factors on rides.Working with a database, you'll analyze data from competitors and test a hypothesis about the impact of weather on ride frequency.  Import Packages<jupyter_code>import pandas as pd 
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats
from scipy.stats import zscore
!pip install pandas-profiling[notebook]
from pandas_profiling import ProfileReport
import warnings
warnings.filterwarnings('ignore')<jupyter_output>Requirement already satisfied: pandas-profiling[notebook] in c:\users\bnoah\anaconda3\lib\site-packages (3.1.1)
Requirement already satisfied: visions[type_image_path]==0.7.4 in c:\users\bnoah\appdata\roaming\python\python38\site-packages (from pandas-profiling[notebook]) (0.7.4)
Requirement already satisfied: multimethod>=1.4 in c:\users\bnoah\appdata\roaming\python\python38\site-packages (from pandas-profiling[notebook]) (1.6)
Requirement already satisfied: tangled-up-in-unicode==0.2.0 in c:\users\bnoah\anaconda3\lib\site-packages (from pandas-profiling[notebook]) (0.2.0)
Requirement already satisfied: matplotlib>=3.2.0 in c:\users\bnoah\anaconda3\lib\site-packages (from pandas-profiling[notebook]) (3.4.3)
Requirement already satisfied: pydantic>=1.8.1 in c:\users\bnoah\appdata\roaming\python\python38\site-packages (from pandas-profiling[notebook]) (1.8.2)
Requirement already satisfied: phik>=0.11.1 in c:\users\bnoah\appdata\roaming\python\python38\site-packages (from pandas-profilin[...]<jupyter_text>Details | Package          | Description                                                          ||:-----------------|:---------------------------------------------------------------------|| pandas           | For data preprocessing and basic descriptive statistics.             || matplotlib       | For creating graphs.                                                 || numpy            | For scientific computing.                                            || scipy            | For calculating statistics.                                          || seaborn          | For creating graphs.                                                 || pandas_profiling | For creating a profile report on the dataset.                        || warnings         | For ignoring warnings after finishing the project for readability.   |  Importing and Cleaning Data  Import Data<jupyter_code>try: 
    taxi_name = pd.read_csv('/Users/bnoah/data/project_sql_result_01.csv')
except: 
    taxi_name = pd.read_csv('/datasets/project_sql_result_01.csv')

try: 
    taxi_dropoff = pd.read_csv('/Users/bnoah/data/project_sql_result_04.csv')
except: 
    taxi_dropoff = pd.read_csv('/datasets/project_sql_result_04.csv')<jupyter_output><empty_output><jupyter_text>Profile Report  Taxi Name<jupyter_code>taxi_name_report = ProfileReport(taxi_name, title="Taxi Name Profiling Report")
taxi_name_report.to_widgets()<jupyter_output><empty_output><jupyter_text>Taxi Drop-off<jupyter_code>taxi_dropoff_report = ProfileReport(taxi_dropoff, title="Taxi Drop-off Profiling Report")
taxi_dropoff_report.to_widgets()<jupyter_output><empty_output><jupyter_text>Conclusions From importing the data, we can see that each dataframe has two columns. The first column is a string variable while the second column is a numerical variable. The taxi_name dataframe shows us the number of trips each cab company had from November 15 to 16th in 2017. Looking at the summary statistics, we can see there are 64 cab companies with an average of 2,145 trips per company. The maximum number of trips a company had was 19,558 and the minimum number of trips a company had was 2. The taxi_dropoff dataframe shows us the average number of trips each drop-off location had in November of 2017. Looking at the summary statistics, we can see there are 94 drop-off locations with an average of 600 trips per location. The maximum average number of trips a location had was 10,727.5 and the minimum average number of trips a location had was 1.8.   Graphing Data<jupyter_code>#making taxi cab company names shorter
taxi_name['company_name'].replace('Taxi Affiliation Service Yellow', 'TAS Yellow',inplace=True)
taxi_name['company_name'].replace('Taxi Affiliation Services', 'TAS',inplace=True)
taxi_name['company_name'].replace('Chicago Carriage Cab Corp', 'Chicago Carriage',inplace=True)
taxi_name['company_name'].replace('Star North Management LLC', 'Star North',inplace=True)
taxi_name['company_name'].replace('Blue Ribbon Taxi Association Inc.', 'Blue Ribbon',inplace=True)
#creating graph
fig, ax = plt.subplots(2,1,figsize=(15, 15))
#creating a boxplot for each Platform based on Total Sales
sns.barplot(ax=ax[0], x='dropoff_location_name', y='average_trips', data= taxi_dropoff.nlargest(10,'average_trips'),palette='mako');
ax[0].set_title('Top 10 Most Common Dropoff Locations in November 2017');
ax[0].set_ylabel('Average Trips');
ax[0].set_xlabel('Location');
#creating a bar graph for the average sales per year for each platform. Reindexed to match boxplot.
sns.barplot(ax=ax[1],x='company_name',y='trips_amount',data =taxi_name.nlargest(10,'trips_amount'),palette="magma_r");
ax[1].set_title('Top 10 Total Trips Per Company From November 15-16 2017');
ax[1].set_ylabel('Number of Trips');
ax[1].set_xlabel('Company');
#adding mean lines
mean_dropoff = taxi_dropoff.nlargest(10,'average_trips')['average_trips'].mean()
mean_name = taxi_name.nlargest(10,'trips_amount')['trips_amount'].mean()
mean_line_0 = ax[0].axhline(mean_dropoff, color='black', linewidth=1);
mean_line_1 = ax[1].axhline(mean_name, color='black', linewidth=1);
#adding arrow and mean value on the side
ax[0].annotate('Mean: {:0.2f}'.format(mean_dropoff),xy=(1,mean_dropoff), xycoords=('axes fraction','data'),xytext=(1.05,mean_dropoff-1000),arrowprops=dict(arrowstyle='-|>', fc='black', shrinkA=0, shrinkB=0,
                            connectionstyle='arc'))
ax[1].annotate('Mean: {:0.2f}'.format(mean_name),xy=(1,mean_name), xycoords=('axes fraction','data'),xytext=(1.05,mean_name+1000),arrowprops=dict(arrowstyle='-|>', fc='black', shrinkA=0, shrinkB=0,
                            connectionstyle='arc'))             
#adding y axis gridlines
ax[0].set_axisbelow(True);
ax[0].yaxis.grid(zorder=0);
ax[1].set_axisbelow(True);
ax[1].yaxis.grid(zorder=0);
#adding space betweeen the graphs
plt.subplots_adjust(hspace = 0.5);<jupyter_output><empty_output><jupyter_text>Conclusions From the top graph, we can see there are a clear top 4 dropoff locations, Loop, River North, Streeterville, and West Loop. Each of the top four seem to have clear reasons for being at the top.- The Loop in Chicago is the cities central business hub.- River North and West Loop are known for their restaurants and art galleries. - Streeterville is known for its shopping and beaches. [US News][1] For the bottom graph, we can see Flash Cab was by far the most popular cab company with the rest of the top 10 being fairly similar.[1]: https://travel.usnews.com/Chicago_IL/Area_Map/  Testing Hypotheses  Importing Data<jupyter_code>try: 
    taxi_airport_trips = pd.read_csv('/Users/bnoah/data/project_sql_result_07.csv')
except: 
    taxi_airport_trips = pd.read_csv('/datasets/project_sql_result_07.csv')<jupyter_output><empty_output><jupyter_text>Profile Report<jupyter_code>taxi_airport_trips_report = ProfileReport(taxi_airport_trips, title="Taxi Airport Trips Profiling Report")
taxi_airport_trips_report.to_widgets()<jupyter_output><empty_output><jupyter_text>Conclusions The taxi_airport_trips dataframe has three variables:1. `start_ts` which is the date and time of the ride pickup.2. `weather_conditions` which is the weather conditions at the time of the ride starting.3. `duration_seconds` which is the length of the ride in seconds.`weather_conditions` is a categorical variable with only two states, Good and Bad. In this dataset, the weather is good over 80\% of the time. Considering this data is from Chicago in November, I am a bit skeptical about having that high a number of good weather days. `duration_seconds` is a numeric variable with an mean ride length of 2071 seconds (~35 minutes). The distribution seems fairly normal except for a spike around 25 minutes. In practice, I would want to consult the data engineer to make sure there are not any explanations for this spike.   Hypothesis Testing<jupyter_code>alpha = 0.05
fig, ax = plt.subplots(1,2,figsize=(10, 4))
#histograms of the density 
sns.kdeplot(ax=ax[0], data = taxi_airport_trips[taxi_airport_trips['weather_conditions'] == 'Good']['duration_seconds'], label = ['Good Weather'], shade=True, color='blue');
sns.kdeplot(ax=ax[0], data = taxi_airport_trips[taxi_airport_trips['weather_conditions'] == 'Bad']['duration_seconds'],label = ['Bad Weather'], shade=True, color ='green');
ax[0].set_title("Distributions");
ax[0].set_ylabel("Density");
ax[0].set_xlabel("Duration(Seconds)");
#boxplots 
box_plot = sns.boxplot(ax=ax[1], x='weather_conditions', y='duration_seconds', data=taxi_airport_trips)
ax[1].set_title("Boxplots");
ax[1].set_ylabel("Duration)(Seconds)");
ax[1].set_xlabel("Weather Conditions");
#adding the median values to the middle of the boxplots
ax = box_plot.axes
lines = ax.get_lines()
categories = ax.get_xticks()

#function to calculate the location of the median and design the format of the box with the median value in it.
for cat in categories:
    # every 4th line at the interval of 6 is median line
    # 0 -> p25 1 -> p75 2 -> lower whisker 3 -> upper whisker 4 -> p50 5 -> upper extreme value
    y = round(lines[4+cat*6].get_ydata()[0],1) 
#formatting of the median value box
    ax.text(
        cat, 
        y, 
        f'{y}', 
        ha='center', 
        va='center', 
        fontweight='bold', 
        size=10,
        color='white',
        bbox=dict(facecolor='#445A64'))

#t-test 
results = stats.ttest_ind(taxi_airport_trips[taxi_airport_trips['weather_conditions'] == 'Good']['duration_seconds'],taxi_airport_trips[taxi_airport_trips['weather_conditions'] == 'Bad']['duration_seconds'],equal_var=True)
print('Null Hypothesis: Saturdays with good weather and bad weather have the same lengths of taxi trips to the airport from The Loop')
print('Alternative Hypothesis: Saturdays with good weather and bad weather have different lengths of taxi trips to the airport from The Loop')
print('\n')
#pull the p-value from the results
print('p-value: ', results.pvalue)
print('\n')
#compare the p-value to our alpha level
if results.pvalue < alpha:
    print("We reject the null hypothesis")
else:
    print("We can't reject the null hypothesis") 
print('\n')<jupyter_output>Null Hypothesis: Saturdays with good weather and bad weather have the same lengths of taxi trips to the airport from The Loop
Alternative Hypothesis: Saturdays with good weather and bad weather have different lengths of taxi trips to the airport from The Loop


p-value:  6.517970327099473e-12


We reject the null hypothesis
{'fraction_non_alphanumeric': 0.11246092005359536, 'fraction_numerical': 0.022420723537293435, 'mean_word_length': 4.564612326043737, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '365461', 'license': 'MIT', 'n_tokens_mistral': 3562, 'n_tokens_neox': 3322, 'n_words': 1044, 'path': 'data_collection_and_storage/data_collection_and_storage.ipynb', 'repo_name': 'bennoah1216/Practicum-Projects', 'size': '11195'}