<jupyter_start><jupyter_text>Data Wrangling  Student Name: <NAME><jupyter_code>#import any required libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import numpy as np<jupyter_output><empty_output><jupyter_text>Step 1 – Programmatically download the required data for December 2019.<jupyter_code>url = 'https://nyc-tlc.s3.amazonaws.com/trip+data/green_tripdata_2019-12.csv' #provide the link to the data for December 2019

#load the data into a dataframe
try:
    green_data = pd.read_csv(url, low_memory = False)
except IOError:
    print ('Unable to read the input file.')
    exit()<jupyter_output><empty_output><jupyter_text>Step 2 – Report the number of rows and columns in the data.<jupyter_code>#Print the number of rows and columns in data.
green_data.shape<jupyter_output><empty_output><jupyter_text>Step 3 – Gather summary/descriptive statistics. State your initial observations (no more than 1 paragraph) and identify any inconsistencies. HINT: you can use the **describe** function<jupyter_code>#display the summary statistics using the describe() function from the pandas dataframe
#state your observations or any inconsistencies that were detected in step 3
green_data.describe()<jupyter_output><empty_output><jupyter_text>Increased fare amounts lead to increased tip percentages which is interesting. The question is, do customers feel the need to tip more because they feel guilty about the long ride, is there a specific of drivers making these longer rides, do the longer rides allow customers more interaction with the drivers and thus a stronger connection leading to a better tip, or something else entirely. There are also very large trip distances as well as negative trip distances which could represent faulty devices.  Step 4 – Examine the data for missing observations and duplicate records. Suggest what methods are appropriate to handle them.<jupyter_code>#Perform the action(s) in step 4 to handle any missing data and duplicate records that were found.
print(green_data.count())
print(pd.DataFrame(green_data.duplicated(), columns = ['tf']).groupby('tf')['tf'].count())<jupyter_output>VendorID                 359902
lpep_pickup_datetime     450627
lpep_dropoff_datetime    450627
store_and_fwd_flag       359902
RatecodeID               359902
PULocationID             450627
DOLocationID             450627
passenger_count          359902
trip_distance            450627
fare_amount              450627
extra                    450627
mta_tax                  450627
tip_amount               450627
tolls_amount             450627
ehail_fee                     0
improvement_surcharge    450625
total_amount             450627
payment_type             359902
trip_type                359900
congestion_surcharge     359900
dtype: int64
tf
False    450626
True          1
Name: tf, dtype: int64<jupyter_text>There are over 90,000 rows of missing data in certain columns. Thankfully passenger count, payment type, trip type are the only columns that are of concern, the others don't play a major role in any analysis we would perform. While we could keep them all by filling them with the averages for their respective columns there is little variation in them and thus their presence in the analysis probably wouldn't create much change. There is also only a singular duplicate in this data set which we can honestly just ignore becaue it won't have any weight on the analysis.<jupyter_code>green_data = green_data.dropna(subset = ['trip_type'])
print(green_data.count())<jupyter_output>VendorID                 359900
lpep_pickup_datetime     359900
lpep_dropoff_datetime    359900
store_and_fwd_flag       359900
RatecodeID               359900
PULocationID             359900
DOLocationID             359900
passenger_count          359900
trip_distance            359900
fare_amount              359900
extra                    359900
mta_tax                  359900
tip_amount               359900
tolls_amount             359900
ehail_fee                     0
improvement_surcharge    359900
total_amount             359900
payment_type             359900
trip_type                359900
congestion_surcharge     359900
dtype: int64<jupyter_text>Step 5 – Plot a histogram showing the distribution of the trip distance. Discuss your observations (no more than 1 paragraph).<jupyter_code>plt.hist(np.array(green_data['trip_distance']), bins = 100)
plt.show()<jupyter_output><empty_output><jupyter_text>Step 6 – Did you detect any outliers in step 5? If so, perform a suitable method to handle them?<jupyter_code>green_data = green_data[green_data['trip_distance'] < 50]
green_data = green_data[green_data['trip_distance'] >= 0]<jupyter_output><empty_output><jupyter_text>Step 7 – Show the frequency of trips for each date. Do you see any interesting patterns or do you detect the need for additional data prep. rules.<jupyter_code>green_data['tpep_pickup_date'] = pd.to_datetime(green_data['lpep_pickup_datetime'])
green_data['tpep_pickup_date'] = green_data['tpep_pickup_date'].dt.normalize()
green_data.sort_values(by = ['tpep_pickup_date'], inplace = True)
green_date_data = green_data.groupby('tpep_pickup_date')['tpep_pickup_date'].count().to_frame()
green_date_data.loc['2019-12'].plot(linewidth = 0.5)<jupyter_output><empty_output><jupyter_text>You can see the cyclical nature of the volume. The demand increases slowly throughout the week and peaks on Saturday before bottoming out on Sunday. This gets completely disrupted during the week of Christmas, with many customers traveling to be with their families and employees going on vacation there is a dip there. On top of that other customers have burned through a decent of their expendable income for the holidays we take another hit on that front.  Step 8   - (a) Create a visualization that shows how people pay for their cab rides (e.g. credit card, cash, etc.). Can you say what is the most popular mode of payment?  - (b) Filter the data to only include customers who paid via credit card.<jupyter_code>green_pay_data = green_data.groupby('payment_type')['payment_type'].count()
plt.bar(green_pay_data.index, green_pay_data.values)
plt.xlabel('Payment Type')
plt.ylabel('Total Payments')
plt.title('Breakdown of Amount of Passengers by Payment Type')

credit_green = green_data[green_data['payment_type'] == 1]<jupyter_output><empty_output><jupyter_text>Step 9 –  Create a compelling visualization, based on your intuition, about the data and any useful insights on what factors contribute to cab driver's gratuity.   For example:   - Is there any correlation between the pickup/drop-off location and the gratuity?   - Does the trip distance or duration influence gratuity?<jupyter_code>x = np.arange(len(green_pmnt_tip.index))
width = 0.35

fig, ax = plt.subplots()
rects1 = ax.bar(x - width/2, green_pmnt_tip.values, width, label='Tips')
rects2 = ax.bar(x + width/2, green_pmnt_trip.values, width, label='Distance Traveled')
ax.set_ylabel('Total Dollar Amount')
ax.set_title('Payment Type')
ax.set_xticks(x)
ax.yaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
ax.set_xticklabels(green_pmnt_tip.index)
ax.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06649156759745646, 'fraction_numerical': 0.041470832181365776, 'mean_word_length': 4.238957277335264, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '170776', 'license': 'MIT', 'n_tokens_mistral': 2303, 'n_tokens_neox': 2066, 'n_words': 751, 'path': 'DataWrangling.ipynb', 'repo_name': 'clarkech95/data-wrangling', 'size': '7244'}