<jupyter_start><jupyter_text>NYC Taxi Data Science ChallengeBy <NAME> Viswanathan  IntroductionFor the challenge, we will use data collected by the New York City Taxi and Limousine commission about "Green" Taxis. Green Taxis (as opposed to yellow ones) are taxis that are not allowed to pick up passengers inside of the densely populated areas of Manhattan. We will use the data from September 2015. We are using NYC Taxi and Limousine trip record data: (http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml).We will also address various different questions as part of the analysis and build different models to predict the 'Tip_percentage' provided on each trip.<jupyter_code>#Import necessary libraries for data import, analysis and models
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import geopy.distance as geo
import datetime
from datetime import timedelta
import seaborn as sns
from math import cos, asin, sqrt
import pickle
import folium
import geohash
import os<jupyter_output><empty_output><jupyter_text>Loading the dataset ProgrammaticallyDataset can be loaded in different ways automatically. Here we are going get the year/month and load the dataset programmicaly. We can use other methods like robotic way to get the data. As this does not have any autentication, we will use simple method.<jupyter_code>#Get year,month 
year = str(input("Enter the year(4 digits): ")).zfill(4)
month = str(input("Enter the month number: ")).zfill(2)
url = 'https://s3.amazonaws.com/nyc-tlc/trip+data/green_tripdata_'+year+'-'+month+'.csv'

try:
    df = pd.read_csv(url,sep=',')
except:
    print("Error in inputs. Downloading default dataset of year 2015 and September month.")
    df = pd.read_csv('https://s3.amazonaws.com/nyc-tlc/trip+data/green_tripdata_2015-09.csv',sep=',')

print("Dataset loaded successfully.")<jupyter_output>Dataset loaded successfully.<jupyter_text>Before proceeding further lets glance through the dataset and perform basic summary statistics on the dataset.<jupyter_code>df.head()
df.info()
df.describe()
print("Total Observations: {}".format(df.count()[0]))
print("Number of Features: {}".format(len(df.columns)))
print('Column names: {}'.format(df.columns))<jupyter_output>Total Observations: 1494926
Number of Features: 21
Column names: Index(['VendorID', 'lpep_pickup_datetime', 'Lpep_dropoff_datetime',
       'Store_and_fwd_flag', 'RateCodeID', 'Pickup_longitude',
       'Pickup_latitude', 'Dropoff_longitude', 'Dropoff_latitude',
       'Passenger_count', 'Trip_distance', 'Fare_amount', 'Extra', 'MTA_tax',
       'Tip_amount', 'Tolls_amount', 'Ehail_fee', 'improvement_surcharge',
       'Total_amount', 'Payment_type', 'Trip_type '],
      dtype='object')<jupyter_text>Summary for Question 1:Below are the list of important points which are noted from above summary statistics.1. Total observations are highlighted in printed above.2. 'Ehail_fee' has no values. It is just `NaN`.3. For some records `Pickup_longitude, Pickup_latitude, Dropoff_longitude, Dropoff_latitude` is '0'. But seeing the minimum value.4. For some records `Passenger_count` is '0'5. `Trip_distance` is given as '0' for some records.6. `Fare_amount, Extra, MTA_tax, Tip_amount, Tolls_amount, improvement_surcharge, Total_amount ` is provided in negative number for some records.7. `Trip_type` has space in the column name.  Analysis on Trip DistanceLets deep dive into on `Trip_distance` feature and plot some charts for it. Also derive any hypotesis on it. From previous stats we found that `Trip_distance` is having `0` values.<jupyter_code>df.Trip_distance.plot(kind='hist')
plt.title("Histogram of Trip_distance")
plt.xlabel('Trip Duration')
plt.ylabel('Frequency of Trip Duration')<jupyter_output><empty_output><jupyter_text>It seems to be heavily skewed and most it seems most of the trip distance is between `0` and `50` miles. We will plot `Trip_distance` with other features to understand it better.<jupyter_code># Find Trip_distance per vendorID.
df.plot(x='VendorID',y='Trip_distance',kind='scatter')
plt.title("VendorID vs Trip_distance")<jupyter_output><empty_output><jupyter_text>Outliers Analysis It seems that trip there are couple of outliers for each vendors. Lets deepdive into those outliers and see if there are any anomalies and find great circle distance to see if those are valid.<jupyter_code>def calculate_distance(x):
    """Return great circle distance of pickup and dropoff point"""
    return(geo.great_circle((x.Pickup_latitude,x.Pickup_longitude),(x.Dropoff_latitude,x.Dropoff_longitude)).miles)
# Select the recods which has more travel time than 100 SD

mean = np.mean(df['Trip_distance'])
sd = np.std(df['Trip_distance'])
df[df['Trip_distance'] >= mean + 100*sd]
# Great circle distance for the outlier 

print("Great circle distance of outlier in Miles: {}".format( \
np.array(df[df['Trip_distance'] >= mean + 100*sd].apply(calculate_distance,axis=1))[0]))<jupyter_output>Great circle distance of outlier in Miles: 2.14152307442842<jupyter_text>Clearly second record is an outlier and wrong data. It has a fare amount of `2.55` and the travel distance around 600 miles. That is impossible and the great circle distance is `2.14 miles`. So the record seems to be wrongly entered into the system. For now we will reject the one outlier record.<jupyter_code># Drop the recods which has more travel time than 100 SD as it is an outlier
df.drop([df[df['Trip_distance'] >= mean + 100*sd].index.values[0]],inplace=True)<jupyter_output><empty_output><jupyter_text>Now, lets validate the hypothesis.1. Trip distance which are 0 and it has latitude and longitude2. Trip distance which are 0 and it does not have either latitude and longitude in pickup and drop.  1. Trip distance which are 0 and it has latitude and longitudeObservations are having a trip distance of 0 miles but has valid latitude and longitude in pickup/dropoff. So we can calculate great circle distance between two points. Which can be used as an approximate distance. A better way to do it is using the actual distance calculation using google maps API. As google API has hit limits, we will just use show the code to use it.<jupyter_code>new_trip_distance = pd.Series(round(df[np.logical_and(df.Trip_distance == 0,~ np.logical_or(df.Pickup_longitude == 0,df.Dropoff_longitude == 0))]\
.apply(calculate_distance,axis=1),2),name='Trip_distance')
#Updating the new trip distance back to the original dataset
df.update(new_trip_distance)
# Trip distance has been updated
df.head(5).Trip_distance
# Zero Trip distance after update
print("Zero Trip Distance after update: {}".format(df[df.Trip_distance ==0].count()[0]))<jupyter_output>Zero Trip Distance after update: 13193<jupyter_text>After updating the trip distance using Great circle distance, we can still see large number of records are having zero distance.   2. Trip distance which are 0 and it has latitude and longitudeThere are many observations which are still having 0 distance but with valid latitude and longitude. These are due to a small delta between pickup and dropoff. Still the histogram is skewed, so we will take two standard deviations of the data and create a histogram chart and perform log transformation to get a good histogram.<jupyter_code>mean = np.mean(df['Trip_distance'])
sd = np.std(df['Trip_distance'])
df_2sd = df[df['Trip_distance'] <= mean + 2*sd]
df_2sd = df_2sd[df_2sd['Trip_distance'] >= mean - 2*sd]
df_2sd.Trip_distance.plot(kind='hist',bins=50)
plt.title("Histogram of Trip distance")
plt.xlabel("Trip distance")
plt.ylabel("Frequency of Trip distance")

#np.log(df_2sd.Trip_distance+1).plot(kind='hist',bins=50)
sns.distplot(np.log(df_2sd.Trip_distance+1),bins=50)
plt.title("Histogram of log Trip distance")
plt.xlabel("Trip distance")
plt.ylabel("Frequency of Trip distance")<jupyter_output><empty_output><jupyter_text>Summary of Question 2:Below are the list of findings which are noted from above analysis.1. Trip distance has many `0` values. Some are because of the very small distance between pickup and drop location.2. Some observations have `0` trip distance, distance might have not been recorded.3. Histogram and log trip distance has been created.4. Hypothesis in the datastructure is trip distance is actually 0 or it does have an actual trip distance.<jupyter_code>del df_2sd<jupyter_output><empty_output><jupyter_text>Mean and median Trip distance by hour of day  As a next step we will calculate the mean and median trip distance by hour of day and plot charts based on it. To calculate mean/median, we need to convert the pickup date time to day, hour, and dayofweek. Using this we can group by operations and plot the graphs.<jupyter_code>#convert pickup to date
df['lpep_pickup_datetime'] = pd.to_datetime(df.lpep_pickup_datetime)
#Day of the month
df['day'] = df['lpep_pickup_datetime'].dt.day
#Hour of the day
df['hour'] = df['lpep_pickup_datetime'].dt.hour
#Day of the week
df['dayofweek'] = df['lpep_pickup_datetime'].dt.dayofweek
df.head()
dayhour_distance = pd.DataFrame(df.groupby(['dayofweek','hour']).Trip_distance.mean()).reset_index()<jupyter_output><empty_output><jupyter_text>Below plot is straight forward, it take the input data and groups by hour. Mean/ median trip distance is calculated and line chart is plotted.<jupyter_code>pd.DataFrame(df.groupby(['hour'])[['Trip_distance']].agg(['mean','median'])).reset_index().plot(kind='line',x='hour',y='Trip_distance')
plt.title("Hour vs Mean/Median Trip distance")
plt.xlabel("Hour")
plt.ylabel("Average Trip distance")
# Below shown is a factorplot, which shows hourly ranges of trip distance.

plt.warnings.filterwarnings('ignore')
sns.factorplot(x="hour", y="Trip_distance", data=dayhour_distance,size=7);
plt.title("Hour vs Mean Trip distance with Low/High")
plt.xlabel("Hour")
plt.ylabel("Trip distance")
# Below barplot overlays line plot hourly average on the dayofweek hourly average .

plt.figure(figsize=(15,7))
plt.warnings.filterwarnings('ignore')
pd.DataFrame(df.groupby(['hour']).Trip_distance.mean()).reset_index().plot(kind='line',x='hour',y='Trip_distance')
sns.barplot(x='hour',y='Trip_distance',data=dayhour_distance)
plt.title("Hour vs Mean Trip distance")
plt.xlabel("Hour")
plt.ylabel("Mean Trip distance")
# Below swarm/boxplot clearly shows the ranges per dayofweek and the trip distance.
plt.figure(figsize=(9,9))
plt.warnings.filterwarnings('ignore')
sns.swarmplot(x='dayofweek',y='Trip_distance',data=dayhour_distance)
sns.boxplot(x='dayofweek',y='Trip_distance',data=dayhour_distance,boxprops={'facecolor':'None'})
plt.title("Dayofweek vs Trip Distance")
plt.xlabel("Trip distance")
plt.ylabel("Frequency of Trip distance")
# Below swarm/boxplot clearly shows the ranges per day of month and the trip distance.

moth_dayhour_distance = pd.DataFrame(df.groupby(['day','hour']).Trip_distance.mean()).reset_index()

plt.figure(figsize=(12,12))
plt.warnings.filterwarnings('ignore')
sns.swarmplot(x='day',y='Trip_distance',data=moth_dayhour_distance)
sns.boxplot(x='day',y='Trip_distance',data=moth_dayhour_distance,boxprops={'facecolor':'None'})
plt.title("Dayofmonth vs Trip Distance")
plt.xlabel("Trip distance")
plt.ylabel("Frequency of Trip distance")<jupyter_output><empty_output><jupyter_text>Summary of Question 3.1:1. Some outliers/wrong data has been removed.2. Created dayofweek vs trip distance boxplot/swarm chart3. Created dayofmonth boxplot/swarm chart.4. Mean/Median of the trip distance were grouped by hour of day.  NYC Airport taxi analysis In this section, we will try to find out the airports and the green taxi activity around this area. By analyzing it, we can try to find any interesting characters on these trips.Below are the differnt NYC Airport Coordinates which I got from google.1. JFK: 40.6413° N, 73.7781° W 2. LaGuardia: 40.7769° N, 73.8740° W3. Newark Liberty: 40.6895° N, 74.1745° WWe also need to assign what is mean by near to the airport. A person might walk for some distance from airport to avail a taxi. So we will assign a value in miles around the airport and consider as the spots nearby airport.Before performing these analysis, we need to correct and filter out the Total amount. Because some of them are missing and has negative values.<jupyter_code>#Below are the records which has total amount of 0 and negative amount. It is still not clear with the dataset about negative values. 
#For now, we will not consider these values while calculating mean fare.

df[df.Total_amount <=0][['Trip_distance','Store_and_fwd_flag','Fare_amount', 'Extra', 'MTA_tax',
       'Tip_amount', 'Tolls_amount', 'improvement_surcharge',
       'Total_amount']].head()
# We will consider that any one mile near to the airport is considered as airport dropoff
nearest_area = 1
airport_locations = {'jfk':(40.6413,-73.7781), 'newark':(40.6895,-74.1745),
                   'lag': (40.7769,-73.8740)}
airport_analysis = df[['Pickup_latitude','Pickup_longitude','Dropoff_latitude','Dropoff_longitude']]
def pickup_distance(x, air_dict):
    """Function which calculates the distance from pickup location to all the other locations in the dataset."""
    result =[]
    
    p = 0.017453292519943295
    d = 12742
    m = 0.621371     
    
    for key,val in air_dict.items():
        
        pickup = d * asin(sqrt(0.5 - cos((x.Pickup_latitude-val[0])*p)/2 + cos(val[0]*p)*cos(x.Pickup_latitude * p) * (1-cos((x.Pickup_longitude-val[1])*p)) / 2)) * m
        dropoff = d * asin(sqrt(0.5 - cos((x.Dropoff_latitude-val[0])*p)/2 + cos(val[0]*p)*cos(x.Dropoff_latitude*p) * (1-cos((x.Dropoff_longitude-val[1])*p)) / 2)) * m

        result.append((pickup,dropoff))
    return result

# Airport distance calculation
airport_analysis['distance'] = airport_analysis.apply(pickup_distance,air_dict =airport_locations,axis=1)

df['jfk_pickup_distance'] = airport_analysis['distance'].apply(lambda x: x[0][0])
df['jfk_dropoff_distance'] = airport_analysis['distance'].apply(lambda x: x[0][1])

df['newark_pickup_distance'] = airport_analysis['distance'].apply(lambda x: x[1][0])
df['newark_dropoff_distance'] = airport_analysis['distance'].apply(lambda x: x[1][1])

df['lag_pickup_distance'] = airport_analysis['distance'].apply(lambda x: x[2][0])
df['lag_dropoff_distance'] = airport_analysis['distance'].apply(lambda x: x[2][1])


del airport_analysis
def airport_analysis(airport):
    
    airport_pickup = airport+'_pickup_distance'
    airport_dropoff = airport+'_dropoff_distance'
    
    airport_df = df[np.logical_or(df[airport_pickup] <nearest_area,df[airport_dropoff] <nearest_area)]
    
    total_count = airport_df.count()[0]
    pickup_count = airport_df[airport_df[airport_pickup] <nearest_area].count()[0]
    dropoff_count = airport_df[airport_df[airport_dropoff] <nearest_area].count()[0]
    
    #Get only the Fare above 0
    fare_mean = round(airport_df[airport_df>0].Total_amount.mean(),2)
    travel_distance_mean = round(airport_df.Trip_distance.mean(),2)
    airport_type = airport
    
    print("\n***Airport "+airport+'***'+'\n')
    print("Total number of pickup/dropoff rides: {}".format(total_count))
    print("Total number of pickup rides: {}".format(pickup_count))
    print("Total number of dropoff rides: {}".format(dropoff_count))
    print("Average fare near airport: {}".format(fare_mean))
    print("Average travel distance near airport: {}".format(travel_distance_mean))
    
    temp_df = pd.DataFrame({'airport_type':airport,'total_count':[total_count],'pickup_count':[pickup_count],'dropoff_count':[dropoff_count],'fare_mean':[fare_mean],'travel_distance_mean':[travel_distance_mean]})    
    return temp_df
analysis_df = airport_analysis('newark')

analysis_df = analysis_df.append(airport_analysis('jfk'),ignore_index=True)
analysis_df = analysis_df.append(airport_analysis('lag'),ignore_index=True)
f, (ax1,ax2,ax3) = plt.subplots(3,sharex=True)
analysis_df.plot(kind='bar',x ='airport_type',y='total_count',ax=ax1)
ax1.set_title("Airport Total Rides")
ax1.set_ylabel("Rides Count")

analysis_df.plot(kind='bar',x ='airport_type',y='travel_distance_mean',ax=ax2)
ax2.set_title("Airport Average Distance")
ax2.set_ylabel("Average Distance")

analysis_df.plot(kind='bar',x ='airport_type',y='fare_mean',ax=ax3)
ax3.set_title("Airport Average Fare")
ax3.set_ylabel("Average Fare")

f.subplots_adjust(hspace=0.3)
f.set_size_inches(9,9)<jupyter_output><empty_output><jupyter_text>Lets calculate the mean fare in airport pickup/dropoff location and non-airport locations. We will get the data from all NYC airports and calculate the mean of it.<jupyter_code># Non Airport Mean Fare
non_airport_mean_fare = pd.concat([df[~np.logical_or(df.jfk_pickup_distance <1,df.jfk_dropoff_distance <1)],
          df[~np.logical_or(df.newark_pickup_distance <1,df.newark_dropoff_distance <1)],
          df[~np.logical_or(df.lag_pickup_distance <1,df.lag_dropoff_distance <1)]]).Total_amount.mean()

# Airport Mean Fare
airport_mean_fare = pd.concat([df[np.logical_or(df.jfk_pickup_distance <1,df.jfk_dropoff_distance <1)],
          df[np.logical_or(df.newark_pickup_distance <1,df.newark_dropoff_distance <1)],
          df[np.logical_or(df.lag_pickup_distance <1,df.lag_dropoff_distance <1)]]).Total_amount.mean()
pd.DataFrame({'Non_airport_Fare':[non_airport_mean_fare],'Airport_Fare':[airport_mean_fare]}).plot(kind='bar')
plt.title("Airport vs Non Airport")
plt.xlabel("Pickup/Dropoff Location")
plt.ylabel("Average Fare")<jupyter_output><empty_output><jupyter_text>Summary of Question 3.2:1. Total average fare amount near the airport is higher than non-airport locations. So if a driver needs to earn more, he needs to be around airports.2. Newark airport has very less riders. But the average fare and distance is high from that airport. So to earn more between the airports, the taxi driver should be around Newark airport.3. JFK airport is having average riders, fare and distance compared to other airports.4. Laguardia has many riders, but the distance and fare is less. So if the driver needs more frequent rides, then this airport should be selected.<jupyter_code>del analysis_df<jupyter_output><empty_output><jupyter_text>Build a derived variable and modelCreated in separate file Model.ipynb  Realtime pickup rideshare searchIn this section, we will create an algorithm to find the nearest pickup requests from the point which we are requesting. So by showing the realtime pickup requests, we can promote ride sharing and create a business out of it.As a first step, we will filter the dataframe to the required columns, so we can fit in memory and see the data easily in the notebook.<jupyter_code>df_geo = df[['Trip_distance','lpep_pickup_datetime', 'Pickup_longitude',
       'Pickup_latitude', 'Dropoff_longitude', 'Dropoff_latitude',
       'jfk_pickup_distance']]<jupyter_output><empty_output><jupyter_text>We are going to use Geohash technique to find out the nearest pickup points. For that we need to convert latitude/longitude to a geohasing point so we can find out the nearest points with less complexity. Geohash spilts the locations into hash string from binary values. For every latitude and longitude position, we can create a hash string. Nearest string will have similar values like that string. More can be read in [wikipage](https://en.wikipedia.org/wiki/Geohash).<jupyter_code>#Calculating geohash for all the available points which are available in the dataset

if os.path.exists('./generated_files/hash.pkl'):
    df_geo['geo_hash'] = pd.read_pickle('./generated_files/hash.pkl','gzip')
else:
    df_geo['geo_hash'] = df_geo.apply(lambda x: geohash.encode(x.Pickup_latitude,x.Pickup_longitude),axis=1)
def taxi_pickup_time():
    """Function which takes current time and september month with 2015 year.
    It also provides, the timestamp after one hour. It is the threshold waiting 
    time for the ridesharing pickup"""
    
    now = datetime.datetime.now()
    request_time = '2015-09-'+str(now.day)+' '+str(now.hour)+':'+str(now.minute)+':'+str(now.second)
    current_time = datetime.datetime.strptime(request_time, '%Y-%m-%d %H:%M:%S')
    #Threshold time
    threshold_time = current_time+timedelta(hours=1)

    return current_time,threshold_time
def nearest_pickups(lat,log):
    """Main function which calculates the nearest point by geohash technique. 
    1. It calculates the hash ofthe lat/log.
    2. Filters out the requested ride for next hour.
    3. Picks minimum of 5 pickup points nearby.
    4. Calculates the distance from pickup point to other riders using great circle distance.
    5. Filters out the output data and provides nearest 30 riders."""
    
    s = geohash.encode(lat,log)
    nearby_cabs=[]
    #geo_hash_cd =[]
    geo_index =[]
    current_time,threshold_time = taxi_pickup_time()
    current_geo_df = df_geo[np.logical_and(df_geo['lpep_pickup_datetime'] > current_time, df_geo['lpep_pickup_datetime'] < threshold_time)]
    for i in range(len(s),0,-1):
        if len(nearby_cabs) ==0 | len(nearby_cabs) <5:
            for origin in current_geo_df['geo_hash'].iteritems():
                if s[:i] in origin[1]:
                    nearby_cabs.append(geo.great_circle(geohash.decode(s),geohash.decode(origin[1])).miles)
                    #geo_hash_cd.append(geohash.decode(s))
                    geo_index.append(origin[0])
    if df_geo.ix[geo_index].count()[0] > 30:
        return df_geo.ix[geo_index].sort_values(['geo_hash']).head(30)
    else:
        return df_geo.ix[geo_index]
def add_points(x):
    """Adds the points to Folium map and returns the pooint details."""
    folium.Marker([x['Pickup_latitude'],x['Pickup_longitude']]).add_to(m1)
    return x<jupyter_output><empty_output><jupyter_text>Below chart shows the NYC maps which is using Folium. This is just an exploratory map, to provide the latitude and longitude of the pickup point. In actual usecase, point location is generated using mobile or any location deduction device.Click on any point near the NYC area to show the lat/long point. This needs to be manually inputed in the next chart.<jupyter_code># This point is NYC. Change if you need a different city and have different dataset
#ELSE DO NOT CHANGE below code.

select_point = (40.7128,-74.0060)
m = folium.Map(location=[select_point[0],select_point[1]])
m.add_child(folium.LatLngPopup())
m<jupyter_output><empty_output><jupyter_text>From the above chart you need to manually input the lat/long in the below area. In actual product, this can be inputed automatically to next field.<jupyter_code># CHANGE THE BELOW POINT using above map. This will be the pickup point
s = (40.6473,-73.7883)
m1 = folium.Map(location=[select_point[0],select_point[1]])
# Pickup point marked in RED color

folium.Marker([s[0],s[1]],icon=folium.Icon(color='red'),popup='Pickup Point').add_to(m1)
#Ride sharing rides is marked in Blue color
nearby_rides = nearest_pickups(s[0],s[1]).apply(add_points,axis=1)

print("Current Pickup time: {}".format(taxi_pickup_time()[0]))
print("Pickup rides within next one hour: {}".format(taxi_pickup_time()[1]))

m1<jupyter_output>Current Pickup time: 2015-09-28 19:00:32
Pickup rides within next one hour: 2015-09-28 20:00:32
{'fraction_non_alphanumeric': 0.11000345244260314, 'fraction_numerical': 0.01855687899188676, 'mean_word_length': 5.628432494279176, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 80, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '22875', 'license': 'MIT', 'n_tokens_mistral': 7714, 'n_tokens_neox': 7254, 'n_words': 2260, 'path': 'Final_analysis.ipynb', 'repo_name': 'bvshyam/NYC_Taxi_Challenge', 'size': '23186'}