<jupyter_start><jupyter_text>Data exploration and preparation for Machine Learning  Hands on case studyExplore NYC Trips dataset to- get knowledge of relationships between variables  - create better features- remove outliers  - get a robuster modelusing  - python in jupyter  - pandas, matplotlib and seaborn modules  Useful shortcuts in Jupyter  - run current cell     - `shift + Enter`  - go into cell to modify     - `Enter` (the frame around cell becomes **green**)  - deselect cell    - `Esc` (the frame around cell becomes **blue**)  - new cell **above** current cell     - `Esc` to deselect followed by `A`  - new cell **below** current cell    - `Esc` to deselect followed by `B`  Introduce dataset - New York City Taxi Trip Duration- challenge on kaggle- goal to predict the `trip_duration` based on few features:  - `vendor_id` (provider of the trip, such as Uber, Lyft, etc.)  - `pickup_datetime`  - `passenger_count`  - `pickup_longitude`  - `pickup_latitude`  - `dropoff_longitude`  - `dropoff_latitude`  Let's get started Downloading the dataset- create an account at https://www.kaggle.com/- go to https://www.kaggle.com/hobilek/nyc-trips-amld and fork it- you can also just download the dataset and work locally on your jupyter<jupyter_code>import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, FormatStrFormatter
import seaborn as sns
# increase the font size of the plots
FONT_SIZE = 14
mpl.rcParams['xtick.labelsize'] = FONT_SIZE 
mpl.rcParams['ytick.labelsize'] = FONT_SIZE
mpl.rcParams['legend.fontsize'] = FONT_SIZE
mpl.rcParams['axes.labelsize'] = FONT_SIZE
mpl.rcParams['figure.figsize'] = (10, 10)
def precision(x, precision):
    """Round a number or array to a given precision
    :Example:

    precision(np.array([3.71, 4.59]), 0.02)
    >> array([3.72, 4.6 ])
    
    precision(np.array([154, 396]), 10)
    >> array([150, 400 ])
    """
    return (np.round(x / precision) * precision).astype(type(precision))

def load_dataset(file_path):
    df = pd.read_csv(file_path, index_col='id', parse_dates=['pickup_datetime'])
    df = df.head(100000)
    # convert the `trip_duration` from seconds to minutes for convenience
    df['trip_duration_minutes'] = df['trip_duration'] / 60.0
    df.drop(['dropoff_datetime', 'store_and_fwd_flag', 'trip_duration'], axis=1, inplace=True)
    return df

def plot_boxplot(df, n_rows, n_cols, figsize=(18, 24)):
    numeric_columns = df.select_dtypes(include=np.number).columns
    fig, axn = plt.subplots(n_rows, n_cols, figsize=figsize)
    for col, ax in zip(numeric_columns, axn.flatten()):
        sns.boxplot(df[col], orient='v', ax=ax)
file_path = '../input/train.csv'

df = load_dataset(file_path)

df.head()<jupyter_output><empty_output><jupyter_text>TASK 1 - Exploring column valuesHow are the values in each column distributed?- Which columns include missing values?- Which columns include only few different values?- Did you find any outliers? Filter them out.- *(optional)* Based on your information, try to sketch the distribution. The function `describe` shows minimum and maximum of each column, as well as mean, quantiles and other statistics.It is similar to R's `summary` function.<jupyter_code>df.describe(percentiles=[0.1, 0.25, 0.75, 0.9])<jupyter_output><empty_output><jupyter_text>We could also visualize the previous summary using boxplots.<jupyter_code>plot_boxplot(df, n_rows=4, n_cols=2)<jupyter_output><empty_output><jupyter_text>TASK 2 - Outliers (part 1)- Eliminate outliers from the column `trip_duration`.<jupyter_code>fig = plt.figure()
ax = fig.gca()
ax.hist(df['trip_duration_minutes'], bins=50)
ax.set_xlabel('trip duration in min')
ax.set_ylabel('number of trips')
ax.grid()<jupyter_output><empty_output><jupyter_text>Vast majority in the first bin. This means there are few tours with a very high trip duration. What now?**Log transform on y-axis.**<jupyter_code>fig = plt.figure()
ax = fig.gca()
ax.hist(df['trip_duration_minutes'], bins=50)
ax.set_xlabel('trip duration in min')
ax.set_ylabel('number of trips')
ax.set_yscale('log')
ax.grid()
df = df[df['trip_duration_minutes'] < 200]<jupyter_output><empty_output><jupyter_text>Continue exploring the distribution of `trip_duration` further and keep filtering out values until you are happy with the result.- Which values did you filter out and why?  - Don't forget to look not only at big values but also very small values.- Show the distribution after filtering outliers.  TASK 3 - Outliers (part 2)Filter out outliers in location columns - `pickup_longitude`- `pickup_latitude`- `dropoff_longitude`- `dropoff_latitude`Visualize the pickup and dropoff locations in a scatter plot.<jupyter_code>fig = plt.figure(figsize=(16, 12))
ax = fig.gca()
s = ax.scatter(df['pickup_longitude'], df['pickup_latitude'], marker='.', s=1, 
               c=df['trip_duration_minutes'], cmap='RdYlGn_r')
ax.set_title('pickup locations', fontsize=20)
ax.set_xlabel('longitude')
ax.set_ylabel('latitude')
ax.xaxis.set_major_locator(MultipleLocator(0.2))
ax.yaxis.set_major_locator(MultipleLocator(0.2))
ax.grid()
fig.colorbar(s);
df = df[df['pickup_latitude'].between(40.55, 40.95) & \
        df['pickup_longitude'].between(-74.1, -73.7)]<jupyter_output><empty_output><jupyter_text>Is filtering on these values sufficient? Once you cleaned the dataset based on pickup location, do the same for dropoff location.  TASK 4 - locations with long tripsAt which pickup / dropoff locations can we find the longest trips? Does it make sense?<jupyter_code>fig = plt.figure(figsize=(16, 12))
ax = fig.gca()
df.plot.hexbin(x='pickup_longitude', y='pickup_latitude', 
               C='trip_duration_minutes', gridsize=20, cmap='Blues', ax=ax);
pickup = df.groupby([precision(df['pickup_longitude'], 0.02), 
                     precision(df['pickup_latitude'], 0.02)]
                   )['trip_duration_minutes'].mean().reset_index().round(2)
pick = pickup.pivot('pickup_latitude','pickup_longitude','trip_duration_minutes')
fig = plt.figure(figsize=(16, 12))
ax = fig.gca()
sns.heatmap(pick, cmap='Greens', ax=ax, annot=True, fmt=".0f")
ax.set_title('mean trip duration [minutes]', fontsize=20)
ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>Analyse mean trip duration based on dropoff location.You can also visualize other variables, such as `vendor_id` or `passenger_count` in relation with locations if you have time.  TASK 5 - Trip length vs time- When during the week do the customers make the shortest trips?- When do they make the longest trips?<jupyter_code>df['weekday'] = df['pickup_datetime'].dt.weekday # 0 is Monday
df['hour'] = df['pickup_datetime'].dt.hour<jupyter_output><empty_output><jupyter_text>Convert each week and hour into a number 0-167 representing number of hours since Monday midnight.<jupyter_code>df['week_hours'] = df['weekday'] * 24 + df['hour']
week_hours = df.groupby('week_hours')['trip_duration_minutes'].mean()
fig = plt.figure(figsize=(20,6))
ax = fig.gca()
ax.plot(week_hours)
ax.set_xlabel('hours since Monday midnight', fontsize=18)
ax.set_ylabel('mean trip duration [minutes]', fontsize=18)
ax.xaxis.set_major_locator(MultipleLocator(24))
ax.xaxis.set_minor_locator(MultipleLocator(6))
ax.grid(which='minor', linestyle=':', linewidth=0.5)
ax.grid(which='major', linestyle='-', linewidth=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1250337929170046, 'fraction_numerical': 0.018788861854555285, 'mean_word_length': 5.286321155480034, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '260176', 'license': 'MIT', 'n_tokens_mistral': 2594, 'n_tokens_neox': 2410, 'n_words': 687, 'path': 'part_3/NYC_Trips_AMLD.ipynb', 'repo_name': 'terezaif/workshops_data_exploration', 'size': '7398'}