<jupyter_start><jupyter_text>An Advanced Featuretools Approach with Custom PrimitivesThe following tutorial illustrates an advanced featuretools model for the NYC Taxi Trip Duration competition on Kaggle using our custom primitive API. You will need to download the following five files into a `data` folder in this repository.Step 1: Download raw data test.csv and train.csv<jupyter_code>import featuretools as ft
import pandas as pd
import numpy as np
import taxi_utils

import warnings
warnings.simplefilter('ignore')

ft.__version__<jupyter_output><empty_output><jupyter_text>Step 2: Prepare the Data<jupyter_code>TRAIN_DIR = "data/train.csv"
TEST_DIR = "data/test.csv"

data_train, data_test = taxi_utils.read_data(TRAIN_DIR, TEST_DIR)<jupyter_output><empty_output><jupyter_text>The `fastest_routes` dataset has information on the shortest path between two coordinates in NYC. We can merge it in with our dataset here and then merge together our test and train datasets after marking them as such.<jupyter_code># Make a train/test column
data_train['test_data'] = False
data_test['test_data'] = True

# Combine the data and convert some strings
data = pd.concat([data_train, data_test], sort=True)
data.head(5)<jupyter_output><empty_output><jupyter_text>Step 3: Custom PrimitivesThe custom primitive API is new to Featuretools version 0.1.9. Our workflow will be as follows:1. Make a new LatLong class which is a tuple of a latitude and longitude2. Define some functions for LatLong3. Make new primitives from those functionsFor the first step, we'll make new columns for the pickup and dropoff locations which are tuples.<jupyter_code>data["pickup_latlong"] = data[['pickup_latitude', 'pickup_longitude']].apply(tuple, axis=1)
data["dropoff_latlong"] = data[['dropoff_latitude', 'dropoff_longitude']].apply(tuple, axis=1)
data = data.drop(["pickup_latitude", "pickup_longitude", "dropoff_latitude", "dropoff_longitude"], axis = 1)<jupyter_output><empty_output><jupyter_text>We can define the `pickup_latlong` to be a `LatLong` type. This is a way to tell DFS that certain primitives are only important in one direction, from the beginning of a trip to the end.<jupyter_code>from woodwork.logical_types import Ordinal

trip_logical_types = {
    'passenger_count': Ordinal(order=list(range(0, 10))), 
    'vendor_id': 'Categorical',
    'pickup_latlong': 'LatLong',
    'dropoff_latlong': 'LatLong',
}

es = ft.EntitySet("taxi")

es.add_dataframe(dataframe_name="trips",
                 dataframe=data,
                 index="id",
                 time_index='pickup_datetime',
                 logical_types=trip_logical_types)

es.normalize_dataframe(base_dataframe_name="trips",
                       new_dataframe_name="vendors",
                       index="vendor_id")

es.normalize_dataframe(base_dataframe_name="trips",
                       new_dataframe_name="passenger_cnt",
                       index="passenger_count")

cutoff_time = es['trips'][['id', 'pickup_datetime']]<jupyter_output><empty_output><jupyter_text>Visualize EntitySet<jupyter_code>es.plot()<jupyter_output><empty_output><jupyter_text>Next, we can create primitives for our new `pickup_latlong` and `dropoff_latlong` which are of type `LatLong`. The distance between two `LatLong` types is most accurately represented by the Haversine distance: the nearest length along a sphere. We can also define the Cityblock distance (also called the taxicab metric), which takes into account that we can't move diagonally through buildings in Manhattan.<jupyter_code>from featuretools.primitives import make_agg_primitive, make_trans_primitive

def haversine(latlong1, latlong2):
    lat_1s = np.array([x[0] for x in latlong1])
    lon_1s = np.array([x[1] for x in latlong1])
    lat_2s = np.array([x[0] for x in latlong2])
    lon_2s = np.array([x[1] for x in latlong2])
    lon1, lat1, lon2, lat2 = map(np.radians, [lon_1s, lat_1s, lon_2s, lat_2s])
    dlon = lon2 - lon1
    dlat = lat2 - lat1
    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2
    km = 6367 * 2 * np.arcsin(np.sqrt(a))
    return km

def cityblock(latlong1, latlong2):
    lon_dis = haversine(latlong1, latlong2)
    lat_dist = haversine(latlong1, latlong2)
    return lon_dis + lat_dist

def bearing(latlong1, latlong2):
    lat1 = np.array([x[0] for x in latlong1])
    lon1 = np.array([x[1] for x in latlong1])
    lat2 = np.array([x[0] for x in latlong2])
    lon2 = np.array([x[1] for x in latlong2])
    delta_lon = np.radians(lon2 - lon1)
    lon1, lat1, lon2, lat2 = map(np.radians, [lon1, lat1, lon2, lat2])
    x = np.cos(lat2) * np.sin(delta_lon)
    y = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(delta_lon)
    return np.degrees(np.arctan2(x, y))<jupyter_output><empty_output><jupyter_text>We can also make some primitives directly from `LatLong` types. In particular, we'll almost certainly want to extract the Latitude, the Longitude and the central point between two such coordinates. Now lets make primitives from those functions!<jupyter_code>from woodwork.column_schema import ColumnSchema
from woodwork.logical_types import Double, LatLong

Bearing = make_trans_primitive(function=bearing,
                               input_types=[ColumnSchema(logical_type=LatLong), ColumnSchema(logical_type=LatLong)],
                               commutative=True,
                               return_type=ColumnSchema(logical_type=Double, semantic_tags={'numeric'}))

Cityblock = make_trans_primitive(function=cityblock,
                                 input_types=[ColumnSchema(logical_type=LatLong), ColumnSchema(logical_type=LatLong)],
                                 commutative=True,
                                 return_type=ColumnSchema(logical_type=Double, semantic_tags={'numeric'}))
from woodwork.logical_types import Datetime, Boolean

def is_rush_hour(datetime):
    hour = pd.DatetimeIndex(datetime).hour
    return (hour >= 7) & (hour <= 11)

def is_noon_hour(datetime):
    hour = pd.DatetimeIndex(datetime).hour
    return (hour >= 11) & (hour <= 13)

def is_night_hour(datetime):
    hour = pd.DatetimeIndex(datetime).hour
    return (hour >= 18) & (hour <= 23)

RushHour = make_trans_primitive(function=is_rush_hour,
                                input_types=[ColumnSchema(logical_type=Datetime)],
                                return_type = ColumnSchema(logical_type=Boolean))
    
NoonHour = make_trans_primitive(function=is_noon_hour,
                                input_types=[ColumnSchema(logical_type=Datetime)],
                                return_type = ColumnSchema(logical_type=Boolean))
        
NightHour = make_trans_primitive(function=is_night_hour,
                                 input_types=[ColumnSchema(logical_type=Datetime)],
                                 return_type = ColumnSchema(logical_type=Boolean))<jupyter_output><empty_output><jupyter_text>We have grouped together some times of day with `RushHour`, `NoonHour` and `NightHour`.<jupyter_code>from featuretools.primitives import TransformPrimitive
from woodwork.logical_types import Boolean, LatLong

class GeoBox(TransformPrimitive):
    name = "GeoBox"
    input_types=[ColumnSchema(logical_type=LatLong)]
    return_type=ColumnSchema(logical_type=Boolean)
    
    def __init__(self, bottomleft, topright):
        self.bottomleft = bottomleft
        self.topright = topright
    
    def get_function(self):
        def geobox(latlong, bottomleft=self.bottomleft, topright=self.topright):
            lat = np.array([x[0] for x in latlong])
            lon = np.array([x[1] for x in latlong])
            boxlats = [bottomleft[0],topright[0]]
            boxlongs = [bottomleft[1], topright[1]]
            output = []
            for i, name in enumerate(lat):
                if (min(boxlats) <= lat[i] and lat[i] <= max(boxlats) and 
                        min(boxlongs) <= lon[i] and lon[i] <= max(boxlongs)):
                    output.append(True)
                else: 
                    output.append(False)
            return output
        
        return geobox

    def generate_name(self, base_feature_names):
        return u"GEOBOX({}, {}, {})".format(base_feature_names[0],
                                            str(self.bottomleft),
                                            str(self.topright))<jupyter_output><empty_output><jupyter_text>The feature `Geobox` finds if a latlong is in a rectangle defined by two coordinate pairs.  Step 4: Using custom primitives in DFS Let's see what features are created if we run DFS now:<jupyter_code>agg_primitives = []
trans_primitives = [Bearing, Cityblock,
                    GeoBox(bottomleft=(40.62, -73.85), topright=(40.70, -73.75)),
                    GeoBox(bottomleft=(40.70, -73.97), topright=(40.77, -73.9)),
                    RushHour,
                    NoonHour,
                    NightHour]

# calculate feature_matrix using deep feature synthesis
features = ft.dfs(entityset=es,
                  target_dataframe_name="trips",
                  trans_primitives=trans_primitives,
                  agg_primitives=agg_primitives,
                  drop_contains=['trips.test_data'],
                  verbose=True,
                  cutoff_time=cutoff_time,
                  approximate='36d',
                  max_depth=3,
                  max_features=40,
                  features_only=True)
features<jupyter_output>Built 20 features<jupyter_text>Our new features were applied exactly where they should be! The distances were only calcuated between the pickup_latlong and the dropoff_latlong, while primitives like `LATITUDE` were calculated on both latlong columns.For this dataset, we have approximately 2 million distinct cutoff times, times at which DFS will have to recalculate aggregation primitives. Calculating those features at specific times every hour and using that number instead will save computation time and perhaps not lose too much information. The `approximate` parameter in DFS lets you specify a window size to use when approximating.With the features from before, we calculate the whole feature matrix:<jupyter_code>agg_primitives = ['Sum', 'Mean', 'Median', 'Std', 'Count', 'Min', 'Max', 'Num_Unique', 'Skew']
trans_primitives = [Bearing, Cityblock,
                    GeoBox(bottomleft=(40.62, -73.85), topright=(40.70, -73.75)),
                    GeoBox(bottomleft=(40.70, -73.97), topright=(40.77, -73.9)),
                    RushHour,
                    NoonHour,
                    NightHour,
                    'Day', 'Hour', 'Minute', 'Month', 'Weekday', 'Week', 'Is_weekend']

# this allows us to create features that are conditioned on a second value before we calculate.
es.add_interesting_values()

# calculate feature_matrix using deep feature synthesis
feature_matrix, features = ft.dfs(entityset=es,
                                  target_dataframe_name="trips",
                                  trans_primitives=trans_primitives,
                                  agg_primitives=agg_primitives,
                                  drop_contains=['trips.test_data'],
                                  verbose=True,
                                  cutoff_time=cutoff_time,
                                  approximate='36d',
                                  max_depth=4)

feature_matrix.head()<jupyter_output>Built 99 features
Elapsed: 03:51 | Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████<jupyter_text>Step 5: Build the Model As before, we need to retrieve our labels for the train dataset, so we should merge our current feature matrix with the original dataset. We use the `log` of the trip duration since that measure is better at distinguishing distances within the cityJust like in the previous notebook, we will perform the modeling with [EvalML](https://evalml.alteryx.com/en/stable/)<jupyter_code># separates the whole feature matrix into train data feature matrix, train data labels, and test data feature matrix 
X_train, labels, X_test = taxi_utils.get_train_test_fm(feature_matrix)
labels = np.log(labels.values + 1)
import evalml
from evalml import AutoMLSearch

automl = AutoMLSearch(X_train=X_train,
                      y_train=labels,
                      problem_type="regression",
                      objective="root mean squared error")
automl.search()
pipeline = automl.best_pipeline
pipeline.fit(X_train, labels)<jupyter_output><empty_output><jupyter_text>Notice that by combining EvalML and the custom primitives, we were able to create a model with an even lower RMSE. This is the power of feature engineering combined with auto modeling<jupyter_code>submission = pipeline.predict(X_test)
submission.head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09929022697137509, 'fraction_numerical': 0.015209422041962406, 'mean_word_length': 3.164339071126989, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '387167', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4027, 'n_tokens_neox': 3774, 'n_words': 1021, 'path': 'predict-taxi-trip-duration/NYC Taxi 4 - Custom Featuretools Primitives.ipynb', 'repo_name': 'Featuretools/predict_next_purchase', 'size': '12821'}