<jupyter_start><jupyter_text>Part 2.1 - Feature Engineering for XGBoost ModelIn this notebook, we will be loading the original train and test dataset so that we can engineer features based on the statistical properties of the flux and observation times.This notebook is broken down into four parts1. Compute the flux skewness for each `object_id`2. Compute other statistical features3. Merge features from Parts 1 and 24. Merge bottleneck features<jupyter_code>import cudf as gd
import pandas as pd
import numpy as np
import math
import seaborn as sns
from termcolor import colored
from cudf_agg import cudf_groupby_aggs, compute_skew
import matplotlib.pyplot as plt
import warnings

from numba import cuda,jit,float32

from utils import scatter, groupby_aggs
warnings.filterwarnings("ignore")
sns.set()
print(gd.__version__)
PATH = '/raid/cjnolet/plasticc_data/'
ts_cols = ['object_id', 'mjd', 'passband', 'flux', 'flux_err', 'detected']
ts_dtypes = ['int32', 'float32', 'int32', 'float32','float32','int32']<jupyter_output><empty_output><jupyter_text>Part 1 - Compute Flux Skewness for each `object_id`Load the training and test datasets back into cuDF DataFrames.<jupyter_code>train_gd = gd.read_csv('%s/training_set.csv'%PATH, names=ts_cols,dtype=ts_dtypes,skiprows=1)
test_gd = gd.read_csv('%s/test_set_sample.csv'%PATH, names=ts_cols,dtype=ts_dtypes,skiprows=1)
test_gd.head().to_pandas()<jupyter_output><empty_output><jupyter_text>Let's visualize the flux for different passbands at different times for a few different objects in our data set.<jupyter_code>for idx, oid in zip(range(1), train_gd.object_id.unique()[:1]):
    train = train_gd.to_pandas()
    mask = train.object_id== oid

    scatter(train.loc[mask,'mjd'].values,
            train.loc[mask,'flux'].values,
            values=train.loc[mask,'passband'].values,
            xlabel='time',
            ylabel='flux',
            title='Object %d class 42'%oid)
    plt.show()<jupyter_output><empty_output><jupyter_text>Since we are going to be computing the skewness of the `flux` for each `object_id`, we can safely drop all the other columns.<jupyter_code>test_gd = test_gd[['object_id','flux']]
train_gd = train_gd[['object_id','flux']]<jupyter_output><empty_output><jupyter_text>The computation of the flux skewness is done using the `groupby() - apply_grouped()` techinque we used in the RNN feature engineering stage.<jupyter_code>test_gd = cudf_groupby_aggs(test_gd,group_id_col='object_id',aggs={'flux': ["skew"]})
train_gd = cudf_groupby_aggs(train_gd,group_id_col='object_id',aggs={'flux':["skew"]})
test_flux_skew_gd = test_gd.sort_values(by='object_id')
train_flux_skew_gd = train_gd.sort_values(by='object_id')
train_flux_skew_gd.head().to_pandas()<jupyter_output><empty_output><jupyter_text>Part 2 - Compute Statistical Summary Features We computed the flux skew in the previous ETL step and only kept those columns on GPU. Next we will need to load our original train & test CSV files back in so that we can do some additional aggregations.<jupyter_code>train_gd = gd.read_csv('%s/training_set.csv'%PATH, names=ts_cols,dtype=ts_dtypes,skiprows=1)
test_gd = gd.read_csv('%s/test_set_sample.csv'%PATH, names=ts_cols,dtype=ts_dtypes,skiprows=1)<jupyter_output><empty_output><jupyter_text>Read in the training metadata fields<jupyter_code>cols = ['object_id', 'ra', 'decl', 'gal_l', 'gal_b', 'ddf',
       'hostgal_specz', 'hostgal_photoz', 'hostgal_photoz_err', 
       'distmod','mwebv', 'target']

dtypes = ['int32']+['float32']*4+['int32']+['float32']*5+['int32']

train_meta_gd = gd.read_csv('%s/training_set_metadata.csv'%PATH, names=cols, dtype=dtypes, skiprows=1)
train_meta_gd.head().to_pandas()<jupyter_output><empty_output><jupyter_text>And the resulting testing metadata fields, excluding the `target` label field<jupyter_code># # Remove the target field from the cols 
del cols[-1],dtypes[-1]

test_meta_gd = gd.read_csv('%s/test_set_metadata.csv'%PATH, names=cols,dtype=dtypes,skiprows=1)<jupyter_output><empty_output><jupyter_text>The remaining statistical summaries perform more `group_by()` and `apply_grouped()` computations.<jupyter_code>def compute_statistical_features(df,df_meta):
    
    aggs = {
        'passband': ['mean'],  # mean passband
        'detected': ['mean'],  # mean detected
        'mjd':['max','min'],   # min / max time range
    }
    
    agg_df = groupby_aggs(df, aggs)
    
    # If flux uncertanty is low, we get a high ratio squared. If the uncertainty is high, 
    # we will get a low ratio squared.
    df['flux_ratio_sq'] = df['flux'] / df['flux_err']
    df['flux_ratio_sq'] = df['flux_ratio_sq'].applymap(lambda x: math.pow(x,2))
    
    # Multiply flux by the ratio squared
    df['flux_by_flux_ratio_sq'] = df['flux'] * df['flux_ratio_sq']
    
    aggs2 = {
        'flux_ratio_sq':['sum'],            # Sum the sq flux ratios
        'flux_by_flux_ratio_sq':['sum'],    # Sum the flux * sq flux ratios
        'flux': ['min', 'max', 'mean'],     # Summary stats for flux
        'flux_err': ['min', 'max', 'mean'], # Summary stats for flux certainty
    }
    
    agg_df2 = groupby_aggs(df, aggs2)
    agg_df = agg_df.merge(agg_df2,on=['object_id'],how='left')
    del agg_df2

    agg_df['flux_diff'] = agg_df['max_flux'] - agg_df['min_flux']
    agg_df['flux_dif2'] = (agg_df['max_flux'] - agg_df['min_flux']) / agg_df['mean_flux']
    
    agg_df['flux_w_mean'] = agg_df['sum_flux_by_flux_ratio_sq'] / agg_df['sum_flux_ratio_sq']
    agg_df['flux_dif3'] = (agg_df['max_flux'] - agg_df['min_flux']) / agg_df['flux_w_mean']
    
    agg_df['mjd_diff'] = agg_df['max_mjd'] - agg_df['min_mjd']
    agg_df.drop_column('max_mjd')
    agg_df.drop_column('min_mjd')
    
    for col in ['ra','decl','gal_l','gal_b']:
        df_meta.drop_column(col)
    
    df_meta = df_meta.merge(agg_df,on=['object_id'],how='left')
    return df_meta
train_final_gd = compute_statistical_features(train_gd,train_meta_gd)
test_final_gd = compute_statistical_features(test_gd,test_meta_gd)
del train_gd,train_meta_gd
del test_gd,test_meta_gd<jupyter_output><empty_output><jupyter_text>Part 3 - Merge Flux Skew & Statistical SummariesWe use cuDF's `merge()` to combine the flux skew and statistical summary Dataframes by `object_id`<jupyter_code>train_final_gd = train_final_gd.merge(train_flux_skew_gd,on=['object_id'],how='left')
test_final_gd = test_final_gd.merge(test_flux_skew_gd,on=['object_id'],how='left')
train_final_gd.head().to_pandas()<jupyter_output><empty_output><jupyter_text>Part 4 - Merge in Bottleneck FeaturesLoad the bottleneck features extracted from the RNN in `Part 1.2` and concatenate with the statistical features. At this point, the notebook from `Part 1.2` should have been executed end-to-end. You can also save time by using a set of features that have already been extracted.<jupyter_code>train_bn_file = "train_bn.pkl"  # Use `train_bn_saved.pkl` for pre-extracted features
test_bn_file = "test_bn.pkl"    # Use `test_bn_saved.pkl` for pre-extracted features
train_bn = gd.from_pandas(pd.read_pickle(train_bn_file))
test_bn = gd.from_pandas(pd.read_pickle(test_bn_file))
train_final_gd = train_final_gd.merge(train_bn,on=['object_id'],how='left')
test_final_gd = test_final_gd.merge(test_bn,on=['object_id'],how='left')
train_final_gd.head().to_pandas()
del train_bn,test_bn  # Save device? memory<jupyter_output><empty_output><jupyter_text>Store Final Train/Test Data to Disk Store our extracted data out to csv files so that we can use it downstream<jupyter_code>train_final_gd.to_pandas().to_pickle("train_gdf.pkl")
test_final_gd.to_pandas().to_pickle("test_gdf.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.127106466361855, 'fraction_numerical': 0.008099281515349445, 'mean_word_length': 5.799289520426288, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '473698', 'license': 'Apache-2.0', 'n_tokens_mistral': 2874, 'n_tokens_neox': 2740, 'n_words': 593, 'path': 'conference_notebooks/KDD_2019/notebooks/plasticc/Part 2.1 - Feature Engineering.ipynb', 'repo_name': 'gumdropsteve/notebooks-contrib', 'size': '7655'}