<jupyter_start><jupyter_text>Exoplanet Transit Classification (Kepler Mission) Data Preprocessing<jupyter_code>### X|y split + training and hold-out dataset creation
feature_df_path = "..\\data\\interim\\feat_df_tot.csv"

df_exo = pd.read_csv(feature_df_path, index_col = ['KIC_ID', 'TCE_num'])
cols_to_drop = ["time_complexity", "Depth", "rms"]
df_exo = df_exo.drop(columns = cols_to_drop)
df_exo.head(3)
# define X and y
X = df_exo.drop(columns = ['target_label'])
y = df_exo['target_label']

#now create train + hold out set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15, stratify = y)
y_test.value_counts()
y_train.value_counts()
# At this point, X_train, y_train, x_test, y_test will be saved as separate files. That way the modeling is being evaluated from a particular train/test sample as saved on file.

base_path = "..\\data\\processed\\"
datasplit_dict = {"X_train": X_train, "y_train": y_train, "X_test": X_test, "y_test": y_test}

for name, data in datasplit_dict.items():
    data.to_csv(base_path + name + ".csv")<jupyter_output><empty_output><jupyter_text>Data preprocessing on train/hold-out test data from file<jupyter_code># load X_train from file
X_train = pd.read_csv(base_path + 'X_train.csv', index_col=['KIC_ID', 'TCE_num'] )
#transformer class to perform sqrt log transform on p_secondary and even_odd_stat

class SqrtLogZeroExceptionTransformer(BaseEstimator, TransformerMixin):

    def __init__(self):
        super().__init__()
        self.trans_evenodd_max_ = None
        self.trans_psec_max_ = None

    def fit(self, X, y = None):
        cols_to_take = ['even_odd_stat', 'p_secondary'] #these are the columns for which we'll do the sqrt log transformation

        with np.errstate(divide='ignore'):
            X_trans = np.sqrt(np.abs(np.log10(X[cols_to_take])))
        # cuts out infinities and gets maxes --> will use this to impute infinities (which are zeros in the original feature set)
        maxsqrtlog = X_trans[~(X_trans == np.inf)].max()

        self.trans_evenodd_max_ = maxsqrtlog['even_odd_stat']
        self.trans_psec_max_ = maxsqrtlog['p_secondary']

        return self
    
    def transform(self, X, y = None):
        cols_to_take = ['even_odd_stat', 'p_secondary'] #these are the columns for which we'll do the sqrt log transformation

        # this will issue some divide by zero warnings. im going to suppress this warning.
        with np.errstate(divide='ignore'):
            X_trans = np.sqrt(np.abs(np.log10(X[cols_to_take])))


        X_toreturn = deepcopy(X)
        X_toreturn['even_odd_stat'] = X_trans['even_odd_stat'].replace(np.inf, self.trans_evenodd_max_)
        X_toreturn['p_secondary'] = X_trans['p_secondary'].replace(np.inf, self.trans_psec_max_)

        return X_toreturn

    def fit_transform(self, X, y = None):
        return self.fit(X).transform(X)<jupyter_output><empty_output><jupyter_text>Set up column transformer for LPP dim reduction on LCBIN data and then construct transformation pipeline<jupyter_code># transformer class to do the LPP dimensional reduction on the LCBIN subset.
#  the columntransformer only performs LPP on the 141 LCBIN features.

LCBINselector = make_column_selector(pattern = "^LCBIN_")
trans = [('lpp', LocalityPreservingProjection(n_components=2), LCBINselector(X_train) )]

# pipeline first sqrt log transforms secondary peak detection features, then dim reduction on LCBIN, then min/max scaling
steps = [('sqrtlog', SqrtLogZeroExceptionTransformer()), ('lpptrans', ColumnTransformer(transformers = trans, remainder = 'passthrough') ), ('scaler', MinMaxScaler() )]

# fit_transforming the pipeline will output a numpy array (LocalityPreservingProjection outputs numpy array). 
# column order is (LPP_1, LPP_2, Period, Duration, even_odd_stat, p_secondary, max, min)

datatrans_pipe = Pipeline(steps)
datatrans_pipe.fit(X_train)<jupyter_output><empty_output><jupyter_text>Before we pickle our preprocessing want to take a look at how the features to see if anything bad happened.<jupyter_code>colnames = ['LPP_1', 'LPP_2', 'Period', 'Duration', 'even_odd_stat', 'p_secondary', 'max', 'min']
normXtrain_df = pd.DataFrame(datatrans_pipe.transform(X_train), columns = colnames, index = X_train.index)
normXtrain_df.head()
normXtrain_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
MultiIndex: 4551 entries, (11853878, 3) to (12356914, 1)
Data columns (total 8 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   LPP_1          4551 non-null   float64
 1   LPP_2          4551 non-null   float64
 2   Period         4551 non-null   float64
 3   Duration       4551 non-null   float64
 4   even_odd_stat  4551 non-null   float64
 5   p_secondary    4551 non-null   float64
 6   max            4551 non-null   float64
 7   min            4551 non-null   float64
dtypes: float64(8)
memory usage: 649.5 KB<jupyter_text>Good, no NaNs. The standard describe statistics are not useful for our features. Let's just do some quick visualization to check if the distributions look the same as when we did the EDA.<jupyter_code>sns.scatterplot(x = 'LPP_1', y = 'LPP_2', data = normXtrain_df)
for col in normXtrain_df.columns:
    normXtrain_df[col].hist()
    plt.show()<jupyter_output><empty_output><jupyter_text>Looks   Pickle data transformation pipeline for later use<jupyter_code>preprocess_object_path = "..\\models\\preprocessing\\preprocess_pipeline.pkl"
pickle.dump(datatrans_pipe, open(preprocess_object_path, 'wb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10863309352517986, 'fraction_numerical': 0.019064748201438848, 'mean_word_length': 4.311365807067813, 'pattern_counts': {'":': 4, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '516736', 'license': 'FTL', 'n_tokens_mistral': 1948, 'n_tokens_neox': 1844, 'n_words': 499, 'path': 'notebooks/Exo_Preprocessing.ipynb', 'repo_name': 'admveen/Exoplanet', 'size': '5560'}