<jupyter_start><jupyter_text>Exoplanet Transit Classification (Kepler Mission) Data Wrangling Part I I. Extracting Kepler Cumulative Table<jupyter_code>#standard data wrangling packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import requests

import os
import os.path


caltech_KOI_url = 'https://exoplanetarchive.ipac.caltech.edu/cgi-bin/nstedAPI/nph-nstedAPI?table=cumulative'
KOI_data = pd.read_csv(caltech_KOI_url)
KOI_data.columns<jupyter_output><empty_output><jupyter_text>These columns index Kepler objects of interests with transit crossing events TCEs identified in the light curves observed on a given star. Data column definitions can be found at https://exoplanetarchive.ipac.caltech.edu/docs/API_kepcandidate_columns.html. Many of the planetary and orbital parameters are quantities derived from model fits to the light curve data. The stellar parameters are derived from ground based spectroscopy.<jupyter_code>KOI_data.shape
KOI_data.head()
KOI_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 9564 entries, 0 to 9563
Data columns (total 50 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   kepid              9564 non-null   int64  
 1   kepoi_name         9564 non-null   object 
 2   kepler_name        2360 non-null   object 
 3   koi_disposition    9564 non-null   object 
 4   koi_pdisposition   9564 non-null   object 
 5   koi_score          8054 non-null   float64
 6   koi_fpflag_nt      9564 non-null   int64  
 7   koi_fpflag_ss      9564 non-null   int64  
 8   koi_fpflag_co      9564 non-null   int64  
 9   koi_fpflag_ec      9564 non-null   int64  
 10  koi_period         9564 non-null   float64
 11  koi_period_err1    9110 non-null   float64
 12  koi_period_err2    9110 non-null   float64
 13  koi_time0bk        9564 non-null   float64
 14  koi_time0bk_err1   9110 non-null   float64
 15  koi_time0bk_err2   9110 non-null   float64
 16  koi_impact         9201 [...]<jupyter_text>Label Identification: Defining the classes<jupyter_code>KeplerCPFPcount = {'Kepler Confirmed Planet': (KOI_data['koi_disposition'] == 'CONFIRMED').sum(), 'Kepler False Positives': (KOI_data['koi_disposition'] == 'FALSE POSITIVE').sum()}
print(KeplerCPFPcount)<jupyter_output>{'Kepler Confirmed Planet': 2358, 'Kepler False Positives': 4840}<jupyter_text>Subsetting on confirmed planets and false positives, we see that we have a healthy number in each class for doing binary classification. The Kepler cumulative table is nice enough to have flags indicating what kind of false positive the transit is believed to be. koi_fpflag_nt is a non-transiting oscillatory phenomenon in the light curve. These can come from variable pulsing stars. This also encompasses instrument noise and the like. koi_fpflag_ss flags secondary signal phenomena. Thus we have a signal with a primary and secondary dip in a given period. These are likely due to eclipsing binaries that orbit the star of interest and create periodic dips in the light curve amplitude that can mimic a transiting planet.  koi_fpflag_co is due to a star near to the star of interest and in the field of view with its motion creating a periodic modulation of the light curve. This results in a fluctuation of the centroid of light on the detecting aperture (centroid offset). Note: Centroid offsets and secondary signal flags and are not mutually exlusive.<jupyter_code>{'Secondary Signal': (KOI_data['koi_fpflag_ss'] == 1).sum(), 'Non Transiting': (KOI_data['koi_fpflag_nt'] == 1).sum(), 'Centroid Offset': (KOI_data['koi_fpflag_co'] == 1).sum()}<jupyter_output><empty_output><jupyter_text>Overlap of secondary signal false positives with centroid offset:<jupyter_code>((KOI_data['koi_fpflag_ss'] == 1) & (KOI_data['koi_fpflag_co'] == 1)).sum()<jupyter_output><empty_output><jupyter_text>An interesting point is that there is a small overlap between secondary signal false positives and a transit that has actually been confirmed to be a planet. It turns out that there are a very small fraction of seemingly eclipsing binary star partners that are in fact 'Hot Jupiters'. These are planets that are really large and close to the star thus sending out a high degree of reflected light from the star. This can result in weak secondary transit phenomena.<jupyter_code>((KOI_data['koi_fpflag_ss'] == 1) & (KOI_data['koi_disposition'] == 'CONFIRMED')).sum()<jupyter_output><empty_output><jupyter_text>Let's also take a look at the overlap between non-transiting flags and confirmed planets / secondary transiting phenomena.<jupyter_code>((KOI_data['koi_fpflag_nt'] == 1) & (KOI_data['koi_disposition'] == 'CONFIRMED')).sum()
((KOI_data['koi_fpflag_nt'] == 1) & (KOI_data['koi_fpflag_ss'] == 1)).sum()<jupyter_output><empty_output><jupyter_text>This is certainly interesting and probably indicates that members of each class live at/near a classification boundary. Maybe these will be reflected in some of the catalog features and time-series extracted features which we can take a look at when doing EDA. The following has confirmed planets, false positive secondaries and false positive non transiting. We're going to do a three-way classification. We have excluded centroid offset because these are usually determined by looking at pixel imaging of the telescope and not solely from image-averaged light curves.<jupyter_code>KOI_intermediate =  KOI_data.loc[(KOI_data['koi_disposition'] == 'CONFIRMED') | ( (KOI_data['koi_disposition'] == 'FALSE POSITIVE') & (KOI_data['koi_fpflag_ss'] == 1) ) | ( (KOI_data['koi_disposition'] == 'FALSE POSITIVE') & (KOI_data['koi_fpflag_nt'] == 1) )]
KOI_intermediate.head()<jupyter_output><empty_output><jupyter_text>Drop intersection of non-transiting and secondary signal false positives and intersection of confirmed planets and nt flags.<jupyter_code>ss_nt_intersection = KOI_intermediate[(KOI_intermediate['koi_fpflag_nt'] == 1) & (KOI_intermediate['koi_fpflag_ss'] == 1)]
ss_nt_intersection.head()
cp_nt_intersection = KOI_intermediate[(KOI_intermediate['koi_disposition'] == 'CONFIRMED') & (KOI_intermediate['koi_fpflag_nt'] == 1)]
KOI_intermediate = KOI_intermediate.drop(index = ss_nt_intersection.index).drop(index = cp_nt_intersection.index)<jupyter_output><empty_output><jupyter_text>All classes mutually distinct: now construct target labels: 1 = Confirmed Planet, 2 = Secondary Signal False Positive, 3 = Non transiting phenomena<jupyter_code># label encode the three classes

KOI_intermediate['target_label'] = (KOI_intermediate['koi_disposition'] == 'CONFIRMED').astype('int') + 2*KOI_intermediate['koi_fpflag_ss'] + 3*KOI_intermediate['koi_fpflag_nt'] 

KOI_intermediate['target_label'].value_counts()<jupyter_output><empty_output><jupyter_text>There's a wierd value in there, let's check it out. It seems to be just one observation. Looks like an error.<jupyter_code>KOI_intermediate[KOI_intermediate['target_label'] == 1396]
KOI_intermediate.loc[KOI_intermediate['target_label'] == 1396, 'target_label'] = 1
KOI_intermediate.loc[KOI_intermediate['koi_fpflag_nt'] == 465, 'koi_fpflag_nt'] = 0
KOI_intermediate['target_label'].value_counts()<jupyter_output><empty_output><jupyter_text>Taking subset of dataframe and data cleanup<jupyter_code>KOI_intermediate.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6061 entries, 0 to 9559
Data columns (total 51 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   kepid              6061 non-null   int64  
 1   kepoi_name         6061 non-null   object 
 2   kepler_name        2357 non-null   object 
 3   koi_disposition    6061 non-null   object 
 4   koi_pdisposition   6061 non-null   object 
 5   koi_score          5337 non-null   float64
 6   koi_fpflag_nt      6061 non-null   int64  
 7   koi_fpflag_ss      6061 non-null   int64  
 8   koi_fpflag_co      6061 non-null   int64  
 9   koi_fpflag_ec      6061 non-null   int64  
 10  koi_period         6061 non-null   float64
 11  koi_period_err1    5813 non-null   float64
 12  koi_period_err2    5813 non-null   float64
 13  koi_time0bk        6061 non-null   float64
 14  koi_time0bk_err1   5813 non-null   float64
 15  koi_time0bk_err2   5813 non-null   float64
 16  koi_impact         5865 [...]<jupyter_text>The kepid and koi_tce_plnt_num are the Kepler Input Catalog ID (corresponding to a given star) and TCE number used to index light curves in the Kepler data validation pipeline. We'll use these indices to extract features from the light curve time series via a custom built class (KOI_class). Thus these are our indices.  Feature that is extracted from Kepler's data validation automated statistical testing is the koi_period. Keep this. Also keep koi_depth and koi_duration which are the depth and duration of the primary transit event. Other columns have to do with stellar parameters (e.g. koi_steff, stellar coordinates, etc) not gained directly from light curves as opposed to spectroscopy as well as parameters fit assuming the TCE was a planet. Let's drop all of those.<jupyter_code>cols_to_keep = ['kepid', 'koi_tce_plnt_num', 'koi_period', 'koi_depth', 'koi_duration', 'target_label']
KOI_cumulative = KOI_intermediate[cols_to_keep]
KOI_cumulative.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6061 entries, 0 to 9559
Data columns (total 6 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   kepid             6061 non-null   int64  
 1   koi_tce_plnt_num  5882 non-null   float64
 2   koi_period        6061 non-null   float64
 3   koi_depth         5865 non-null   float64
 4   koi_duration      6061 non-null   float64
 5   target_label      6061 non-null   int64  
dtypes: float64(4), int64(2)
memory usage: 331.5 KB<jupyter_text>Take care of any remaining NaNs<jupyter_code>KOI_cumulative = KOI_cumulative.dropna(how = 'any')
KOI_cumulative.info()
KOI_cumulative.head()<jupyter_output><empty_output><jupyter_text>Finish up cleaning on KOI_cumulatve: data types, relabel columns Still good amount in each class after clean up<jupyter_code>KOI_cumulative['target_label'].value_counts()<jupyter_output><empty_output><jupyter_text>The values for koi_tce_plnt_num are clearly integers...but are interpreted as floats. change dtype to int<jupyter_code>KOI_cumulative['koi_tce_plnt_num'].unique()
KOI_cumulative['koi_tce_plnt_num'] = KOI_cumulative['koi_tce_plnt_num'].astype('int')
print(KOI_cumulative['koi_tce_plnt_num'].dtype)
print(KOI_cumulative['koi_tce_plnt_num'].unique())<jupyter_output>int32
[1 2 3 4 5 6 7 8]<jupyter_text>Rename columns<jupyter_code>KOI_cumulative.columns
KOI_cumulative = KOI_cumulative.rename(columns={'kepid': 'KIC_ID', 'koi_tce_plnt_num': 'TCE_num', 'koi_period': 'Period', 'koi_depth': 'Depth', 'koi_duration': 'Duration'})<jupyter_output><empty_output><jupyter_text>II. Light Curve Time Series Download<jupyter_code>KOI_cumulative = KOI_cumulative.set_index(['KIC_ID', 'TCE_num']).sort_index()
KOI_cumulative.head()
kic_tcenum_index = KOI_cumulative.index # generates 2-tuple Multindex
#import customized Kepler Object of Interest class
import sys
sys.path.append("..\\src\\")
from KOIclass import KOIObject # loads custom library to take care of API requests, extracting the data, download, and logging<jupyter_output><empty_output><jupyter_text>WARNING! This next command can take some time as there's about 26 GB of data to be downloaded to the local file system. The download is necessary as the main speed bottleneck in transforming the data and feature creation would be in actually loading it remotely via API requests.  Some KIC ID/TCEs that exist in the kepler input catalog don't have data in the data validation pipeline. These are, when they do happen, almost always non-transiting phenomena. We don't save a file for these offending TCEs and instead log the KIC and TCE in the download.log file.<jupyter_code>kic_tcenum_index.map(lambda x: KOIObject(x[0], x[1]).download_data() )<jupyter_output><empty_output><jupyter_text>Remove all KICs with empty data in data validation from Kepler cumulative table.<jupyter_code>logfile_path = "..\\data\\external\\DVSeries\\download.log"
emptydvdata_list = []
with open(logfile_path) as f:
    for x in f.readlines():
        kicstring, tcenumstring = x.strip().split(":")[2].split('_')
        emptydvdata_list.append((int(kicstring), int(tcenumstring)))
print(emptydvdata_list[0:5])<jupyter_output>[(1161949, 1), (1297263, 1), (1433531, 1), (2163226, 1), (2305311, 1)]<jupyter_text>There are 365 entries in KOI_cumulative that have no data in the data validation pipeline.<jupyter_code>KOI_cumulative.loc[emptydvdata_list].head()<jupyter_output><empty_output><jupyter_text>Mostly non-transiting, but there are some secondary eclipses / confirmed planets missing in the data validation pipeline<jupyter_code>KOI_cumulative.loc[emptydvdata_list]['target_label'].value_counts()<jupyter_output><empty_output><jupyter_text>Drop all of these from the cumulative table then save to file.<jupyter_code>KOI_cumulative = KOI_cumulative.drop(index=emptydvdata_list)<jupyter_output><empty_output><jupyter_text>There are quite a bit fewer non transiting phenomena in our set than the other two classes...but it may be alright.<jupyter_code>KOI_cumulative['target_label'].value_counts()
KOI_cumulative.to_csv('..\\data\\external\\KOI_cumulative.csv', sep=",")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08883305003325696, 'fraction_numerical': 0.03214839997043825, 'mean_word_length': 4.83779119930975, 'pattern_counts': {'":': 1, '<': 94, '<?xml version=': 0, '>': 94, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '524791', 'license': 'FTL', 'n_tokens_mistral': 4811, 'n_tokens_neox': 4311, 'n_words': 1308, 'path': 'notebooks/Exo_DataWrangling_Part1.ipynb', 'repo_name': 'admveen/Exoplanet', 'size': '13531'}