<jupyter_start><jupyter_text>Cleaning Astronomical DatasetsI have posed two problems for you to work on in this hands-on exercise.  - Concept drift: Do the training and test set distributions differ?- Find mislabeled examples in the labeled ZTF data providedMore information for each is provided below.  You may want or need to cut and paste code from your other notebooks.  But first...  0a. ImportsThese are all the imports that will be used in this notebook.  All should be available in the DSFP conda environment.<jupyter_code>import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
# You can add anything you need as you work<jupyter_output><empty_output><jupyter_text>0b. Data LocationWe will use the same data from the Day 2 clustering exercise (see [that notebook](https://github.com/LSSTC-DSFP/LSSTC-DSFP-Sessions/blob/master/Session7/Day2/Clustering-Astronomical-Sources.ipynb) to download the data).Please specify paths for the following:<jupyter_code>F_META = '../Day2/dsfp_ztf_meta.npy'
F_FEATS = '../Day2/dsfp_ztf_feats.npy'
D_STAMPS = '../Day2/dsfp_ztf_png_stamps'<jupyter_output><empty_output><jupyter_text>0c. Load Data<jupyter_code>meta_np = np.load(F_META)
feats_np = np.load(F_FEATS)

COL_NAMES = ['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap',
             'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 'sky',
             'magdiff', 'fwhm', 'classtar', 'mindtoedge', 'magfromlim', 'seeratio',
             'aimage', 'bimage', 'aimagerat', 'bimagerat', 'elong', 'nneg',
             'nbad', 'ssdistnr', 'ssmagnr', 'sumrat', 'magapbig', 'sigmagapbig',
             'ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 'scorr', 'label']

# NOTE FROM Umaa: I've decided to eliminate the following features. Dropping them.
#
COL_TO_DROP = ['ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 
               'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 
               'classtar', 'ssdistnr', 'ssmagnr', 'aimagerat', 'bimagerat', 
               'magapbig', 'sigmagapbig', 'scorr']
feats_df = pd.DataFrame(data=feats_np, index=meta_np['candid'], columns=COL_NAMES)
print("There are {} columns left.".format(len(feats_df.columns)))
print("They are: {}".format(list(feats_df.columns)))
feats_df.drop(columns=COL_TO_DROP, inplace=True) 
#feats_df.describe()<jupyter_output>There are 36 columns left.
They are: ['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap', 'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 'sky', 'magdiff', 'fwhm', 'classtar', 'mindtoedge', 'magfromlim', 'seeratio', 'aimage', 'bimage', 'aimagerat', 'bimagerat', 'elong', 'nneg', 'nbad', 'ssdistnr', 'ssmagnr', 'sumrat', 'magapbig', 'sigmagapbig', 'ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 'scorr', 'label']<jupyter_text>1. Concept DriftIn the last exercise, you created a training and test set for the purposes of building a classifier.  The goal of this exercise is to note any changes in the feature distributions for these two sets.Per feature, can you:- plot test vs. train distributions for both real and bogus, and note areas that does not overlap- quantiatively measure this using [Kullback-Leibler divergence](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html), and print or plot the scores for all features.Which feature exhibits the most drift between train and test?<jupyter_code>from sklearn.preprocessing import MinMaxScaler, StandardScaler
nids = meta_np['nid']
feats_df['nids'] = nids
print(feats_df.columns)
'''labels = feats_plus_label[:,-1] # labels only
feats_plus_label = feats_plus_label[:,:-1] # features only
feats_plus_label_scaled = np.column_stack((feats_plus_label, labels))
train = feats_plus_label_scaled[nids < 550]
test = feats_plus_label_scaled[nids >= 550]
train = print(feats_df[feats_df.])'''<jupyter_output>Index(['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap',
       'sky', 'magdiff', 'fwhm', 'mindtoedge', 'magfromlim', 'seeratio',
       'aimage', 'bimage', 'elong', 'nneg', 'nbad', 'sumrat', 'label', 'nids'],
      dtype='object')
{'fraction_non_alphanumeric': 0.14397266959492436, 'fraction_numerical': 0.004880429477794046, 'mean_word_length': 5.4046875, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '326820', 'license': 'MIT', 'n_tokens_mistral': 1467, 'n_tokens_neox': 1436, 'n_words': 388, 'path': 'Session7/Day3/Cleaning-Astronomical-Datasets.ipynb', 'repo_name': 'ccdoughty/LSSTC-DSFP-Sessions', 'size': '4098'}