<jupyter_start><jupyter_text>![LaSPACE_logo.jpg](attachment:LaSPACE_logo.jpg) Making NASA’s Open-Innovation Data More Machine-Learning Friendly: A Case for the MAVEN Datasets LURA Project 2020-2021LURA Project Title: "Making NASA's Open-Innovation Data More Machine-Learning Friendly: A Case for the MAVEN Datasets"  This research grant was awarded by the Louisiana Space Grant Consortium (LaSPACE). URL: https://laspace.lsu.edu/about-us/.  LaSPACE Grant Number: ---TBA---, Primary NASA Grant Number: 80NSSC20M0110.Project P.I.: Dr. <NAME>  Apprentice Student: <NAME>  Department of Chemical Engineering, University of Louisiana at Lafayette, URL: https://chemical.louisiana.edu/Recent Notebook Revision: 26 Aug 2020  Notebook Overview:This Jupyter Notebook is running Python shows that step-by-step workflow from the download to the analysis of the MAVEN datasets.The PyDIVIDE package is used to download the MAVEN datasets and to perform som graphical analysis of the data. It was created by the MAVEN Team at the University of Colorado - Boulder: https://github.com/MAVENSDC/Pydivide. It is main used in this project to download and read the datasets.The Self-Organizing Map (SOM) package is available from this GitHub repository: https://github.com/sevamoo/SOMPY.git.  Install the PyDIVIDE package:<jupyter_code>pip install pydivide # no need to run if pydivide is already installed, just import (next line)<jupyter_output><empty_output><jupyter_text>Import the Python Packages:<jupyter_code>import pydivide 
import matplotlib.pylab as plt
%matplotlib inline
import pandas as pd
import numpy as np
from time import time
import sompy
import random
from sklearn.externals import joblib
import glob<jupyter_output><empty_output><jupyter_text>Download MAVEM Data from NASA Datacenter:Download the target MAVEN dataset into the local directory. This may take minutes depending on the netwrok speed.After running the code line below, you do not have to run it again; unless you are downloading from another time period.The download function allows for the specification of destination folder using the 'local_dir=path' argument.<jupyter_code>pydivide.download_files(start_date='2018-06-01',
                              end_date='2018-06-30',
                              local_dir='/Users/dhan-lordfortela/Documents/EIL/Proposals/LURA 2020/LURA 2020 - Grant Awarded/LURA_2020 - Working/FINAL_works/on_MAVEN_data',
                              insitu=True,
                               iuvs=True)<jupyter_output><empty_output><jupyter_text>Extract the raw data to working dataset Reading the downloaded datset.<jupyter_code>insitu= pydivide.read(['2018-06-2', '2018-06-29'],insitu_only=True)<jupyter_output><empty_output><jupyter_text>Check the Keys of the data array:<jupyter_code>insitu.keys()<jupyter_output><empty_output><jupyter_text>We will focus on the SWEA subset of the data:<jupyter_code>insitu_SWEA = insitu.get("SWEA")
insitu_SWEA.head()<jupyter_output><empty_output><jupyter_text>Check the number of data:<jupyter_code>insitu_SWEA.count()<jupyter_output><empty_output><jupyter_text>Remove the last column as it has no data:<jupyter_code>insitu_SWEA= insitu_SWEA.iloc[: , 0:-1]
insitu_SWEA.head()<jupyter_output><empty_output><jupyter_text>Remove observations with missing entries:<jupyter_code>insitu_SWEA.isnull().sum()
insitu_SWEA_clean = insitu_SWEA.dropna()
insitu_SWEA_clean.count()<jupyter_output><empty_output><jupyter_text>Saving the extracted SWEA data as '.xlsx. file for other use:<jupyter_code>insitu_SWEA_clean.to_excel('Data_SWEA.xlsx')<jupyter_output><empty_output><jupyter_text>Import the MAVEN Data:Trying the MAVEN data:<jupyter_code>MAVEN_data = insitu_SWEA_clean
#MAVEN_data = pd.read_csv('/Users/dhan-lordfortela/Documents/EIL/Proposals/LURA 2020/LURA 2020 - Grant Awarded/LURA_2020 - Working/FINAL_works/sample_sompy_implement/Basic uses of SOMPY library/cleaned_SWEA_info_csv.csv')<jupyter_output><empty_output><jupyter_text>Examine the structure of the data:<jupyter_code>MAVEN_data.head()
MAVEN_data.count()<jupyter_output><empty_output><jupyter_text>Training the SOM on the dataset Set the size of the SOM to be [row,col]. Set the other SOM parameters in the argument.As the data is relatively high, the model takes some time to train. We didn't finetune the hyperparameters of the algorithm and this is a potential improvement topic.<jupyter_code>%%time
# Train the model with different parameters. 
#The more, the better. 
#Each iteration is stored in disk for further study

for i in range(2):
    som = sompy.SOMFactory().build(MAVEN_data.values, 
                                   mapsize=[random.choice(list(range(15, 25))),random.choice(list(range(10, 15)))],
                                   normalization = 'var', 
                                   initialization='random', 
                                   #component_names=component_names, 
                                   lattice="hexa")
    som.train(n_job=4, train_rough_len=5, train_finetune_len=10)
    joblib.dump(som, "model_{}.joblib".format(i))<jupyter_output>Training...
 random_initialization took: 0.005000 seconds
 Rough training...
 radius_ini: 6.000000 , radius_final: 1.000000, trainlen: 5

 epoch: 1 ---> elapsed time:  0.910000, quantization error: 42.172873

 epoch: 2 ---> elapsed time:  0.790000, quantization error: 3.222791

 epoch: 3 ---> elapsed time:  0.795000, quantization error: 3.063902

 epoch: 4 ---> elapsed time:  0.812000, quantization error: 2.905707

 epoch: 5 ---> elapsed time:  0.814000, quantization error: 2.450950

 Finetune training...
 radius_ini: 1.500000 , radius_final: 1.000000, trainlen: 10

 epoch: 1 ---> elapsed time:  0.802000, quantization error: 1.699155

 epoch: 2 ---> elapsed time:  0.828000, quantization error: 1.822348

 epoch: 3 ---> elapsed time:  0.789000, quantization error: 1.765931

 epoch: 4 ---> elapsed time:  0.813000, quantization error: 1.722748

 epoch: 5 ---> elapsed time:  0.823000, quantization error: 1.683453

 epoch: 6 ---> elapsed time:  0.822000, quantization error: 1.645454

 epoch[...]<jupyter_text>Calculate the SOM model Topographic error and Quantization error:<jupyter_code># Study the models trained and plot the errors obtained in order to select the best one
models_pool = glob.glob("./model*")
errors=[]
for model_filepath in models_pool:
    som = joblib.load(model_filepath)
    topographic_error = som.calculate_topographic_error()
    quantization_error = som.calculate_quantization_error()
    errors.append((topographic_error, quantization_error))
e_top, e_q = zip(*errors)<jupyter_output><empty_output><jupyter_text>Plot the Topographic error and Quantization error:<jupyter_code>plt.scatter(e_top, e_q)
plt.xlabel("Topographic error")
plt.ylabel("Quantization error")
plt.show()
mapsize = [55,50]
som = sompy.SOMFactory.build(MAVEN_data.values, mapsize, mask=None, mapshape='planar', lattice='rect', normalization='var', initialization='pca', neighborhood='gaussian', training='batch', name='sompy')  # this will use the default parameters, but i can change the initialization and neighborhood methods
som.train(n_job=1, verbose='info')  # verbose='debug' will print more, and verbose=None wont print anything

topographic_error = som.calculate_topographic_error()
quantization_error = som.calculate_quantization_error()
print(topographic_error, quantization_error)
som.component_names = list(MAVEN_data.columns)
print(som.component_names)

v = sompy.mapview.View2DPacked(55, 50, 'test',text_size=8)  
# could be done in a one-liner: sompy.mapview.View2DPacked(300, 300, 'test').show(som)
#v.show(som, what='codebook', which_dim=[0,1], cmap=None, col_sz=17) #which_dim='all' default
v.show(som, what='codebook', which_dim='all', cmap='jet') #which_dim='all' default
# v.save('2d_packed_test')<jupyter_output><empty_output><jupyter_text>Showing selected components. Select components by specifying the index in 'which_dim=[...]' argument.<jupyter_code>v = sompy.mapview.View2DPacked(50, 50, 'test',text_size=8)  
v.show(som, what='codebook', which_dim=[0,1], cmap='jet') #which_dim='all' default<jupyter_output><empty_output><jupyter_text>Clustering the SOM:<jupyter_code># c = sompy.mapview.View2DPacked()
v = sompy.mapview.View2DPacked(2, 2, 'test',text_size=8)  
#first you can do clustering. Currently only K-means on top of the trained som
cl = som.cluster(n_clusters=4)
# print cl
getattr(som, 'cluster_labels')
v.show(som, what='cluster')
h = sompy.hitmap.HitMapView(10, 10, 'hitmap', text_size=8, show_text=True)
h.show(som)
u = sompy.umatrix.UMatrixView(50, 50, 'umatrix', show_axis=True, text_size=8, show_text=True)

#This is the Umat value
UMAT  = u.build_u_matrix(som, distance=1, row_normalized=False)

#Here you have Umatrix plus its render
UMAT = u.show(som, distance2=1, row_normalized=False, show_data=True, contooor=True, blob=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10632797572215354, 'fraction_numerical': 0.038889513319096326, 'mean_word_length': 5.077868852459017, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 81, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '104155', 'license': 'MIT', 'n_tokens_mistral': 3230, 'n_tokens_neox': 2922, 'n_words': 767, 'path': 'LURA_2020_Final_JupyterNotebook.ipynb', 'repo_name': 'dhanfort/NASA-LURA-2020-dlbfortela', 'size': '8921'}