<jupyter_start><jupyter_text>Grouping XAS spectra represented as MCFF spectra, using DBSCAN in sci-kit learnFrom the preprocess we have normalized XAS spectra above the edge, and transformed the data into an array of coefficients, called the Mel-cepstrum frequency coefficients. With this we can now select elements of interest, and group all of the spectra at the edge.As we tweak DBSCAN parameters, we can look at the distribution of clusters to decide what the most helpful way to cluster spectra is. When satisfied we can save the dataframe of As done in the preprocessing, we will download, install and initiate a Spark session with pyspark. Java8 is also installedFor pyspark set-up in colab environment, I am following https://towardsdatascience.com/pyspark-in-google-colab-6821c2faf41c except for the fact I am wget'ing an archived spark (2.3.3)<jupyter_code>!wget http://archive.apache.org/dist/spark/spark-2.3.3/spark-2.3.3-bin-hadoop2.7.tgz
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!tar -xvf spark-2.3.3-bin-hadoop2.7.tgz
!pip install -q findspark
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.3.3-bin-hadoop2.7"
import findspark
findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()<jupyter_output><empty_output><jupyter_text>Loading CSV of mfcc vecs from exafs<jupyter_code>!wget "http://exafs.ucsc.edu/bridges/mfcc_gmini.zip"<jupyter_output>--2020-02-26 23:41:08--  http://exafs.ucsc.edu/bridges/mfcc_gmini.zip
Resolving exafs.ucsc.edu (exafs.ucsc.edu)... 192.168.127.12
Connecting to exafs.ucsc.edu (exafs.ucsc.edu)|192.168.127.12|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-02-26 23:41:09 ERROR 404: Not Found.<jupyter_text>We use a Spark session and immediately convert to pandas. This is only done in the interest that future development can be done via SparkML for ease of scaling up to large XAS databases.<jupyter_code>from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
import pandas as pd
!unzip -o Femcff_gmini
dataset = spark.read.load("mcff_edge_2col_gmini.csv",format="csv",inferSchema="true")
pdf=dataset.toPandas()<jupyter_output><empty_output><jupyter_text>This greek alphabet dict is used for category labels in plot later ...<jupyter_code>greek_alphabet = {
    u'\u0391': 'Alpha',
    u'\u0392': 'Beta',
    u'\u0393': 'Gamma',
    u'\u0394': 'Delta',
    u'\u0395': 'Epsilon',
    u'\u0396': 'Zeta',
    u'\u0397': 'Eta',
    u'\u0398': 'Theta',
    u'\u0399': 'Iota',
    u'\u039A': 'Kappa',
    u'\u039B': 'Lamda',
    u'\u039C': 'Mu',
    u'\u039D': 'Nu',
    u'\u039E': 'Xi',
    u'\u039F': 'Omicron',
    u'\u03A0': 'Pi',
    u'\u03A1': 'Rho',
    u'\u03A3': 'Sigma',
    u'\u03A4': 'Tau',
    u'\u03A5': 'Upsilon',
    u'\u03A6': 'Phi',
    u'\u03A7': 'Chi',
    u'\u03A8': 'Psi',
    u'\u03A9': 'Omega',
    u'\u03B1': 'alpha',
    u'\u03B2': 'beta',
    u'\u03B3': 'gamma',
    u'\u03B4': 'delta',
    u'\u03B5': 'epsilon',
    u'\u03B6': 'zeta',
    u'\u03B7': 'eta',
    u'\u03B8': 'theta',
    u'\u03B9': 'iota',
    u'\u03BA': 'kappa',
    u'\u03BB': 'lamda',
    u'\u03BC': 'mu',
    u'\u03BD': 'nu',
    u'\u03BE': 'xi',
    u'\u03BF': 'omicron',
    u'\u03C0': 'pi',
    u'\u03C1': 'rho',
    u'\u03C3': 'sigma',
    u'\u03C4': 'tau',
    u'\u03C5': 'upsilon',
    u'\u03C6': 'phi',
    u'\u03C7': 'chi',
    u'\u03C8': 'psi',
    u'\u03C9': 'omega',
}<jupyter_output><empty_output><jupyter_text>Now we can load the MFCC arrays of a certain element and let [density-based spatial clustering of applications with noise (DBSCAN)](https://en.wikipedia.org/wiki/DBSCAN)Epsilon [can be determined](https://iopscience.iop.org/article/10.1088/1755-1315/31/1/012012/pdf) quantitatively if needed<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from sklearn.cluster import DBSCAN
from sklearn import metrics
import time
from ast import literal_eval

# Select element to be clustered
eoi='Fe'
# Epsilon for dbscan 
epsnum=10
# Min samples in a group
minner=5

# Select mfcc arrays of element-of-interest and load into a literal array for dbscan
pdf_sel=pdf[pdf['_c2']==eoi]
pdf_sel['_c3']=pdf_sel['_c3'].apply(literal_eval)
X=np.array(pdf_sel['_c3'].values.tolist())

# Fits the array of mfcc, X with dbscan
dbs=DBSCAN(eps=epsnum,min_samples=minner).fit(X)
ele_subgroups=pd.concat([pdf_sel.reset_index(),pd.Series(dbs.labels_)],axis=1)

# Show the counts for each arbitrary group
pd.Series(dbs.labels_).value_counts()<jupyter_output><empty_output><jupyter_text>To plot this, we want to load normalized XAS and color them according to their groups that were found above. When a cluster parameters are finalized, we can save the dataframe by uncommenting the bottom 3 lines. These csv's will be the datasource for the interactive Plotly application to observe color-coded clustered spectra.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from sklearn.cluster import DBSCAN
from sklearn import metrics
import time
from ast import literal_eval
import re

# Load and match normalized XAS to their defined clusters
df_raw=pd.read_csv("http://exafs.ucsc.edu/bridges/data_agg_mini_cleanest.csv")
subgroup_ix=np.array(ele_subgroups['_c1'].apply(lambda x: int(x)))
dfr_sel=df_raw.iloc[subgroup_ix][['_2','e_zset','mu_zset']]
dfr_sel['_c1']=dfr_sel['_2']
ele_subgroups=ele_subgroups.set_index(pd.Series(ele_subgroups['_c1']).apply(int))
subg_anl=pd.concat([dfr_sel, ele_subgroups],join='inner',axis=1)
subg_anl['e_zset']=subg_anl['e_zset'].apply(literal_eval)
subg_anl['mu_zset']=subg_anl['mu_zset'].apply(literal_eval)

# The plot legend will include cluster counts
groupct=pd.Series(dbs.labels_).value_counts()

from matplotlib.lines import Line2D
custom_lines = [Line2D([0], [0], color='black', lw=4),
                Line2D([0], [0], color='blue', lw=4),
                Line2D([0], [0], color='red', lw=4)]

from matplotlib import colors as mcolors
colors = dict(mcolors.BASE_COLORS, **mcolors.CSS4_COLORS)

for pp in subg_anl.index:
  plt.scatter(subg_anl['e_zset'][pp], subg_anl['mu_zset'][pp], s=.1, color=list(colors.keys())[subg_anl[0][pp]+1])

custom_lines=[]
labes=[]
for i in groupct.index:
  custom_lines.append( Line2D([0],[0],color=list(colors.keys())[i+2],lw='3') )
  if i<0:
    labes.append(str('unclassed group with count = '+ str(groupct.iloc[i])))
  else:
    labes.append(str(str(list(greek_alphabet.keys())[i+2])+' group with count = '+ str(groupct.iloc[i])))

plt.legend(custom_lines, labes)
plt.show()

# Uncomment to save appropriate DBSCAN clustering of a certain element
#subg_anl.to_csv(str(eoi)+'_subg_eps'+str(epsnum)+'_min'+str(minner)+'.csv')
#from google.colab import files
#files.download(str(eoi)+'_subg_eps'+str(epsnum)+'_min'+str(minner)+'.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15065751858204687, 'fraction_numerical': 0.04302458547741567, 'mean_word_length': 4.934690415606446, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '618390', 'license': 'MIT', 'n_tokens_mistral': 2819, 'n_tokens_neox': 2638, 'n_words': 634, 'path': 'assets/ML_dbscan_mfcc.ipynb', 'repo_name': 'cmackeen/cmackeen.github.io', 'size': '6998'}